Double Horizon Model-Based Policy Optimization
Authors: Akihiro Kubo
Paavo Parmas
Shin Ishii
Venue: Accepted by TMLR ()

Abstract: Model-based reinforcement learning (MBRL) reduces the cost of
real-environment sampling by generating synthetic trajectories (called
rollouts) from a learned dynamics model. However, choosing the length of the rollouts
poses two dilemmas: (1) Longer rollouts better preserve on-policy training
but amplify model bias, indicating the need for an intermediate
horizon to mitigate distribution shift (i.e., the gap between
on-policy and past off-policy samples). (2) Moreover, a longer
model rollout may reduce value estimation bias but raise the variance
of policy gradients due to backpropagation through multiple steps,
implying another intermediate horizon for stable gradient estimates.
However, these two optimal horizons may differ. To resolve this
conflict, we propose Double Horizon Model-Based Policy Optimization
(DHMBPO), which divides the rollout procedure into a long
``distribution rollout'' (DR) and a short ``training rollout'' (TR).
The DR generates on-policy state samples for mitigating distribution
shift.  In contrast, the short TR leverages differentiable
transitions to offer accurate value gradient estimation with stable
gradient updates, thereby requiring fewer updates and reducing overall
runtime.  We demonstrate that the double-horizon approach effectively
balances distribution shift, model bias, and gradient instability, and
surpasses existing MBRL methods on continuous-control benchmarks in
terms of both sample efficiency and runtime. Our code is available at \url{https://github.com/4kubo/erl_lib}.

Published in Transactions on Machine Learning Research (4/2025)
Double Horizon Model-Based Policy Optimization
Akihiro Kubo1.2.*
wubb-a@@sss..kkyot-oo
Paavo Parmas
paavo. anavo.parmaas@ueblab.... tok-ooooooooo. ac.jp
Shin Ishii!:2
schi@i.kyolo-U ac.jp
Advanced Telecommunications Research Institute
Kyoto University
The University of Tokyo
Corresponding Author
Reviewed on OpenReview: https: // // openreview. net/ forum? id= HR
Abstract
Model-based reinforcement learning (MBRL) reduces the cost of real-environment sampling by generating synthetic trajectories (called rollouts) from a Iearned dynamics model. How- ever, choosing the length of the rollouts poses two dilemmas: (1) Longer rollouts better preserve on-policy training but amplify model bias, indicating the need for an intermediate horizon to mitigate distribution shift (i.e., the gap between on-policy and past off-policy samples). (2) Moreover, a longer model rollout may reduce value cstimation bias but raise the variance of policy gradients due to backpropagation through multiple steps, implying another intermediate horizon for stable gradient estimates. However, these two optimal hori- zons may differ. To resolve this conflict, we propose Double Horizon Model-Bass Policy Optimization (DHMBPO), which divides the rollout procedure into long "distribution roll- out" " (DR) and a short "training rollout" (TR). The DR generates on-policy state samples for mitigating distribution shift. In contrast, the short TR leverages differentiable transitions to offer accurate value gradient estimation with stable gradient updates, thereby requir- ing fewer updates and reducing overall runtime. We demonstrate that the double-horizon approach effectively balances distribution shift, model bias, and gradient instability, and sur- passes existing MBRL methods on continuous-control benchmarks in termss of both sample efficiency and runtime. Our code is available at https: // github. com/ htt///uub/ee// erl lib.
1 Introduction
Reinforcement. learning (RL) is framework for finding an optimal policy in sequential decision-making ap- plications through trial-and-error interactions with an environment. Model-based RL (MBRL), which learns a dynamics model an environment in a data-driven manner, leverages the learned model to generate syn- thetic samples, thereby reducing costly real-environment interactions. However, if the model imperfect, long synthetic rollouts can accumulate significant crrors-known as model esaeciasccaalll as the rollout length increases. Additionally, data collected under poorly performing policy may differ from that encoun- tered by the current policy, resulting in distribution shift in both policy evaluation (Hallak and Mannor, 2017) and policy improvement (Liu et al., 2019). Several MBRL methods address these issues by tailoring the length and the usage of the model's predictions.
Broadly, these successful approaches can grouped into two types. The first type employs differentiable rollouts (Deisenroth and Rasmusson, 2011; Amos al., 2021; Hafner et al., 2019; Hansen et al., 2024) or leverages model derivatives of predictions (Zhang al., 2023). By backpropagating through few model- predicted transitions, these methods compute model-based value expansion (MVE) estimator (Feinberg

---

Published in Transactions on Machine Learning Research (4/2025)
et al., 2018), which sums the on-policy rewards for intermediate states with the estimated return from the terminal state (or a critic's prediction). As the MVE, estimator refines value estimation, the policy can be improved using more accurate on-policy value estimates even with a lower update-to-data (UTD) ratio iee. fewer policy parameter updates per environment step. This lower UTD ratio also helps suppress the increase in computation time (Hiraoka et al., 2022). However, extending disferentiable rollouts in conjunction with the reparamctrization trick (Kingma and Welling, 2013) can inflate the variance of the policy gradients (Parmas et al., 2018), thereby limiting the effective length of these training rollouts. In this work, we refer to the policy gradient computed via differentiable rollouts the "value gradient" (Fairbank and Alonso, 2012; Heess et al., 2015; Zhang et al., 2023), distinguishing it from the gradient estimated by purely model-free algorithm (Sutton et al., 1999; Silver et al., 2014).
Meanwhile, another line of MBRL research the model-bases policy optimization (MBPO) algorithm (Jan- ner et al., generatess one-step transitions that are treated as if they were obtained from the real environment. Starting from states stored in the replay buffer (which contains states, actions, and rewards), MBPO rolls out a few steps with the learned model to generate synthetic states that more closely match the current policy's s state distribution. MBPO has demonstrated excellent empirical performance on the OpenAl Gym benchmark (Brockman et al., 2016). Although the MBPO formalization and theorctical analysis do not necessarily support the benefits of model- genee--eeerated on-policy data, we interpret its empirical success as evidence that optimizing the policy over an on-policy state distribution can improve learning in practice.
In this work, we propose combining these two ideas: a "distribution rollout" " (DR) that approximates the on-policy state distribution (inspired by the model rollouts in implementation of MBPO algorithm), and a "training rollout" (TR) that produces an MVE estimator by exploiting differentiable transitions. However, the optimal horizons for DR and TR may differ. A longer DR helps maintain on-policy samples but also increases model bias, implying an intermediate horizon to minimize distribution shift. In contrast, longer TR can reduce value estimation bias while raising the variance of the policy gradients, suggesting another intermodiate horizon for stable gradient estimates.
Therefore, we propose using different rollout horizons for DR and TR a long horizon for DR and a short horizon for TR. Since we perform rollouts with different horizon lengths, we refer to this method as the Double Horizon MBPO (DHMBPO) algorithm. In DHMBPO, the TR begins from on-policy states provided by the long DR, enabling to maintain short TR horizon while still ensuring accurate value estimation and effective policy improvement. Figure 1 illustrates the concept and key differences between DR and TR.
In experiments on contimnous-contro benchmarks (Section 4), we demonstrate that DHMBPO not only surpasses existing MBRL methods in sample efficiency but also achieves lower runtime due to a reduced UTD ratio (Hiraoka et al., 2022). Notably, DHMBPO achieved comparable sample efficiency to the stato-of- the-art MACURA (Frauenknecht al., 2024) algorithm on the Gymnasium (Towers al., 2023) tasks while requiring only one-sixteenth of the runtime average, all using shared set of hyperparameters (see A). The remainder of this paper reviews the necessary background (Section 2), details our approach (Section 3), and presents empirical evaluations that highlight the benefits of combining these two complementary rollouts.
2 Background
2.1 Notation and Problem Setting
In this work, we consider a Markov decision process (MDP) with entropy regularization (Geist et al., 2019). At each environment step t, an agent in state s, selects an action a a, according to a stochastic policy Te(a| st) parameterized by 0, transitions to the next state $141 based on the transition probability p(s st, at), and receives reward r. = r(st, az). Given a discount factor y E 10, for weighting future rewards, distribution μ(s) from which initial states so are drawn, and a parameter 2 0 for the entropy bonus, the objective is to find a policy that maximizes
](TO)= == Ee (r(s log (a(ss))
(1)
where Pioo is the discounted state visitation distribution under no. defined as
Pry (s) == (1 7) 7 e = m0), with p(s = slu,,o) representing the probability of observing
2

---

Published in Transactions on Machine Learning Research (4/2025)
8
ATTA V V
S
MBPO TR-based DHMBPO SAC
Eq. (12)
Model Buffer
Model-free gradient
Distribution Rollout
Q
r
E [o
VT rt QT
Eq. (11)
Environment
Replay Buffer
Training Rollout
Model-based gradient
Figure 1: An overview of four approaches to policy gradient estimation: MBPO, TR-based methods, DHMBPO, and an off-policy model-free method. (1) MBPO (blue lines): Starts from states stored in the replay buffer and runs a distribution rollout (DR) with horizon D, generating synthetic samples that are then stored in model buffer. A model-free policy gradient is computed over these samples. (2) TR-based (green): Methods that directly use states from the replay buffer and perform short-horizon training rollout (TR) with horizon T. A model-based gradient estimate- the value gradient-is computed. (3) DHMBPO (red, proposed): Combines both DR and TR. It first applies DR on samples from the replay buffer to obtain on-policy states, which serve as the imitial states for a short-horizon TR. The value gradient then computed from the TR. (4) Model-free (yellow): Represents a fully model-free algorithm that relies solely on replay-bur buffer data and computes a model-free gradient estimate.
state time and p(so s|u, m8) = u(s) Moreover,
VI
(r(se, ae) - a log To (a((ass))) 80
(2)
is soft value function, and the objective function (1) is rewritten as J(T] = Eswn [s). The ex- pectation in eq. (2) is taken over the stochastic trajectory (so, ao, .1,...) drawn from its distribution Ta (ssel,, TT).
We use an associated soft Q-function
(s,a) = r(s, a) + Bar (s)) [V.
(3)
We assume that, for all S, (s) = Baws(-le) 10% (8. a) - a log lals))
If a = O, eq. (2) is equivalent to the objective of an unregularized MDP, Le,,EI>r what follows we do not strietly differentiate between the eetropy-regularized MDP and the unregularized MDP; for simplicity, we write (s) for the value function and Q* (s, a) for the Q-function.
7'r(st, at).. 7
Hence, in
2.2 Model-based Ector-Critic Methods
In this study, we consider a deep model-based actor-critic architecture. Specifically, we have neural networks for a policy =a(as) as an actor, a critic (s a) that approximates (s a), and a dynamics model p (s'(s,, a) along with reward model (s, a). The parameters é, and is denote the weights of the respective neural networks.
During critic learning, for an input (s,a), target signal ÿ is computed, and the critic Qs is up- dated to minimize the squared difference between its output and ÿ. Meanwhile, the actor To is up- dated to maximize the sample average (sampled from a given state distribution) of the estimated value Vau (s) 2 Ea-Ro( [Q((s a) a log (a(as))] Following existing works (Janner ct al., 2019; Hafner et al., 2019; Hansen al,, 2022; 2024), we perform alternating updates of the critic and actor within policy optimization loop. This alternation is explicitly shown in the for loop (line 8) of Algorithm 1.
3

---

Published in Transactions on Machine Learning Research (4/2025)
2.3
Model-Based Policy Optimization (MBPO)
The MBPO algorithm (Janner et al., 2019) applies the policy optimization procedure from SAC (Haarnoja et al., 2018) to fictitious samples generated via model rollouts. Here, we refer to these rollouts as DR. To execute DR, we randomly select initial states from the replay buffer and then simulate D steps using the learned model. The resulting trajectories are stored in a "model buffer", D which is separate from the replay buffer.
The critic Qo is updated to minimize:
(D(0)= = B(saano)D (Q)(s,a)- - -0)
(4)
where the larget signal is defined as = r(8,a) a) + (a ( (s'ss'a)) a log (aa'6'))) Here, Q3 is target critic with parameters computed as an exponentially moving average ø (Mnih et al., 2015). The notations (s, a, s') D and s DR denote independent sampling operations from the model buffer. The policy is then updated by stochastic gradient ascent the surrogate loss given by
JD(0) = EDDD [o.o(s)]
(5)
where
Vo.e()) = E Eawno (-1x) (tn (0((s)) (o)(())) log (a(as)) .
(6)
2.4 Training collout-based Methods
In this work, refer to methods that employ the MVE estimator (Feinberg et al., 2018) as TR- based with differentiable rollout (TR). The MVE estimator for state s uses a T-step TR trajectory
= (80- 8, up, 81, $1,01, ... 01, generated by the policy my and the model pe, to compute:
T-1
(o))((s)= = Σ r (ru( 0z) a (og + n2 (GT),, 1=
(7)
where Vo($T)= Q1(81, aT) log (o(aal(T)) (&r) se (owno()ler) [Qa(6T, a -olooo(((57))) Even when (a( is imperfect-dur to the critic Qo struggling to keep pace with rapidly changing policy-the the on-policy reward sequence to - steps improves the overall value estimate at 8 (Feinberg et al., 2018). Although Feinberg et al. (2018) assumes that the reward function is known, learning a reward model ro typically poses fewer challenges than learning a dynamics model or Q-function, so similar benefits are expected.
Following Hafner et al. (2019), we utilize the MVE estimator for learning both of the critic and the policy. Specifically, the critic is trained to approximate the Q-function Q*a a) by minimizing
(o(0) = (sa))D, [Qo(s,a) - (0.r(())))))
(8)
where
T-1
(s)r(s,))=rv(s00 +
Σ
r2 (ro - (og + 7" VV((T)).
(9)
For the policy, we perform stochastic gradient ascent on the surrogate loss
(r) = ESDD. [ooo((s)
(10)
The gradient Volo.er(s) is computed by combining the reparametrization (RP) trick (Kingma and Welling, 2013) with backpropagation through the TR in eq. (7). We refer to the gradient VgVo,,, (s), as the "value gradient field," " which is function of the state It has been shown that when T becomes large, this process can be prone to gradient variance explosion (Parmas et al., 2018). Hence, in practice T is often limited to about 5 steps to maintain stable optimization (Amos et al., 2021).
We write "surrogate loss" hecause the reward in eq. (1) is replaced with the value function, akin the Off- Pfl-Pol actor-critic framework Degris et (2012).
4

---

Published in Transactions on Machine Learning Research (4/2025)
2.5 Policy-Vatle Value Gradient
A key subproblem in policy optimization is to estimate the gradient of the policy parameters, VoJ((), for the objective in (1). We assume the stochastic policy is reparametrized so that for a function v(a):
En-ms(als) [b(a]] Be-z [s) where Z is simple distribution (e.g, Gaussian), and To(als, c) is a deterministic function mapping (8,6) to a. In this work, we estimate the policy gradient via the RP trick.
The policy gradient can be expressed as
VoJ (TO)
Esyn VoV = -evruss (s Exppro,c Voa V. (s, as) (amm))))))(
(11)
(12)
as derived in (Zhang et al., 2023), with the final expression derived by extending the deterministic policy gradient theorem (Silver et al., 2014). Here, we assume a 0, and the notation s." denotes an element-wise product summed over the action dimension.
The last term (12) considers only one-step change in action, which enables us to replace QT4 with the critic Qa when approximating the partial derivative V. @**((ss (s)a)) (== We refer to the resulting estimate, Voa- Va (o((s.)) as "a model-free policy gradient field".
On the other hand, the first term is the (stochastic) value gradient, where VoV**(s) captures how changes in future actions affect the value function. In RL selting where the true transition probability is unknown, dynamics model is required to estimate this term. The gradient field VoVo,,r(()) of the MVE estimator (7) can be used for this purpose. However, it is important to note that the terminal term in this gradient is a model-free policy gradient field V ar Va Qo(êr, oT) (a)aa==o((rrr))
Additionally, note that the expectation in equation (12) is taken with respect the discounted state- visitation distribution, whereas, in the first. and second terms of (11), it taken with respect to the initial state distribution. As implied in Zhang et al. (2023), mixture of the initial state distribution and the discounted state-visitation distribution may be preferable, the terminal term of the value gradi- ent approximated by the MVE estimator is a model-free policy gradient field.
2.6 Bootstrap ensemble model and its learning
Following Chua et al. (2018); Janner et al. (2019), we use a bootstrap ensemble of probabilistic neural networks, where the outputs are independent multivariate Gaussian distributions. Individual probabilistic models capture aleatoric noise, while the bootstrapping procedure accounts for epistemic uncertainty. The details of the model, including its architecture, objective function, and early stopping, are described in Appendix A (from Section A.1 to Section A.3).
3 Double Horizon Model-Based Policy Optimization
This section introduces the Double Horizon Model-Based Policy Optimization (DHMBPO) algorithm, which integrates DR and TR to approximate both the state distribution and value estimation. Before detailing DHMBPO, we highlight the differences between DR-based methods and TR-based methods.
A common procedure for both DR-based and TR-based methods is to alternate between (i) sampling states and actions from distribution d(s, a) and (ii) updating policy using samples from the estimated policy gradient field.
MBPO samples states and actions from the model buffer DR, which closely approximates an on-policy distribution, and computes the model-free policy gradient field:
d(s,a) DD and Vo ()) V.
where, due to DR, sampling from the model buffer be interpreted approximating T in (12).
5

---

Published in Transactions on Machine Learning Research (4/2025)
Algorithm Double Horizon Model-Based Policy Optimization
Require: Models Pe and Tap: actor network Ra, critic network Qø target critic network Qg. replay buffer Pe. model buffer D for == 0,,,, do
2: Interact with the enviromment using we and add observed transitions to De
3: Fit the models Pe and Te to samples from D.
4: Sample states from De; then perform D-step DR and store the generated transitions in D Policy optimization loop
5: for i= 0 to L - do
Sample states from De and execute T-step TR Compute the MVE estimates (7) and (9)
Update to minimize (13) and then update 0 to maximize (14) via eq. (11)
Clear D
TR-based methods use a replay buffer De in place of d(s, a) and employ a value gradient field via the T-step MVE estimator, V (oVo..()))
d(s,a) De and Velo.o(()).
Note that, as discussed in Section 2.5, sampling from a more on-policy distribution is preferable to sampling from the replay buffer.
In DHMBPO, we propose to replace Pro (s) with model buffer D generated via DR, and to use value gradient field of the T-step MVE estimator:
d(s,a) DD and VoVoeer(s),
That is, randomly select samples from the model buffer, similar to MBPO, and for the actor update, calculate VoVo.o,7(s) using the same procedure TR-based methods. This approach reduces distribu- tion shift through DR without requiring on-policy and real-environmen interactions while improving value estimation by leveraging the MVE estimator from TR.
Concretely, the critic loss function is defined as
(p) (( (Qo(s,a) Q
(13)
and the actor's objective is given by
(er(0))EEnpp) (o.o((s))
(14)
Algorithm 1 presents the procodure for the DHMBPO algorithm. Here, denotes the length of an episode, and repeating the policy optimization times corresponds to UTD ratio of 1.
Long DR rollouts help maintain ou-policy training but increase model bias, suggesting an intermediate rollout length to minimize distribution shift. In contrast, longer TRs reduce value estimation error but raise the variance of policy gradients, implying another intermediate horizon for stable gradient estimates. We keep TRs short to ensure stable policy gradient estimation and compensate by using longer DRs, thereby leveraging model predictions to achieve adequate value estimation accuracy while enabling effective policy improvement. In this work, we set the DR horizon to 20 steps and the TR horizon to 5 steps.
4 Experiments
We validate our DHМBPО on standard continuous control tasks (from Section 4.1 to Section 4.2). We also present results for sensitivity analyses with respect to rollout lengths, including the special case where either DR or TR entirely removed (Section 4.3 and 4.4), as well as experiments investigating the impact on efficient critic learning (Section 4.5).

---

Published in Transactions on Machine Learning Research (4/2025)
4.1 Continuous control benchmark tasks
We evaluate the DHMBPO algorithm on suite of MuJoCo-based (Emanuel el al., 2012) continuous control
tasks from Gymnasium (GYM) (Towers et al., 2023) and DMControl (DMC) (Tunyasuvunakool et al., 2020). These tasks range from basic control problems to high-dimensional robot locomotion, enabling us to assess the general effectiveness of our approach.
These experiments aim to answer three key questions:
Does the combined use of DR and TR enhance sample efficiency?
Does facilitate efficient critic learning?
How does DHMBPO compare to other state-of-the-art MBRL algorithms?
All DHMBPO runs share a common set of hyperparameters, which, along with other implementation details, are described in Appendix A.
After z environment steps, measure the algorithm's performance using test return. Specifically, the test return for DHMBPO is computed as the sample mean of the cumulative rewards over 10 episodes, whereas some other methods use fewer episodes (see Appendix B for details).
Evaluation Protocol.
Performance Visualization and Statistical Analysis. We illustrate each method's performance on individual tasks by plotting the mean test return over random scods (solid linc) along with its 95% confidence interval (shaded area). To aggregate and compare results across multiple tasks, we utilize rliable (Agarwal et al., 2021), which provides various statistical tools, including sample efficiency curves and aggregation metrics.
However, because existing works typically focus on either GYM or DMC (but not both), their performance ranges are not directly comparable. We thus apply task-specific normalization: (i) the number of environment steps is normalized by the maximum step budget for each task, and (ii) the test return is divided by the final performance of a designated baseline. Note that this additional normalization is our adaptation unify GYM and DMC results. In our GYM comparisons (Figure in Section 4.2), however, we present raw (unnormalized) returns to align with common practice (Chua et al., 2018; Janner et al., 2019; Amos et al., 2021; Prauenknecht et al., 2024).
In Section 4.3, present ablation study that isolates the effect of combining DR and TR, and in Section 4.2, we benchmark DHMBPO against several high-performance MBRL algorithms.
4.2 Comparison with State-of-The Algorithms
In this scction, we compare the proposed DHMBPO algorithm with several high-performance deep MBRL algorithms whose implementations are publicly available.
For the comparison on the GYM suite, we selected five tasks and compared DНМBPО with MBPO (Janner et al., 2019), SAC-SVG(H) (Amos et al., 2021), RP-PGM (Zhang et al., 2023)] and MACURA (Frauenknecht et al., 2024). In addition to MBPO, SAC-SVG(H) serves as an important baseline algorithm that employs the SVG method-a a type of TR-basod algorithm. The primary difference between DHMBPO without DR and SAC-SVG(H) that DHMBPO uses deep ensemble models (Lakshninarayanan et al., 2017), whereas SAC-SVG(H) employs GRU (Cho, 2014) its deterministic recurrent neural network. Additionally, SAC- SVG(H) learns the critic using one-step TD targets during the TR. RP-PGM was verified using a modified implementation based on SAC-SVG(H), which was adjusted to mitigate the exploding variance issue in policy gradient estimation. MACURA based on the MBPO algorithm but adaptively adjusts the length of the DR and is reported to achieve state-of-the-art performance on the GYM suite. Furthermore, as a reference, we included the model-free and off- off-policy SAC algorithm as comparison target. Although SAC is model-free, its objective function formulation highly relevant those optimized by DHMBPO, MBPO, and SAC-SVG(H) algorithms, particularly in that it includes entropy regularization term.

---

Published in Transactions on Machine Learning Research (4/2025)
HailChuetab
Hopper
Humanoii
Wallerrd
0.000
7500
7000
5000
2000
5.00
5030
Text
1000
2503
2006
Enrbommnn skoye
X10
Eastroument stopo
Eavinssnees steps
Enricusaeto stope
Easiroument stopo
DIMBDO
MBBO
MICCURAA
SAC-VC//B)
RP-PGM
SAC
scapp
(a) Sample efficiency
HaltCheetah
Hopper
Humanood
Wallser2d
6000
110100
5107
7500
50.00
1000
6000
Text
0200
2000
1000
2003
2500
15
[Moors]]
Runtiime [Houm]]
[ftor]]
x004
[ticu]]
DHMRPO
MIRPO
MACURA
SAC-SVG(R)
RI-DGM
SAC
(b) Test return curve as a function of runtime at each constant environment step.
Figure 2: Comparison with state-of-the-art methods on GYM tasks in terms of sample efficiency (top) and runtime up to 500K environment steps (bottom). The solid line represents the sample mcan over cight random seeds, and the shaded area indicates the 95% confidence intervals. Dashed lines for SAC represent the average test return across 5 random seeds, ten times the maximum environment step budget used for the model-based methods. The runtime plots show that DHMBPO reaches the highest test returns the fastest, highlighting its efficiency in both sample and runtime cost.
For the comparison on the DMC suite, we selected 18 tasks and compared DHMBPO with TD-MPC2 (Hansen et al,, 2024) and Dreamer v3 (Hafner et al., 2023). Both of these are latent model-based actor-critic algorithms that have demonstrated exceptional performance across multiple benchmark suites, including DMC, using common hyperparameters. TD-MPC2 combines representation learning and reward learning, and during behavior execution, it performs model predictive control. Dreamer v3 combines representation learning with the MVE estimator from TR, which is used for training both the critic and the actor.
4.2.1 Comparison on GYM Tasks
Sample efficiency The results on GYM tasks are shown in Figure 2a, where the sample mean over eight random seeds is plotted as a solid line and 95% credible intervals are depicted as the shaded area. For MBPO, five random seeds were used. Our experiments confirm that DHMBPO achieved the highest sample efficiency across all tasks even without per-task tuning. This substantial practical performance improvement is arguably the most important contribution of our work. For reference, we include horizontal lines indicating the asymptotic performance of SAC. Specifically, these lines represent the average test return across 5 random seeds, at 5M environment steps, a number of steps equal to ten times the maximum environment step budget used for the model-based methods in each task.
Runtime Comparison For the runtime comparison of DHMBPO, MACURA, and SAC-SVG(H) on GYM tasks, we present the results in Figure 2b and summarize them in Table 1. Each experiment was executed until 500K environment steps system configured with 8 NVIDIA RTX A4000 16GB GPUs. Considering the sample efficiency results mentioned carlier, DHMBPO reached 500K steps in less than one-sixteenth of the runtime while achieving the same or higher sample elficiency as MACURA. The primary reason for MACURA's longer runtime is its higher UTD ratio. For example, the UTD ratio for MACURA on Humanoid task was 20, while that of DHMBPO (and SAC-SVG(H) was 1. Although a high UTD ratio can promote higher sample efficiency (Chen et al., 2020; D'Oro al., 2023), we observed that DHMBPO performed well with UTD ratio of 1, thereby reducing runtime. In fact, increasing the UTD ratio yielded only
8

---

Published in Transactions on Machine Learning Research (4/2025)
Table I: Runtimes (in hours), until 500K environment steps, of DHMBPO, MACURA (Frauenknecht et al., 2024) and SAC-SVG(H) (Amos al., 2021), on GYM tasks and each ratio DHMBPO's mean runtime.
Task
DHMBPO
MACURA
SAC-SVG(H)
Ant
3.6
58.3
5.2
HallCheelah
3.3
32.6
6.7
Hopper
3.7
73.3
6.7
Humanoid
5.2
104.3
6.9
Walker2d
4.0
63.0
6.6
Ratio
1.0
16.8
1.6
Median
IQM
SAC
Dreamer v3
TD-MPC2
DHMBPO
0.60
0.75
0.90
1.05
0.60
0.75
0.90
0.60
0.75
0.90
1.05
Figure 3: Comparison of aggregation metrics for TD-MPC2 (Hansen et al., 2024), Dreamer v3 (Hafner et al., 2023) and SAC (Haarnoja et al., 2018) on 18 DMC tasks. The metrics are normalized relative DHMBPO's scores.
marginal improvements in sample efficioncy while significantly increasing exccution time (sce Figure 20 in Appendix D.5).
4.2.2 Comparison on DMC Tasks
Next, in Figure 3, we present the aggregated metrics on DMC lasks. The metrics are about the test return at 50% normalized steps. The three metrics for each method are shown, with the vertical bars representing the metric values and the width of the bars indicating the 95% confidence intervals. The inter quantile mean (IQM) score demonstrates that DHMBPO exhibited significant. performance improvements over Dreamer v3 and less variation than TD-MPC2.
Additional task-specific learning curves and runtime results are provided Scction C.
4.3 Ablation study
Figure shows the experimental results evaluating the sample efficiency of the proposed DHMBPO algorithm and its variants on a set of 10 continuous control benchmark tasks. We selected five representative tasks from the GYM suite (upper row) and five relatively challenging tasks from the DMC suite (lower row). All compared methods share the same model-learning modules and implementation details, differing only in how DR and TR are combined.
We consider three configurations of DHMBPO corresponding different (DR horizon, TR horizon) settings: DHMBPO (20, 5), the proposed algorithm, which uses both DR and TR. DHMBPO without DR (0, 5): Initial states are sampled from the replay buffer and used for TR; this setup corresponds to SAC- SVG(H). DHMBPO without TR (20, 0): Initial states are sampled from the model buffer, and the policy optimization procedure follows SAC as described in Section 2.3; this setup corresponds to MBPO.
For each configuration, we ran experiments using 8 different random scods. The solid lines in Figure represent the mean performance across these seeds, while the shaded areas indicate the 95% confidence intervals.

---

Published in Transactions on Machine Learning Research (4/2025)
Ani
HailChuesab
Hopper
Humamaid
WallerRd
5000
0000
9000
6.000
7500
4000
2000
4.00
1000
2003
2000
skrge x101
steps
<tege
strge
XII
strpp
cuttpolo-wwiggg sparss
quddruped-rum
huroonold-staad
hopper-stand
fingm-tarn hard
500
750
500
002
3003
Test
190
170
200
200
Enviomaen stope
Eastroument stopo
Eaviooancss steps
Eurbenaees: stape
Eustroument stopo
DIDEPP
DEMBPO w/jo DR DEMBPO w/o TT
Figure 4: Performance comparison of DHMBPO (orange) against its two ablated variants: DHMBPO with- out DR (w/o DR, corresponding to SAC-SVG(H), green) and DHMBPO without TR (w/o TR, corresponding to MBPO, blue). The top row shows results on five representative tasks from the GYM suite, and the bottom row shows results on five more challenging tasks from the DMC suite. Each solid line represents the mean test return over random seeds, and the shaded regions denote the 95% confidence intervals. The proposed DHMBPO configuration (DR horizon = 20, TR. horizon = 5) consistently outperforms both variants.
The results show that combining DR and TR outperforms using either DR or TR alone. In particular, the DHMBPO configuration (20, 5) achieves higher returns and better sample efficiency than the two re- duced variants. For example, the TR-only variant (0, 5), which corresponds to SAC-SVG(H), benefits from improved value estimation; however, its reliance on off-policy samples leads to lower sample efficiency in challenging tasks such as Humanoid, Walker2d, and quadruped-run. Meanwhile, the DR-only variant (20, 0), corrosponding MBPO, can approximate the on-policy state distribution, but the absence of TR's MVE estimation results in ineffective value corrections and notably inferior performance across all tasks. Overall, these results demonstrate that the synergy achieved by combining DR and TR in DHMBPO leads to faster learning than using either technique alone.
We include additional experiments in Appendix D.1 for DHMBPO without TR (correspouding to MBPO) but with different UTD ratios. Increasing the UTD ratio improved sample efficiency (Figure 11) but also resulted in longer execution times (Figure 12). In other words, achieving high sample efficiency without TR requires sacrificing computation cost by raising the UTD ratio. In contrast, DHMBPO achieved high sample efficiency without increasing the UTD ratio (Figure 4), indicating that introducing TR helps reduce runtime cost. Indeed, as shown in Section 4.2, DHMBPO attained sample efficiency comparable to state-of- state-of-hhe-rtt MBPO-based methods in a shorter execution time.
4.4 Hyper-parameter sensitivity
Figure rilusttaaes our sensitivity analysis of the distribution-rollout (DR) and training-rollout (TR) horizon hyperparameters. We evaluated 10 tasks under 8 random sceds, mcasuring the test return after 500K envi- ronment steps. To avoid bias toward any single metric, the figure reports three summary statistics median, inter-quantile mean (IQM), and mean-along along with their 95% confidence intervals.
We chose DR-20 and TR=5 our bascline. In Figure 5a, we varied only the DR horizon (0. 10, 20, 40), while keeping TR fixed at 5. In Figure 5b, we varied only the TR horizon (1, 3, 5, 7, 9), while keeping DR fixed at 20. The results show that: 1) DR benefits from being relatively long (e.g., 20) compared to short horizons (o.g., 0). 2) TR=5 offers a good balance between sample efficiency and gradient stability. short TR reduces sample efficiency (similar "DHMBPO w/o TR" in Figure 4), while excessively long TR can cause gradient norms to explode (see Appendix C.3, Figure 10b), destabilizing learning. Furthermore, in
10

---

Published in Transactions on Machine Learning Research (4/2025)
Media:
IQB
KQM
20
10
1-
1,022
1.08
1:34
1,02
1:06
1.08
1.11
3.08
1.14
0.9
1.0
0..5
0.0
0.0
1.0
(a) Variants of DR horizon.
(b) Variants of TR. horizon.
Figure 5: Aggregation metries on 10 tasks for (a) variants with different DR horizons and (b) variants with different TR horizons.
Ant
HalfCheetah
Hopper
Humaaooi
Walker2d
1.2
RMoASR
0.9
0.9
0.8
0.4
0.6
0.4
0.3
0.3
0.0
0.0
0.0
0.0
0.0
10
10'
10
10'
10"
10'
Iteration
Iteration
Iteration
Ineration
Incration
with DR
without DR
Figure 6: Development of the RMedSE for DНмBPО (orange) and DHмBPО without DR (blue). The æ- axis represents the number of iterations used to update the critic parameters while all other models remain fixed. The y-axis shows the normalized RMedSE with respect to the Monte Carlo return (N=256) from the target environment.
Scction D.2, we investigate the trade-off in policy gradient estimation through TR and propose practical method to determine an optimal TR horizon based small number of real samples.
Overall, these findings suggest setting DR to a moderately long horizon and TR to shorter, carefully chosen horizon to prevent gradient explosion and maintain robust performance.
4.5 Efficient Critic Learning
We demonstrate the benefit of DR in critic learning, a more accurate value estimation. We ran DHMBPO for 100K environment steps on five environments from the GYM suite. After training, the learned dynamics model and actor network were fixed, and the critic networks were re-initialized to compare two variants for critic learning: with DR: Train the critic using model-generated rollouts stored in a model buffer. without DR: Train the critic using only samples from the replay buffer, without additional model-based rollouts.
As ground truth values, we computed Monte Carlo return (s, a) using 2,048 rollouts in the real envi- ronment, for each of 256 randomly sampled state-action pairs (s,a) drawn from the replay buffer. We then computed the MVE estimate Q2,0,5(s, a) for these 256 pairs, and measured its discrepancy from (8, using a normalized root median squared error (RMedSE):
- -aael(sa)) 27
B(i) =
Median
2 (7(8,))
where is the critic optimization step. "Median" denotes the median over the selected 256 state-action pairs. For each setup, we repeated the procedure with 32 different random seeds for the critic's initialization and plotted the mean (solid line) and the 95% confidence intervals (shaded area) of E(i) as a function of
As shown in the last three pancls of Figure 6, we observe that the variant with DR exhibits a significantly lower estimation error, particularly during the early stages of training. This difference indicates that the MVE istimotor more accurately approximates the true value function. Because the policy is continuously updated
11

---

Published in Transactions on Machine Learning Research (4/2025)
throughout training, an early advantage in value estimation compound and benefit each subsequent optimization step. In the Humanoid environment specifically, we believe that this improved value estimation under DR contributes to the notable gains in sample efficiency observed in Figure 4.
5 Related Work
Many practical off-policy actor-critic methods, including the SVG algorithm, rely on uniform sampling from the replay buffer. Although importance sampling corrections have been proposed in model-free reinforcement learning (Hallak and Mannor, 2017; Zhang et al., 2020), they are often prone to issues such as high variance in the likelihood ratios (Liu et al., 2019).
In MBRT, MBPO (Janner et al., 2019) introduced DR, an extension of Dyna-style rollout which generates one-step transitions used along with real samples (Sutton, 1990). Although MBPO achieves high sample efficiency, it sulfers from extensive runtime requirements. In contrast, while MBPO uses the model solely for generating virtual data, the SVG method leverages the differentiability of the model directly during policy optimization, enabling more efficient learning.
PILCO (Deisenroth and Rasmussen, 2011), which employs Gaussian processes (GPs) as ils model, demon- strates very high sample efficiency by performing policy optimization using SVG on the analytical distribution of the cumulative reward. However, when using sample-based predictions instcad of assuming an analyt- ical distribution, gradient estimation becomes custable--ssecialll for long prediction horizons- errors accumulate over time (Parmas et al., 2018). This issue is particularly pronounced in tasks where accurate long-term predictions are crucial.
Furthermore, as discussed in Section 3, SVG methods have not adequately addressed the issue distribution shift in the objective function. Specifically, Feinberg et al. (2018) explored the benefits of model-based value estimation and derived conditions for achieving an on-policy stationary distribution; however, the utility of combining these conditions with SVG for policy optimization remained unclear. In this study, we introduce novel approach that employs the MVE estimator for both actor and critic learning, thereby resolving the distribution shift issuc. This constitutes the key novelty of our work.
Finally, while it is possible to use importance sampling (IS) to address distribution shift- as is done in the model-free reinforcement learning literature- our study focuses on achieving practical performance improve- ments. We demonstrate that simply adding DR leads to performance gains without significantly incrcasing runtime. This simple yet effective approach highly beneficial for developing RL technologies where sample efficiency is critical.
6 Conclusion
In this study, we introduced DHMBPO, novel MBRL algorithm that integrates two distinct model rollouts long-horizon horizon DR and a short-horizon TR- to address two critical trade-offs: (i) state distribution shift versus model bias, and (ii) value and value gradient. accuracy versus policy gradient instability. Long DR resamples from the model to better approximate the on-policy state distribution, while short TR leverages differentiable transitions to provide accurate on-policy value estimation with stable gradient updates, thereby reducing the number of required updates and overall runtime. By assigning different horizon lengths to DR, and TR, we preserve sample efficiency without incurring excessive model bias or increased runtime.
Experimental results multiple contimmous-control tasks indicate that DHMBPO consistently achieves faster learning and shorter runtime compared to existing MBRL baselines. We observe that the synergy of long DR with short TR enables carly improvements in value estimation, which accumulate over successive policy updates. Although the method provides a more balanced solution to the model bias versus off policc data challenge, exploration remains an open issue. For instance, certain tasks such as finger-spin (sce Figure 8 in Appendix C) can still lead to local optima. Nonctheless, the relatively short runtime of DHMBPO facilitates iterative refinement and experimentation, offering a cost-effective path toward practical performance gains in reinforcement learning applications where sampling cost is critical.
12

---

Published in Transactions on Machine Learning Research (4/2025)
Acknowledgement
PP was supported by JSPS KAKENHI Grant Number JP22H04998.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G. Bellemare. Deep reinforcement learning the edge of the statistical precipice. In Proceedliggs of the 35th International Conference on Newral Information Processing Systems, NIPS22, pages 29304-2968 Curran Associates Inc., 2021. 4.1
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynomics and Control, pages 6-20. PMLR, 2021. 2.4, 4.1, 4.2, I, B, D.3
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ar preprint NiXiv:1607.06500 2016. A.1.1
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeicch Zaremba. OpenAl Gym. arXiv preprint 160i: 01540, 2016.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-efficient re- inforcement learning with stochastic ensemble value expansion. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, N8PS'18, pages 9234-8244. Curran Associates Inc., 2018. A.5
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross. Randomized ensembled double Q-learning: Learning fast without model. In International Conference Learning Represenlations, 2020. URL https://openrevvew http:://peerreeeeeeeeeeetonnnaae
4.2.1, A.1.2
Kyunghyun Cho. Learning phrase representations using RNN encoder-decoder for statistical machine trans- lation. arXie preprint wrXiv:1406.1075, 2014. 4.2, D.3
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, volume 31, pages 4754-4765. Curran Associates, Inc., 2018. 2.6, 4.1, A.2
Thomas Degris, Martha White, and Richard S Sutton. Off-Policy Actor- Actor-Critic. Proceedings of the 29th International Conference on Machine Learning, ICML 2012, 1:457 464, 2012.
Marc Deiscnroth and Carl E Rasmussen. PILCO: A model-based and data-efficient approach to policy scarch. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, pages 465 472. Omnipress, 2011. 1,
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In International Conference Learning Representations, 2023. URL
4.2.1
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. semnoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3 11, 2018. A.1.1
Todorov Emanuel, Frez Tom, and Tassa Yuval. MuJoCo: A physics engine for model-based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, page 5026, 10 2012. doi: 10.1109/ tros.2012.6386100 4.1
Michael Fairbank and Eduardo Alonso. Value-gradient learning. In The 2012 International Joint Conference on Neural Networks (IJCNN), 2012. doi: 10.1109/JJCNN.200 22225.2211..625299.1
13

---

Published in Transactions on Machine Learning Research (4/2025)
Vladimir Feinberg, Alvin Wan, lon Stoica, Michael Jordan, Joseph E Gonzalez, and Sergey Levine. Model- based value estimation for efficient model-free reinforcement learning. arXiv preprinl arXiv: 1a::1803000 008000101 2018. 1, 2.4, 2.4,5 5
Bernd Frauenknecht, Artur Fisele, Devdutt Subhasish, Friedrich Solowjow, and Sebastian Trimpe. Trust the model where it trusts itself: Model-based actor-critic with uncertainty-awan rollout adaption. In Proceedings of the 41st International Conference on Machine Learning, ICML'24, pages 13973-14005. JMLR.org, 2024. 1, 4.1, 4.2, 1,
Matthicu Gcist, Bruno Scherrer, and Olivier Pictquin. A theory of regularized markov decision processes. In International Conference on Machine Learning, volume 97, pages 2160 2169. PMLR, 2019. 2.1
Tuomas Haarnoja, Aurick Zhou, Picter Abbecl, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870 PMLR, 2018. 2.3, 3,
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1rXi::191..011 00000, 2019. 1, 2.2, 2.4
Danijar Hafner, Jurgis Pasukonis, Jimuny Ba, and Timothy Lillicrap. Mastering diverse domains through world models. orXiv preprint arXiv:2301.04104 2023. 4.2, 3,
Assal Hallak and Shie Mannor. Consistent on-line off-policy evaluation. In International Conference on Machine Learning, pages 1372 1383. PMLR, 2017. 1,
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In International Conference on Learning Representations, 2024. URL https://openrevier:
2.2, 4.2, 3, B
Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning, pages 8387-8406. PMLR, 2022. 2.2
Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap, Yuval Tassa, and Tom Frez. Learning continu- control policies by stochastic value gradients. In Proceedings of the 29th International Conference Newrul Information Processing Systems Volume 2, NIPS'15, pages 2944 2952. MIT Press, 2015. 1
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout Q-functions for doubly efficient reinforcement learning, In International Conference Learning Repre- sentations, 2022. URL
1,
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Proceedings of the 33rd International Conference on Newral Information Processing Systems, pages 12519-12530, Curran Associates Inc., 2019. 1, 2.2, 2.3, 2.6, 4.1, 4.2, 5, A.1.1,
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 1, 2.4
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Newral Information Processing Systems, NIPS'17, pages 6405 6416. Curran Associates Inc., 2017. 4.2
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. arXiv preprint arXiv:1904.08/77 2019. 1,
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreviie nett/forrum2id--ggB......
Volodymyy Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):5299555 533, 2015. doi: https://doi.og: /t.p.//11...00.. 2.3
14

---

Published in Transactions on Machine Learning Research (4/2025)
Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. PIPPS: Flexible model-based policy search robust to the curse of chaos. In International Conference on Machine Learning, pages 4065-4074. PMLR, 2018. 2.4, 5
Luis Pineda, Brandon Amos, Amy Zhang, Nathan O. Lambert, and Roberto Calandra. MBRL-Lib: A modular library for model-based reinforcement learning. arXiv preprint arXiv:2104. 10159, 2021. B, D.4
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deter- ministic policy gradient algorithms. In Eric P Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 387-398 395- PMLR, 2014. 1, 2.5
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Nla Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 1929 1958, 2014. A.1.1
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximaling dynamic programming. In Machine Learning Proceedings 1990, pages 216 224.. Elsevier, 1990. 5
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 13th International Conference on Neural Information Processing Systems, NIPS'99, pages 1057-1063. MIT Press, 1999.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca Cola, Tristan Deleu, Manuel Andrea Pierré, Sander Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Gymnasium, 2023. URL https: Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. www.:d/or/rrreere/eeee//e /8127025. 1, 4.1
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Mcrel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. doi: https://di..rrg/ http:///doi..rg11.11//////e/// 10..1016)ssimma2222....0.
4.1
Shangtong Zhang, Bo Liu, Hengshuai Yao, and Shimon Whiteson. Provably convergent two-timescale off- policy actor-critic with function approximation. In International Conference on Machine Learning, pages 11204-11213. PMLR, 2020.
Shenao Zhang, Boyi Liu, Zhaoran Wang, and Tuo Zhao. Model-based reparameterization policy gradient methods: Theory and practical algorithms. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS'23, pages 68391-68419. Curran Associates Inc., 2023. 1, 2.5, 4.2,
B
A Implementation Detail
The neural network architectures for the dynamics and reward model, well as for the actor and critics, are described in Section A.1. Regarding the dynamics and reward model, Section A.2 explains how the model is used for prediction, while Section A.3 details its training. Hyperparameters are provided in Section A.4. Finally, Section A.5 describes techniques for achieving stable critic predictions.
A.1 Neural network models
A.1.1 Dynamics and reward model
Following Janner et al. (2019), we employ bootstrap ensemble of M = 8 models,
(p " PPria }
15

---

Published in Transactions on Machine Learning Research (4/2025)
We refer to them deep ensemble (DE) models. As discussed in Section A.3, each ensemble member is trained on (virtually) independent bootstrapped datasets. The ensemble models capture aleatoric noise by fitting independent multivariate Gaussian distributions.
Concretely, the m-th model Pvm (s, a) takes a state-action pair x = (s,a) as imput and outputs the mean vector (m E SR$1 of a multivariate Gaussian distribution. Here, S is the dimension of the contimuous state space, and we jointly predict the next state (S dimensions) plus the reward (1 dimension), hence the total output dimension S ++ 1. For the covariance, assume homoscedastic noise, meaning that each model's diagonal covariance elements om R are constant with respect to the input.
We use Layer Normalization (Ba et al., 2016) and Dropout (Srivastava et al., 2014) for regularization, and apply SiLU (Elfwing et al., 2018) as the activation function.
Below PyTorch-like summary of our network architecture:
(model): Sequential(
(0): Busemble@ineertaayer
in_ out_eize=256, decay=0.00025)
(1): (pm0.
(2) S110()
(3): LayerNorn (yyerNorn(((266)) ((26,))
(4): Busemblelinnerriayerr
in_size-256, out_eize=256, decay=0. 0005)
(5): (p-p.ut(p-OO0000
(6): SilD()
(7): LayerNorra
(8) Rusemblelinearrtayer
in_eise-256, out_eize=256, 0000)))
(9): Dropout (p=0.ut(==000025
(10): SiLU()
(11): LayerNoza((256,)
Ensemblolimesrlayer
8, 1n_size=256, out, siz_sizen,+,, decays0.001)
EnsemblelLnnearLaaer" denotes the layer which calculates the feed forward pass in parallel.
A.1.2 Actor and Critics
We also use ensemble MLP as critic networks used for randomized ensemble double Q-learning, or RED- Q (Chen al., 2020). We summarize the neural network architecture using Pytorch-like notation:
for model,
(critics): Sequeatial(
(0): Busomblelineerlayyr(
6, in, in_size=S+A, out, size=112, dacay=0.0)
Dpopout(p=D..00)
(2): S110()
(3) Layartiorn
(4): RusemblelLnearlaaer
5, in_eize=S12, out_size=612, decay=0.0)
Oropouz(p=0.0000 (6): stin()
(verl)orn((256,))
(8): BusembleLinearrayar
5, in_size==12, out_ out_size-512, decay=0.0)
Dropout(p=0.00011 (10): SiLU()
(11) (2266))rr(((666))
(12): Ensemblal.tneerLayerr
5, in_eize=512, out_eize=l, decay=0.0)
for critic networks, and
16

---

Published in Transactions on Machine Learning Research (4/2025)
(actor): Sequential(
(0): Linear(in,
featureasS, (inarrti,_fatuueers,
featuresm512)
(1): S1LU()
(2): Linear Linear(in_faaurreee fo1tuurstaa out, featuureee-))1
(3): S1LU()
(4): Linear(ia_featureeee out (et_uuerur))))1)
(5): SiLU()
(6): Linoar(ta__faauurrrr5h. out featuuerr))))2)
for actor networks, respectively, where is the dimension of state space, and A the dimension of action space. The total number of weight parameters, for example Humanoid task from GYM suite is about 2M,
where 4-45 and A-17.
A.2 Model prediction
We propose to normalize the signals targeted for regression (see Section A.3 for more detail). Given variable 2, we denote its normalized version by Σ. We perform normalization by the empirical mean μz and standard deviation σN statistics of the variable with the equation
محك Similarly, we can perform denormalization s' E Rd respectively, and
by 20, + Ma. We denote the state vectors and the next-state vectors the i-th element of the displacement between them as A; == s': 8,. Our dynamics models, T are trained to predict the displacements in states from one step to the next, thus we normalize the oulputs based on their used targets. Let μAd, and da, be the mean and standard deviation of the observed displacements in the data up to now. During training the model, the i-th element the model's target signal for the n-th data point is defined tni = Ani-HA, We summarize the different normalizations we perform: all inputs EA;
to models are normalized; the rewards are normalized during training the reward model, but the predicted rewards are denormalized during policy learning; the actions are never normalized (as we choose their scale to be reasonable); the transition model targets normalized during model training and also during the actor training rollouts.
The rollout predictions, in particular, require care in the implementation. Suppose the normalized state is 5 and the transition model has predicted a change A. We can add the normalized displacement to the normalized state by first denormalizing, then adding them, then normalizing them again. This leads μs + aAA, (iii) s' 8 A μs + 0,5 + μa oΛ (iv) to We thus that to make correct predictions in the normalized space using
the process: (i) 8 lls + ag5, (ii)
= d' displacement model, we need to denormalize the displacements, then normalize them using the normalization scale of the states, 8, and add the displacement to the normalized state, 3.
de
Note that although a similar normalization was performed in SAC-SVG(H), the output was normalized using the sample mean and standard deviation of the state, just like the input.
When predicting next-state and reward during rollout, we adopt the TS-1 algorithm proposed in Chua el al. (2018), which randomly selects ensemble member for each input and timestep in order to propagate uncertainty through bootstrapping.
A.3
Model Learning
A.3.1
The objective function
The objective of the model learning is minimizing the sum of the negative log-likelihood:
M
N
C(0) = MND
Wmen Cyn
m=ln= n=1
where
D
(nn Hmadd 2 Omd
Lmn =
Σ
log and +
17

---

Published in Transactions on Machine Learning Research (4/2025)
Pmnnd Pat (In)) represents the dth outpul of the mth neural network pe for the nth imput 2n == (sn)an); and mmd denotes the noise regarding the dth dimension output of the mth ensemble member dm.
(d)
Wmn E [0, oo] is weight for nth sample allocated to the mth model, in order to make the dataset virtually independent bootstrapped one. The value is randomly sampled from the exponential distribution [(w(1] 1) with the parameter À 1 and fixed, thus the effective number of data used for each model's learning is oquivalent on average.
A.3.2 Early stopping
When In is the imput, let Amn and om represent the mean and scale paramcter, of the mth model's predicted Gaussian distribution. We aggregate the predictions across all ensemble members, and define the prediction of the DE model as a Gaussian distribution with mean parameters for the dth dimension when the input is Fn as And Mi Em=1 Pmnd and the scale paramcter as o = snd + -m=1 o md: where s is the sample variance among the ensemble of predictions {Ummd] This approach is often also called moment matching.
Then, we define the evaluation score Sn, for the nih data as:
D
S =
Σ
(und - lnd)²
+ log and
0nd
d=1
under the assumption that the predicted distribution follows a Gaussian distribution with diagonal co- variance matrix. Afler each epoch of model learning, we evaluate the score for each validation data point and compare the distribution of these scores based on the parameters of the model with those from the previous parameters. If the distribution of the new scores shows significant improvement according to a Z-lest with a significance level of 0.1, we continue training the model.
Our stopping strategy the same as in the MBPO implementation: the new model does not show improvement over fixed number of epochs, the training stopped. We set the threshold as b bog da, where is a scalar-valued syper-parameten representing the base number of epochs, and d, the dimension of the state space.
A.4 Hyper- papee-pprameters
We summarize in Table hyper- pepre-parrmeters shared for all experiments in Section 4, for DHMBPÓ and SAC. Since the DHMBPO algorithm shares common modules with the SAC implementation, such the architecture of the critic network, the hyperparameters and their values also identical. Exceptions to this are explicitly indicated in parentheses.
A.5 Bounding target critic's output
In this study,
the MVE estimator for Q-function, eq. (9) the target values in critic learning. How- ever, due to the recursive dynamics model predictions, the model errors may accumulate value estimation errors may increase (Buckman et al., 2018). In order to avoid this risk, we bound the values predicted by the target critics to stabilize critic learning.
Since the Q-value is within + times the upper and lower bounds the reward value (Proof. E T s En 7 r -_ and similar for the lower bound), we approximate these upper and lower reward bounds. The procedure for estimation of bounds on QT is as follows: (1) Perform training rollouts. (2) Set the 1-st and 99-th percentiles of rewards from the rollouts as r and ru, respectively.
(3) Qi 1 and Qu = [ (4) When calculating the bounds for the first time, set Qi and Qn as the initial values, and subsequently update them by taking exponential moving averages with a hyperparameter n. Then we clip the target critic networks' outputs in eq. (9) by Qi and Qu. Since the process simple, its computation cost is negligible.
18

---

Published in Transactions on Machine Learning Research (4/2025)
Hyperparameters commonly set for DHMBPО and SAC across all experiments. The first half of the table presents the hyperparamcters shared between DHMBPO and SAC.
Table 2:
Hyper-parameter
Value
Discount factor
0.995
Seed steps
5000
Action repeat
1 (Gym)
2 (DMControl)
Batch size
256
Update-to-data ratio
1 1M
Replay buffer size
Learning rate for the actor, critics and Initial value of a Momentum coefficient с for target critic Ensemble size of critie
10-1
0.1
0.995
5
Length of DR D Length of training rollout T
20
5
Iteration per DR Ensemble size of model Optimizer for training model Learning rate for model
20
8
AdamW (Loshchilov and Hutter, 2019)
1 10-3
We do not perform, on the other hand, clipping on the output of the critic used when training actor with eq. (7). This is because if the critic network outputs a value outside the bounds, the gradient to encourage prodicting values within that range would vanish because of the clipping.
The motivation for using percentiles in the step (2), instead of using hard max and min, is based on the observation in our preliminary experiments (Appendix D.7). There occasional large outliers in the rewards that would make the bounds extremely broad. Ignoring these outliers, but estimating the bounds based on the most of the samples gives more efficient and stable results.
Experiment Details
We describe configuration for comparison in Section 4.2. In this study, when the other algorithms' experi- mental results under idantical seltings were available, we used those results as part the results (MBPO and Dreamer v3). Instead of using the official implementation by Haarnoja et al. (2018), we utilized results based own implementation of the SAC algorithm. This approach allows for high degree of implementation commonality between SAC and DHMBPO, except for the components involving the model. Specifically, described in Appendix A.4, we shared various implementation aspects, including the architecture of the critic network. For the rest of the results, we ran the publicly available source code with its default hyperparameter settings.
The number of samples measure test return varies depending on the implementation of the algorithm. Like DHMBPO, SAC-SVG(H) (Amos et al., 2021), RP-PGM (Zhang al., 2023), SAC (Haarnoja et al., 2018), and TD-MPC2 (Hansen al., 2024) used 10 test episodes, while MACURA (Frauenknecht et al., 2024) used 3 test episodes and MBPO (Janner et al., 2019; Pineda et al, 2021) and Dreamer v3 (Hafner et al., 2023) used 1 test episode.
For each GYM tasks, except for DHMBPO, each algorithms' hyper-parameters were tuned per-task. Follow- ing (Janner et al., 2019), we used variants for the Humanoid and Ant environments from GYM, with some state dimensions truncated to lower dimensions.
For DMC tasks, Each of three algorithms' paraeeess set commonly over all DMC tasks.
19

---

Published in Transactions on Machine Learning Research (4/2025)
Normalize lizd IQM Store
L.00
1.00
Fraction Runs >
0.75
0.755
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.25
0.50
0.75
0.25
0.5
0.75
Normalized Environment Steps
Normalized Test Return (72)
DHMBPO w/o
DHMBPP w/o TR
DHMBPО
Figure 7: Performance profile (left) and score distributions (right) of ablated algorithms 5 GYM tasks and 5 DMC tasks.
C Result Details
C.1 Ablation study
Figure shows performance profile and score distributions of ablated algorithms on GYM tasks and DMC tasks, of which individual plots are shown in Section 4.3. We see that DHMBPO's statistically significant sample efficiency in early phase against the ablated algorithms.
C.2 Comparison with other algorithms
For DMC Lasks, we provide plots for individual sample efficiency curve Figure 8, plots for runtime Figure 9 and Table 3 for summarizing their runtimes. For reference, we include horizontal lines indicating the asymptotic performance of SAC. These lines represent the average test return across random seeds, ten times the maximum environment step budget used for the model-based methods in each task.
20

---

Published in Transactions on Machine Learning Research (4/2025)
aerobot-swingap
arrohot-swingug www.ooswwwwggpppp....
cartpole- cartpole-ullanee bularzee
curtpole curtpole-wwingup swingup
1000
800
180
Test retumn
800
600
300
611)
400
150
200
200
Environment stepe
Environment stoge
Environment steps
Environment steps
cazl swl-pgs-swiggu sparse
cheetah-ruu
cup-cateh
finger-spin
1000
Tett ceturs
750
750-
760
750
500
300
250
250
250
2.0
Environmestt steps
Envtronment stepe
Envtronment steps
Environment steps
finger-turu caaayy
Engerr-turn
hopperr-har hagp
hopperr-taan
1000
1000
750
450
750
Tost ietann
500
500-
300
1000
250
160
0
01
0
Environmest steps
X10
X30
steps
X1U
steps
Jomenoid-staue
guadruped-run
reacher-bard
walleer-run
1000
600
750
Test 205um
450
3010
300
4000
250
200
01
Environmmee
X1CP
Environment
1010"
Enviromment
x10
Envirunment
X.10P
walllerr--taaa
wallserr-wald walk
1000
750-
Text returo
750
500
300
Envirommeet steps
Eavironment stepe
DHMBPO
TD-MPC2
SACB& strpe
Figure 8: Comparison of sample efficiency with latent model-based methods DMC tasks. The solid line represents the sample mean over 8 random seeds, and the shaded area indicates the 95% confidence intervals.
21

---

Published in Transactions on Machine Learning Research (4/2025)
acrobot-swe ingup
acrobot-swingg www.ootttwiinuspppppp...
cartpole-balance
cartpoleewiingup
1000:
800
Tea: teturr
150+
600
100
6U0
400
1.0
50-
400
3.0
8.0
0.0
3.0
6.0
0.5
1.5
0.6
1.2
1.8
Rantimm [Hours]]
Ranniime [Houus]]
Rontime [Hours)
Runtlame [Houss]
cars cartpole-ewngup sparse
cheetal--ru
cup-catec
Einer-spii
1000
750
150
Test reare
750-
750
300
500
500
250
250
0
01
0.0
01.8
1.6
2.4
1.2
4.0
0..0
1.5
3.0
4.5
6.0
0.0
0.4
01.8
1.2
1.6
20
0.0
0.8
1.6
244
322
4.0
Rantime [Hours]
Rantimm: [Hours]
Rantiim [Hours]
Rantlime [Houss]
Hinger-turn.ce
finges-tum_hard
hopper-hop
hopper-stand
10000
1000
460
750
Tost ictiom
500
500
250-
000
0.5
2.3
0.2
0.0
1.5
23
4.
3.0 Runtime [Hours]
300 Runtime [Hous]]
Runtine [Boors]]
Rundinee [Hours)
Jumaanokd-stand
quadruped-run
reacber-hard
walker-run
1000
6000
750
Test rec0rs
300
500
300
250
200
0
Rantine [Hours]
Rantime [Houre]]
Rantime [Hours]
Rantllme [Hous]]
wallsen-staadd
walkerrwallk
1000
750-
Toxt ecturoo
75th
500
250
0.0
0.3
12
0.11
10.8
20
Runtime [Nlorrs]
Renntime [[our))
DHMBPO
TD-MPQ2
Figure 9: Runtime comparison with latent model-based methods DMC tasks.
22

---

Published in Transactions on Machine Learning Research (4/2025)
Table 3: Runtimes on DMC tasks until maximum environment step budget (250K or 500K) and its ratio to DHMBPO's mcan runtime (in hours).
DHMBPO Dreamer v3
TD-MPC2
acrobot-swingup
1.7
2.9
3.8
acrobot-swingup sparse
1.6
2.9
2.6
cartpole-balance
0.9
1.5
1.2
cartpole-swingup
0.9
2.9
1.9
cartpole-svinguu_ssaaae
1.9
2.9
2.3
cheetah-run
2.7
2.8
cup-catch
1.2
1.6
1.9
finger-spin
2.2
3.2
2.9
finger-turn easy
2.2
3.2
2.7
finer-turn_haat
2.2
3.2
2.6
hopper-hop
2.7
3.1
3.8
hopper-stand
3.2
3.3
2.3
humanoid-stand
4.3
2.9
5.0
quadruped-run
3.7
3.2
2.7
reacher-hard
2.0
2.9
3.7
walker-run
2.7
3.2
3.9
walker-stand
1.2
1.6
1.8
walker-walk
1.7
1.5
3.7
Ratio
1.0
1.3
1.3
23

---

Published in Transactions on Machine Learning Research (4/2025)
C.3 pyper-parameter sensitivity
In Figure 10, we show individual results of experiments in Section 4.4 with respect to sample elficiency and to Euclidean norm of policy gradient 0 E RR: VE:
D 02.
24

---

Published in Transactions on Machine Learning Research (4/2025)
Hopper
Humonoid
Waller2d
4000
100%
6000
7200
71900
0,00
2000)
50000
2500
01
Envirotmnen skope
Esviroutennt stcpp
Eevinoament abopa
Enviocuanto stope
Eaviooument stopp
cou' cortpollo-wwnggg.pppp wppploowwwwggpppppr....
quddruped-rum
hurmmnidd-staad
hogperr-ttand
fingr-ttannJaae
470 1000
1000
Tox tctuEs
500
176
200
Enelonaenn stope
x16
Eastroument stepo xt0
Eavbounet steps
Environmen stage
X1M
Eustrcument stopo
(a) Sample efficiency curves.
Aui
HolCCheetal:
Hopper
Humanood
Walleer2d
Noma prabent
107
309
12a.
LOF
101
10p
201
19h
10
Euviromment ataps
Emrinennanti stope
Esvinuame strze
Eaviroement stappe
Euvinoment stepe
cortpoleswinggp..o
quadreque-rrn
hummancdd-ssacl
heppee-sandd
Gugee-turn Jari
Neen of poley graiierr
207
13a
10
100
100
10
1000
10-1
19..
10
107
10-1
10P
10
2
10
K13
xld*
111
<33
Euviromentt
100
(b) Norm of policy gradient.
Figure 10: Development of test return and critic loss environment steps for different TR lengths
25

---

Published in Transactions on Machine Learning Research (4/2025)
D Additional Experiments for Highlighting improvement over DR-based and TR-based algorithms
D.1 DHMBPO without Training Rollout but with different UTD ratios
We show the results of DHMBPO without TR but with different UTD ratios in Figure for sample efficiency curves, in Figure 13 for performance profiles, in Figure 12 for runtime comparison and in Table 4 for summary of runtimes. We
that the higher UTD ratio the higher performance. Conversely, DR-based algorithm, DHMBPO without TR, needs higher UTD ratio in order to attain high sample efficiency, requiring longer runtime shown in 12. And this results reflect that MBPO (and MACURA) were sample efficient while took so long runtime.
D.2 Trade-off in Policy Gradient Estimation
We investigate how the horizon length of the training rollout (TR) affects the bias and variance value gradient estimates. In principle, increasing 7 reduces the bias by incorporating longer reward sequences, but it can simultaneously raise the variance of the policy gradient.
Experimental Setup
For each benchmark lask, we train DHMBPO agent for 500k euvironment. steps and then fix its policy, critic, dynamics, and reward networks, as well as the replay buffer. We draw B 2048 states { b- at random from the replay buffer. For each state 8 and each TR horizon {1,3, 7, 9}, we run R = 2048 moori-h rollouts under the learned model and compute the corresponding MVE-based value gradients {9ar 1 1==1" where Stbr is a normalized vector by the dimension of policy parameter which differs across tasks.
B
We define
R
B
Gtb =
Othrs
gt
9tb-
b=1
Following the approach in our hyperparameter sensitivity analysis (Appendix C.3), we designate t = =9 =:: as the ground truth horizon:
g 9T-
In other words, we treat the gradient computed at 9 as a baseline for evaluating shorter horizons.
Table 4: Runtime until 500K environment step. the case with UTD 1.
Tha last row shows the ratio of mean runtime that of
The UTD ratio
8
16
Ant
3.8
5.1
7.9
13.7
27.0
HalfCheetah
3.0
4.4
6.6
11.8
23.2
Hopper
3.3
4.7
7.5
12.6
26.5
Humanoid
4.6
6.8
9.7
15.9
28.1
Walker2d
3.6
5.2
8.7
14.5
26.7
cartpole-swingup spaase
1.7
2.3
3.7
6.3
12.2
finger-turn hard
2.3
2.3
3.5
6.3
12.9
hopper-stand
2.1
2.6
4.5
6.8
11.2
humanoid-stand
3.7
4.5
5.8
8.0
14.6
quadruped-run
3.8
4.2
5.4
7.7
11.9
Ratio
1.0
1.3
2.0
3.3
6.1
26

---

Published in Transactions on Machine Learning Research (4/2025)
HosCCh
Hogpost
Humanoid
Wallecdd
10000
3700
Tol. veluuo
3000
200M
1500
000
stepe
Restronmnnt strge
staye
X105
Eastormmnt stepe
strge
cartpoloewrnngrp-pprr
quddraped-m
Immanciddstann
hopper-tand
finge-turmJuee
1000
100
T0A
134
500
224
25
250
Eavincnnate sicgo
Eaviroamastt andpe
Resteonsaani stapps
Enrivomarn shopa
UTD
OTD
Figure 11: Sample efficiency curves of DHMBPO without TR (corresponding to MBPO) but with different UTD ratios.
HelCChe
Hoppae
Humanoid
Wallerdd
4200
1200
3500
1400)
8000
Tot retum
14320
2100
3000
3332
500
2000
15
15
10 Rantin: Hour)
Rastin: [Boun]
Rusline (BAAn)
Rantine [Romm]]
Runlinx [Baza]]
quadeopeeoemn
humennccs-sttul
hoppes-Land
fingr-turnJnnl
1033
7741
100
750-
75
010
330
2000
1.0
2.1
33
735
10.0
125
00
23
0.00
25
37
9.5
150
310
235 500
7.6
100 125
Tanrinm [Bound]
Raztine (Blan)
Rannine [Romn]
Runtine [Boua]]
Tantine Bounj
UID
UTD -2
UTD
UTD
UID
Figure 12: Runtime comparison of the DHMBPO variants without TR but with different UTD ratios.
To quantify bias, we measure the mean squared difference bet ween each sample-averaged gradient geb and g. averaged over all b:
Bias; = > 9rb g||².
1
6=1
In addition, we compute the standard error of the mean (SEM) to capture the variability of the gradient across states in the replay buffer:
2
1= 9lb 9t
SEM, =
B 1)
where || denotes the Euclidean norm over the policy parameter dimension (i.c., the gradient dimension). Intuitively, Bias, reflects how far the sample-averagel gradient al horizon / is from the ground truth al. while SEM, indicates how much variability remains across different states.
Experimental Results
Figure 14 presents results using T 5,7, and 9 as the ground-truth horizon. In all tasks except finger-turn hard, see that increases from 1 to 9, the bias steadily declines while the SEM
27

---

Published in Transactions on Machine Learning Research (4/2025)
Normalizod IQM Score
1.6
Fraction of Rums >> It
1.00
1.2
0.75
0.50
0.4
0.25
0.0
0.00
0.00
0.25
0.50
0.75
1.00
0.0
0.25
Normalized Environment Steps
Normalized Test Return (7)
UTD
CTD
UTD
UTD
Figure 13: Performance profiles and score distributions of DHMBPO without TR (corrcsponding to MBPO) but with different UTD ratios. The baseline here is the variant with UTD 1.
grows revealing a clear trade-off between bias and variance in the policy gradient. By contrast, in finger-turn, hard, the bias when T-9 did not show clear trend to decrease, suggesting the batch size of 2048 was not enough for 9T=9 to be as ground truth, and that setting f = 5 does not necessarily prevent gradient explosion in certain challenging domains.
Notably, all these estimates rely on replay buffer of 2048 reviromnt states, while the multiple trajectories at each state generated by the model, although the model trained after 500k samples. Therefore, this method can be carried out without any additional real-enviromment interaction and used to set relatively stable length of TR horizon, demonstrated in unclear trend of the bias in f finger-turn, hard.
D.3 Application of DR to SAC-SVG(H) Implementation
In order to see the effect of DR on an existing implementation, we added DR procedure into the official code of SAC-SVG(H) algorithm (Amos et al., 2021), which employs the SVG method with single GRU (Cho, 2014) as the recurrent neural network and one-step TD errors for training the critic networks. We refer to this variant as DHSAC-SVG(H) and evaluated it on five GYM tasks with varying DR lengths, repeating the execution with different seeds. As shown in Fig. 15, the variants with longer DRs exhibited either poor learning performance or drop in performance after certain point. We investigated the training metrics and found that the critic losses started to explode the DR length increased, shown in Fig. 16. Here, the solid line is mean a and shaded area shows the 95% Cls in log-scale: s(1 + exp(log SEM 1.96)) where
log SEM is defined as log SEM = V -. and l, is critic loss with s-th seed. S(S-1)
One potential cause these results is the deterministic nature of the model used in the SAC-SVG(H) algorithm. Predictions generated by a fixed deterministic model are likely to introduce consistent biases. Tn contrast, the DE model approach adopted by DHMBPO and MBPO uses multiple ensemble members, cach potentially having similar biases. However, the manner in which these biases accumulate random across ensemble members. During predictions, ensemble member is randomly selected at each timestep, making the bias accumulation in the predicted states also random. As a result, within single trajectory, these biases may cancel out to some extent, leading to an overall smaller average bias, albeit with larger variance.
Policy evaluation and improvement based on the higher-vate variance predictions a DE model may slow down learning, but they avoid the risk of extreme predictions caused by deterministic models with large biases. On the other hand, policy evaluation and optimization using predictions with significant bias from deterministic models carry the risk of focusing extreme values, potentially destabilizing the training process.
28

---

Published in Transactions on Machine Learning Research (4/2025)
D.4 Comparison of Model Learning Performance
We also conducted quantitative analysis lo compare the predictive performance of models used in DHMBPO, SAC-SVG(H), and MBPO (Pineda et al., 2021), using common training and evaluation datasets.
Data Generation Procedure For each GYM lask, we ran the DHMBPO algorithm for up to 500K steps. From the replay buffer, we created three input-output datasets containing data collected up to 5K, 20K, and 50K steps, respectively. Additionally, we created another dataset from all data collected between 495K and 500K steps. Each dataset was split. into training (80%) and validation (20%) subsets, while the 495K 500K dataset exclusively used for evaluation.
Evaluation Methodology For each of the three models, we trained them using the respective train- ing datasets and measured Root Mean Squared Error (RMSE) on the training, validation, and evaluation datasets at each epoch. This procedure was repeated for three random seeds.
The results of the experiments are plotted separately for each dataset size, comparing training data versus validation data and training data versus evaluation data, as shown in Figs. 17 19. Additionally, the results after 100K epochs are summarized in Table The shaded areas in the plots and the values the table both represent the standard error three seeds.
Results
29

---

Published in Transactions on Machine Learning Research (4/2025)
Table 5: Summary of RMSE values after 100K epochs. "Size" denotes the size of training data.
Method
DHMBPO
MBPO
SAC-SVG(H)
Metric
Task
Size
Training RMSE
Ant
5000
0.29 + 0.0
1.86 0.02
1.1 + 0.01
20000
0.48 + 0.0
1.21 + 0.01
0.67 + 0.0
50000
0.0
0.79 0.02
0.59 0.0
HalfCheetah
5000
0.18 = 0.0
1.46 = 0.01
0.01
20000
0.35 = 0.0
0.73 = 0.01
0.57 = 0.0
50000
0.49 1 0.0
1,58 1 0.08
0.76 0.0
Hopper
5000
0.01 1 0.0
0.06 0.0
0.11 0.0
20000
0.02 + 0.0
0.09 0.0
0.18 0.0
50000
0.02 + 0.0
0.04 + 0.0
0.21 + 0.0
Humanoid
5000
0.44 = 0.0
2.02
2.99 0.02
20000
1.18 = 0.0
1.99 = 0.0
1.65 = 0.0
50000
1.55 = 0.0
0.01
1.69
Walker2d
5000
0.16 1 0.0
1.74 1: 0.08
2.88 0.01
20000
0.36 1 0.0
11 0.01
1.48 0.03
50000
0.43 + 0.0
0.72 = 0.02
0.92 0.0
Validation RMSE Ant
5000
6.44 = 0.03
4.29 0.12
8.59 0.07
20000
1.51 = 0.02
1.62 = 0.02
3.64 0.03
50000
0.93 = 0.01
1.07 = 0.01
1.43 0.01
HalfCheetah
5000
3.72 + 0.01
3.82 0.03
8.78 0.01
20000
1.99 + 0.02
1.87 + 0.03
3.16 0.05
50000
1.61 + 0.03
1.97 + 0.11
3.9 = 0.14
Hopper
5000
0.19 = 0.01
0.16 = 0.01
0.25 - 0.0
20000
0.25 = 0.02
0.2 0.0
0.95 0.03
50000
0.13 1 0.01
0.1 0.01
1.15 1 0.02
Humanoid
5000
5.14 1 0.06
5.09 0.02
7.9 = 0.1
20000
3.25 + 0.04
3.49 + 0.02
9.25 + 0.08
50000
2.66 + 0.01
0.02
0.0
Walker2d
5000
4.08 = 0.06
4.54 = 0.16
6.04 0.04
20000
2.69 = 0.02
2.79 = 0.08
7.22 = 0.06
50000
1.46 = 0.02
1.54 = 0.03
2.49 0.02
Evaluation RMSE Ant
5000
8.61 1 0.21
6.45 0.18
10.9 0.22
20000
2.49 1 0.06
2.05 1 0.05
6.29 0.11
50000
1.13 + 0.01
1.25 + 0.01
2.61 + 0.02
HalfCheetah
5000
21.11 0.24
274.94 51.26
25.31 + 1.15
20000
12.76 - 0.19
18.12 + 1.23
16.56 = 0.35
50000
5.89 = 0.06
27.91
9.51 0.23
Hopper
5000
1.97 = 0.01
20.67 = 1.34
2.05 = 0.03
20000
0.99 1: 0.04
0.19
0.13
50000
0.4 1 0.0
0.62 1 0.16
0.88 1 0.01
Humanoid
5000
12.91 0.18
14.25 = 0.54
17.22 - 0.69
20000
12.64 + 0.21
14.63 + 0.19
13.24 - 0.46
50000
11.47 1 0.18
11.46 0.17
12.86 1 0.14
Walker2d
5000
11.27 0.28
251.58 - 54.54
13.91 + 0.59
20000
11.37 0.49
16.42 + 2.58
12.38 + 0.76
50000
8.89 + 0.18
22.69 + 10.32
9.72 0.3
30

---

Published in Transactions on Machine Learning Research (4/2025)
Ant
HalfCheetah
Hopper
Humanoid
Walker2d
(0.04)
(0.11)
(0.01)
(0.04)
(0.05)
x10-1
x10
x10-3
x10
5.0
2.0
2.
4.5-
0.5
1.5
3.0
1.5
Horison
Horison
Horison humanoid-sttad
Horisca
Horizon
cartpole-swiggup.ppr papaaeee
quadrupedzzun (0.01)
hopper-stand (0.01)
finger- burn hard (0.13)
(0.01)
(0.00)
x10
x10
x10
5
x10
x10
110
x10
x10
0.8
1.6
1.2
7.0
0.5
1.2
3,0
2.4
1.6
2.000175
0.50
1.00
7.5
0.4
0.4
0.5
1,750.25
0.0
6.5
0.0
0.0
0.00
Horizon
Horison
Horizon
Horison
Harison
(Left)
Standard etvor meam (Right)
(a) T=
Ant (0.04)
HalfCheetah
Hopper
Humanoid
Walker2d
(0.11)
(0.01)
(0.04)
(0.05)
x10-
x10=3
x10-
x10-
x10
-10-3
x10-2
-10-1
x10
6:22
6.0
6.0
4.5
2.0
0.5
1.2
1.00 1.2
3.0
0.8-
7.5
2.5
1.5
1.5
0.75 0.4
0.0
0.0
0.0g
5.0
Horizon
Horioon
Horizon
Horizca hopper-stand (0.01)
Horizon
nattpol-swingup.ppeee
quadruped-run
humanoid-staan -tannd (0.00)
(0.01)
(0.01)
(0.13)
5
6
2-4
1.00
3.2
2.4
4.5
2.1
2.4
0.6
0.75 0.5
1.6
1.5
0.5
1.0
5.0
0.8
1.6
3
3
3
Horloom
Horison
Horizon
Horizon
Horison
Blas (Teft)
Standard ernor mean (Right)
(b) T-7
Ant (0.04)
HalfCheetah
Hopper
Humanoid
Walker2d
(0.11)
(0.01)
(0.04)
(0.05)
x10
x10
L.00 x10
x10
100
x10
10
x10
-10
x10
1.2
1.6
7.5 0.75
2.1
2.4
2.1
3.3
1.0
0.8
5.0 0.50
1.2
1.6
1.6
0.4
0.0
2.5 0.00
0.23
0.0
0.6
0.0
0.8
0.0
0.8
0.5
Horizon
Horison
Horizon hunranoid-stand (0.00)
Horizon hopper-stand (0.01)
Horizon
cartpole-swingup.ooo sspurs (0.01)
quedruped-run (0.01)
finger-tura bard
(0.13)
x10
x10 -
-1
x10
x10
x10
x10
x10
x10
6.0
2.0
3.0
4.5
3.0
1.5
7.5 0.75
5.0 0.50
7.6
1.5
1.0
3
1.5
1.3
0.23
0.8
5.0
0.5
0.0
0.0
2.5 0.00
3
0.0
0.00
0
Horison
Horison
Horizon
Horizon
Horizon
Bus (Teft)
Standard ereor meam (Right)
(c) T=9
Figure 14: Comparison of different ground-truth horizons. Each plot shows how the bias (left vertical axis) and SEM (right vertical axis) of the value gradient vary with the TR horizon t, using cither T = 5, 7, or 9 as the "ground truth" baseline (Ø). Across all three choices of ground-truth horizon, we observe consistent trend: the bias decreases with longer i, whereas the SEM increases.
31

---

Published in Transactions on Machine Learning Research (4/2025)
Hepyprr
1900
3000
3903
4500
The
2000
9X0
L013
3000
1500
Eoveooieeet stegs
Envtrommens stope
Enstronment strpe
EavioounCE steps
Enviromant stegs
SAC SVCIED)
Figure 15: Sample efficiency curve of DHSAC-SVG(H))
Ant
HallChectah:
Happer
Humanoid
Wallordd
107
10a
107
Onte:
1071.
1014
101t.
107
10
107
105
10
Eecinuamnt X132
Enstronment x10
Fovtooment
X107
Rovinement
X327
Fastronment X.P
Figure 16: Development of critic loss over the number of environment steps of DHSAC-SVG(H)), Solid lines is mean and shade are is for 95% Cls in log-scale. See the text in Appendix D.3 for the detail of the Cls.
Ant.
HalfCheetah
Hopper
Humanoid
Walker2d
10
10"
10"
10-1
10a
10a
10-2
10a
10
10a
10
10
10
10a
10a
10
10
10
10a
10
Iteration
Iteration
Iteration
SAC-SVG(H)
(a) Training Validation RMSE
Ant.
HalfCheetah
Hopper
Humanoid
Walker2d
101
10a
10a
10P
10"
10
10"
100
10
10
10a
10*
10
10
10a
10a
10
10P
10a
DIMBPO
MBPO
SAC-SVG(I)
(b) Evaluuttn
Figure 17: RMSE of model trained 5K environment steps dataset. Solid line is for training RMSE and dashdot line is for validation RMSE or evaluation RMSE
32

---

Published in Transactions on Machine Learning Research (4/2025)
Ant
HallCheetah
Hopper
Humanoid
Walker2d
10
10
10
10"
10
Iteration
Iteration
Iteration
Iteration
Iteration
DHМBPО
MBPO
SAC-SVG(H)
(a) Training Validation RMSE
Ant
HalfCheetah
Happer
Hurnaooid
Walker2d
10
10a
10P
10
10"
10"
10
10
10
10P
10a
10a
10a
10
10a
10a
10a
10P
10a
10a
10P
10a
Itenation
Itoration
Iteration
Iteration
Iteration
DHMBPO
MBPO
SAC-VG(())
(b) Daaiing-Evaauution
Figure 18: RMSE of model trained on 20K environment steps dataset. Solid line is for training RMSE and dashdot line is for validation RMSE or evaluation RMSE
Ant
HalfCheetah
Hopper
Humanoid
Walker2d
10a
10
10a
10
10
10P
10"
Iteraton
Iteration
Iteraion
Iteration
Iteration
DHMBPO
MBPO
SAC-SVG(H)
(a) Training Validation RMSE
Ant.
HalfCheetah
Happer
Humanoid
Walker2d
10a
10
10
10
10P
10
Iteeation
Iteration
Iteration
Iteration
Iteration
DHMBPO
MBPO
SAC-SVG(H)
(b) Training-Evaluattin
Figure 19: RMSE of model trained on 50K environment steps dataset. Solid line is for training RMSE and dashdot line is for validation RMSE or evaluation RMSE:
33

---

Published in Transactions on Machine Learning Research (4/2025)
D.5 Different UTD ratio
We executed DHMBPO with the different UTD ratios, shown in Figure 20. All value for each metrics are normalized by the value for the UTD ratio = 1.
D.6 Ablation study of model architecture online RL performance
We additionally conducted ablation study the network architecture of the dynamics and reward models to Figures 21 and 22. Specifically, we evaluated three variants: (DHMBPO w/o LayerNorm) model without LayerNorm, (DHMBPO w/o Dropout) model without Dropout, and (DHMBPO w/ ReLU) a model in which all activation functions were changed from SiLU to ReLU.
From the results in Figure 21, we observed that removing LayerNorm caused the critic loss to diverge and led to decline in policy performance. For the other two variants, the critic loss did not diverge; however, shown in the aggregation metrics in Figure 22, their performance significantly deteriorated.
34

---

Published in Transactions on Machine Learning Research (4/2025)
Normalized IQM Soore
Fraction of Runs >> It
0.9
0.75
0.6
0.50
0.25
0.0
0.00
0.00
0.25
0.50
0.75
1.00
0.0
0.25
0.5
0.75
1.0
Normalized Environment Steps
Normalized Test. Returrn (R)
8
(a) Profiling plot
Ant
HalfCheetah
Hopper
4000
7500
12000
Test return
3000
5000
9000
2000
6000
2500
3000
0
6
10
15
20
5
10
15
20
25
10
15
20
Runtime
[Hours]
Runtime [Hours]
Runtime [Hours]
Humanoid
Walker2d
12000
10000
Test retur
7500
6000
5000
2500
3000
0
15
20
10
15
20
25
Runtime [Hour]]
Runtime [Hour]]
(b) Runtime
Figure 20: Results on GYM tasks of DHMBPO with different UTD ratios. Legends are for the UTD ratios
35

---

Published in Transactions on Machine Learning Research (4/2025)
Ant
HallChestah
Hopper
Humanolé
Waller2d
8000
0004
1901
6500
9000
7000
6:00
5000
Text
4.03
2000
1191
1000
2500
Envinormant
XII
Enviromment
XUP
Revimement
512
Envinammee
XII
Envinomment XUP
cattpole-wingggpp.o sppaee
qualrupeed-ng
Iumnnnid
hopper
finger Liad
1300
900
450
Toat cdhazs
500
300
100
330
310
-000
stoge X10
stope
stege
strge
X1I
stope 100
DEMEPO Deep
DEBBIPO
DEMBPO BAL
(a) Sample efficiency
HallCheetah
Hopper
Humanodd
Wallorr34
Catte
LOP
X137
x10
x327
XIP
cartpole swingup spanne
quadruped rum
Iummancid stand
hopper stand
Einger turn hard.
1160
204
10
100
1(63
Contr
10
19a.
10
10%
Eusteoonmett 5:pps
Environconn stope
Eorboranct: stegs
Eavtroonest stops
Emvronnect stope
DHMBPPO
DUNISPO w/jo Dtmp
DIISSPO w/o LayreNore
DUMBPPO w/ RAUU
(b) Critic loss
Figure 21: Ablation study of model architecture on sample efficiency and critic loss. The graphs compare DHMBPO with three variants about model: (DHMBPO w/o LayerNorm) a model without LayerNorm, (DHMBPO w/o Dropout) a model without Dropout, and (DHMBPO w/ ReLU) model in which all activation functions were changed from SiLU to ReLU.
Modian
IQM
Mean
DHMBPO w/ ReLU
DHBMPO w//o LayerNorm
DHMBPO w/o Drop
DHMMPPO
0.8
0.9
1.0
0.88
0.96
1.04
0.80
0.88
0.96
1.04
Figure 22: Aggregation metrics about the final test return the ablation study of model architecture.
36

---

Published in Transactions on Machine Learning Research (4/2025)
D.7 Distribution of predicted rewards and target values for critic
We applied the MBPO algorithm HallCheetah using the MBRL-lib and created histograms of batch size rewards and critic losses calculated from the rollout buffer sampling every 10,000 interaction steps. We repeated this process up to 50,000 steps and summarized the results in Figure 23. While there are only a few outliers that can be considered as extreme values for rewards, the target values exhibit more variability, occasionally resulting many outliers. Considering that gamma is 0.99, it can be stated that the samples the 30,000th and 50,000th steps are extreme valued samples.
Although the causal relationship between the outliers in rewards and the anomalies in target values is not clear, it is speculated that the target values start to exhibit variability earlier than the rewards. From this observation, bounding the target crities' outputs described in Section A.5 is suggested as an effective remedy. Additionally, since the reward value can take extreme value, we use 1-st and 99-th percentiles of rewards for estimation of the lower bound and the upper bound, instead the minimum and the maximum.
102
Rowaed
10-1
Tanges valua otte
101
10%
10-1
1001
1001
2010
-1.653
b.e+0
+0.0+4
0.c+0
.1.098
0.e+0
step
10000
At
step
step 40000
At 20000
Figure 23: The transition of histograms for rewards (upper row) and target values (lower row) every 10,000 environment steps during the exccution of the MBPO algorithm on the "HalfCheetah" task. In each panel, the x-axis represents the value, and the y-axis represents the count. Noticeable outliers occur in the target values at the 30,000th and 50,000th steps.
37


Reviews:
[{"review_id": "EwCDcl9fXq", "reviewer": "Reviewer_xjto", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The paper is clearly written and includes enough details, such as their motivation and how their method relates to previous methods.\n2. The experiments and ablation studies also support the claims well.\n\nWeaknesses:\n1. The novelty of the proposed method is unclear. It is not obvious how the proposed method is different from [1, 2]. Specifically, could the authors elaborate on the difference between DHMBPO and the 'Model Derivatives on Predictions' variant in [1], which was originally proposed in [2]?\n2. The name Double Horizon Method is somewhat confusing and lacks an explanation.\n3. The experiments can be further enhanced by comparing them with model-free baselines.\n\n[1] Zhang et al. \"Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithms.\"\\\n[2] Clavera et al. \"Model-Augmented Actor-Critic: Backpropagating through Paths.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies model-based policy gradient and proposed Double Horizon Model-Based Policy Optimization (DHMBPO), which performs policy gradient on long distribution rollouts and short training rollouts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Can the authors discuss the distinctions or add experimental comparisons between [1, 2]?\n2. More clearly motivate the naming of the method.\n3. (Optional) Add model-free baselines to experiments.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UTBaGoALqU", "reviewer": "Reviewer_CSUu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Well written paper and extremely easy to follow\n- Strong results and good comparison against existing models as well as ablations\n\n**Weaknesses**\n- Largely this is an incremental contribution, with little deep mathematical or technical insight. Nonetheless the results are very impressive.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work contributes a novel combination of strategies for reinforcement learning which are then shown to be highly efficient in learning RL tasks, especially when compared in run-time (wall-clock). This combination is motivated by expressing the fundamental trade-offs in selecting one's distribution rollout length or training rollout length and how these two processes can be combined with a 'double horizon' such that each has an optimised rollout length for better training of the actor and critic models. Results show a significant benefit of the proposed Double Horizon Model-Based Policy Optimization (DHMBPO) method over alterative, even state-of-the-art models, particularly in runtime though also in ease of application.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None. I do believe that this paper is well written, concise, and quite honest about it's contribution. I am not an expert in this particular sub-field, but should there be no fundamental issues raised by the other reviewers, I would be inclined towards a straightforward accept.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jBQKZw6xSP", "reviewer": "Reviewer_UNGT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The problem studied in this paper is well-motivated and important. I believe that choosing the rollout length is one of the most fundamental problems in RL.\n\n+ This paper conducts extensive and comprehensive experiments to test various aspects of their algorithm.\n\n+ The proposed algorithm performs well compared to existing methods in terms of overall return and sample efficiency.\n\n+ The presentation is clear and easy to follow.\n\n\nWeakness:\n\n+ I am unsure how the actor objective given in (14) works. According to equation (7), differentiating both sides w.r.t. the policy parameter $\\theta$ yields $ \\nabla_\\theta \\hat{V} (s) = -\\sum_{t=0}^T \\gamma^T \\alpha \\nabla_\\theta \\log \\pi_\\theta(s_t|a_t) $ since other terms are independent of $\\theta$, which is quite different from the DDPG objective described in (12) that differentiates through the Q network. I checked with the attached code and it differentiates every action $a_t$, i.e., treating $a_t$ as a function of $\\theta$ instead of just a sample point as in policy gradient methods. This could be a critical gap between the methodology presented in the paper, and the implementation leading to the empirical results. I hope that the authors can either correct my understanding or explain this gap.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Choosing a proper rollout length in model-based RL can be a challenging problem. To address it, this paper proposes a double-horizon scheme that uses long rollouts to learn the model and short rollouts to update the actor and critic network.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There are a few minor changes I would suggest:\n\n+ In section 2.2, line 8, an equal sign may be mistakenly inserted after \"Following existing works\";\n\n+ The objective function of model learning is now located in the appendix, which should be moved to the main text if space allows;\n\n+ Ablation studies on the network architecture (e.g., the use of layer norm, dropout rate and activation function) would further strengthen the paper.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper suggests a new model-based reinforcement learning algorithm that combines ideas from two somewhat different approaches in using a model.
In one approach, one rolls out the model for multiple step in order to generate the distribution over states according to the current policy. One can then use the fictitious/imagined samples to update the critic or policy.
Another way is to use the model to expand the value function for multiple steps, and then compute the policy gradient by differentiating through the expanded value function and the model.

The main contribution of this work is proposing Double Horizon Model-Based Policy Optimization (DHMBP), which combines these two approaches of using the rolled out model. The paper empirically studies the proposed method, and compares with other model-based approaches, as well as a model-free one.


The reviewers are generally positive about the paper. Among the positive aspects, they mentioned:

- Clear and easy to follow presentation
- Strong experimental results

They were some initial concerns such as unclear novelty of the paper compared to some prior work and the lack of model-free baseline. These were clarified during the rebuttal phase and a model-free baseline was added in the revised version.

At the end, the reviewers are satisfied with this paper, and the final recommendation of all of them are an Accept (3x). Therefore, **I recommend the acceptance of this paper**.


I'd like to add some final comments based on my own reading of the paper. I think the paper has a reasonable idea. Since the paper does not have much of theoretical studies, the burden of its validity is mostly on the empirical results. The empirical results show good performance of the method. One aspect, however, is missing in my opinion, and that is a careful study of the role of each of the rollout parameters. The paper currently sets the DR horizon to 20 steps and the TR horizon to 5 steps throughout most experiments, except those where these values are set to the extremes of (20,0) and (0,5). The paper would be stronger if there are some studies of the effect of each of these parameters through more gradual changes. I realize this experiment is computationally expensive, so I think doing it for a limited set of experiments is acceptable. Such a result would give a better understanding of the role of each of these central hyper-parameters of the method.
This is an optional suggestion and is not a requirement for the acceptance of the camera ready version.


Typo:
- In Section 2.5, right after (12), "... by derived by extending ..." requires a revision.
Decision: Accept (poster)