Bootstrap Off-policy with World Model
Authors: Guojian Zhan
Likun Wang
Xiangteng Zhang
Jiaxin Gao
Masayoshi Tomizuka
Shengbo Eben Li
Venue: NeurIPS 2025 poster ()
Keywords: Model-based Reinforcement Learning
Online Planning
Primary area: reinforcement_learning
TL;DR: We propose BOOM, a model-based RL that uses a soft value-weighted likelihood-free alignment loss  to bootstrap the policy from non-parametric planner with world model, achieving state-of-the-art performance.

Abstract: Online planning has proven effective in reinforcement learning (RL) for improving sample efficiency and final performance. However, using planning for environment interaction inevitably introduces a divergence between the collected data and the policy's actual behaviors, degrading both model learning and policy improvement. To address this, we propose BOOM (Bootstrap Off-policy with WOrld Model), a framework that tightly integrates planning and off-policy learning through a bootstrap loop: the policy initializes the planner, and the planner refines actions to bootstrap the policy through behavior alignment. This loop is supported by a jointly learned world model, which enables the planner to simulate future trajectories and provides value targets to facilitate policy improvement. The core of BOOM is a likelihood-free alignment loss that bootstraps the policy using the planner’s non-parametric action distribution, combined with a soft value-weighted mechanism that prioritizes high-return behaviors and mitigates variability in the planner’s action quality within the replay buffer. Experiments on the high-dimensional DeepMind Control Suite and Humanoid-Bench show that BOOM achieves state-of-the-art results in both training stability and final performance. The code is accessible at \url{https://github.com/molumitu/BOOM_MBRL}.

Bootstrap Off-policy with World Model
Guojian Zhan 1.2 Likun Wang', Xiangteng Zhang', Jiaxin Gao', Shengbo Eben Li
Masayoshi Tomizuka",
College of Al & School of Vehicle and Mobility, Tsinghua University 2 Berkeley Al Research (BAIR), UC Berkeley
Abstract
Online planning has proven effective in reinforcement learning (RL) for improving sample efficiency and final performance. However, using planning for environment interaction inevitably introduces divergence between the collected data and the policy's 's actual behaviors, degrading both model learning and policy improvement. To address this, we propose BOOM (Bootstrap Off-policy with WOrld Model), a framework that tightly integrates planning and off-policy learning through a bootstrap loop: the policy initializes the planner, and the planner refines actions bootstrap the policy through behavior alignment. This loop is supported by jointly learned world model, which enables the planner to simulate future trajectories and provides value targets to facilitate policy improvement. The core of BOOM is a likelihood-free alignment loss that bootstraps the policy using the planner's non- parametric action distribution, combined with soft value-weighte mechanism that prioritizes high-returt behaviors and mitigates variability the planner's action quality within the replay buffer. Experiments Control Suite and Humanoid- -unch show that BOOM achieves state-of-the-ar results in both training stability and final performance. The code is accessible at btttps://gthii com mo umi tuttutu/BBB BBU//MOM_BBBB BBOLM_BRLL
the high-dimensional DeepMind
Introduction
Reinforcement learning (RL) has achieved impressive performance in wide range of domains, from
industrial automation to autonomous driving and embodied intelligence 43
36 49 Among the
various techniques developed to enhance RL, online planning stands out for its predictive optimization ability to improve control performance using learned dynamics [29 3 50) By performing look-ahead
rollouts, it enables agents to anticipate future consequences and iteratively refine actions 42 37
Compared to model-free approaches, which rely solely on trial-and-error learning, model-based planning offers an effective tool generate high-quality actions for environment interaction
growing body of research has explored how to more effectively integrate planning into RL 117 Early methods such as PETS 4 and PlaNet B have proven that planning with learned dynamics and reward signals can achieve impressive control performance. To further improve performance in high-dimensional tasks, recent approaches have combined online planning with policy learning, where the policy can provide good initial solution to speed up planning [ For example, LOOP [39 builds on SAC [IT). an off-policy model-free algorithm, and integrates planner guided by learned dynamics model to enable higher-quality environment interaction. Recently, TD-MPC
and TD-MPC2 19 [8] jointly learn the dynamies, reward and value functions through temporal
differential (TD) learning, achieving strong performance through both algorithmic innovations and
"Corresponding author: lishbo@tsinghua. edu. cn.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

implementation advances. These methods successfully deliver substantial gains over model-free baselines such as SAC, particularly on high- dith-emmnniinal benchmarks.
However, this class of planning-driven model-based RI. algorithms inevitably suffers from a fun- damental issue known as actor divergence: the data used for learning is collected by the planner, which acts as a different actor from the policy network [39]. Under the paradigm framework of off-policy RL, this issue leads to two problems: (1) Distribution shift in value learning: The value function is trained on the data collected by the planner rather than the policy itself. As a result, it learns accurately within the planner's 's actennction distribution but tends to overestimate values in out-of- of-isstrbiuuuion regions that are rarely visited (23). (2) Unreliable policy improvement: The policy network is updated using value estimates from the value network, which are influenced by the distributional shift. These biased estimates may mislead the policy, impairing its ability to distinguish between good and bad actions, ultimately leading to severe performance degradation (34) To conclude, the misalignment between the training data and the policy's 's actual behavior can severely hinder the learning process, particularly in complex, nigh-dimensional environments where accurate value estimation is already challenging [32]. The resulting value bias not only misguides policy updates but also risks destabilizing the entire training process (27]. More critically, since online planning algorithms typically rely on sample-based optimization, the resulting action distributions
are non- non- parametric and difficult to access. This means that they cannot be explicitly represented and the likelihood is intractable [47]
To address these challenges, we propose Bootstrap Off-policy with WOrld Model (BOOM), a novel framework that seamlessly integrates online planning with off-policy RI., effectively mitigating the negative impact of data distribution shifts caused by actor divergence. This is accomplished through bootstrap loop: the policy initializes the planner, and the planner refines actions to bootstrap the policy via behavior alignment, alleviating the actor divergence issue. This loop is supported by a jointly learned world model, which enables the planner to simulate future trajectories and provides value targets that facilitate policy improvement. We refer to it as bootstrap alignment because the planner typically generates higher-quality actions via model predictive optimization. Aligning the policy with these actions also offers strong guidance for improvement and accelerates learning.
In this paper, we introduce three key contributions: (1) To facilitate alignment with the online planner's sample-based non- non-parametric distribution, we adopt likelihood-free aligument loss that measures the divergence between the policy and the planner without requiring explicit likelihoods of actions from the online planner. (2) We introduce soft weigh-we mechanism that prioritizes high-return behaviors, driven by the planner's value-guided action selection principle. Additionally, to maintain training efficiency, we align the policy with the stored planner actions in the replay buffer. Since these historical actions may vary in quality, our weightteg mechanism ensures the policy prioritizes the high-valued promising experiences, accelerating learning while handling the variability in the planner's past actions. (3) BOOM combines ease of implementation with state-of- state-of-the-ar (SOTA) performance on sigh-dimensional continuous control benchmarks, including the DeepMind Control Suite [1 and Humanoid- -Bnnch [38].
2 Preliminaries
2.1 Reinforcement Learning
Reinforcement learning (RL) offers a powerful framework for sequential decision-making (24], formalized as a Markov Decision Process (MDP) (S, A, P,r:7), where S and A denote the state and action spaces, P(x') (// a) is the transition dynamics, r(s, a) is the reward function, and 7 E [0, 1) is the discount factor. At each timestep, an agent observes state s;, selects action ag, receives reward rt. and transitions to the next state st+1 according to P. A fundamental concept in RL is the action-value function: (s( a) Ex Σ 7'r 7'r(s,, a4) sp = 5, ag = a) which captures the expected return of taking action a in state s, followed by policy æ. Off-policy RL aims to discover the optimal policy T by maximizing the Q-value function using transitions generated from a different behavior policy 3. By decoupling data collection from policy improvement, off- off-policy methods can achieve high sample serienncy-primariyy due to the effective reuse of past transitions stored in a replay buffer. However, when the behavior policy 33 deviates too far from the resulting distributional shift can severely undermine value estimation, a delicate yet critical challenge that continues to motivate much of the recent progress in off- off-policy learning [8).
2

---

2.2
Online Planner
Online planning optimizes action sequences at each step by simulating future trajectories under predictive model, enabling informed action selection [48]. This approach typically yields high-quality decisions by leveraging foresight over potential future outcomes. As a result, it is widely adopted in model-based RL. for environment interaction, generating higher-quality interaction samples that facilitate more efficient learning and policy improvement.
Among various planning methods, Model Predictive Path Integral (MPPI) is widely adopted sampling-based planner for continuous control tasks due to its high efficiency [47]. Formally, at each planning step, with the help of a dynamics model, N, action trajectories are sampled from factorized Gaussian: of ((),, for f = 0,..., H and i = .....: Np, where H is the predictive horizon. Each trajectory is evaluated using a reward and value model to estimate the H-1 === r + iiz, where rÿ denotes predicted rewards and 0/ is the terminal value estimate. To update the trajectory distribution, MPPI reweights the N samples using a softmax over the returns: w" = exp(G" - max, /² exp(G* max, G ). Next, the parameters of the trajectory distribution are updated using weighted statistics: H4 t E, w' w'af. This process is repeated over several iterations, gradually converging toward an optimal solution. At test time, the planner executes the first action Mo, while during training, Gaussian noise is added for exploration.
future return: G -E=o
Despite its effectiveness, MPPI and similar planners suffer from key limitation: the resulting planned action is obtained via weighted averaging over sampled candidates and does not directly arise from a parameterized probabilistic policy. Although Gaussian noise is used during the sampling stage, the final action distribution no longer adheres to a true Gaussian form because of the reweighting and resampling process, As a result, computing the precise likelihood of MPPI-generated actions intractable in practice.
3 Method
3.1 Inevitable Actor Divergence When Off-policy RL Meets Online Planning
common approach to integrating online planning with off-policy RL is jointly learn latent world model and a policy from replay buffer data, as exemplified by TD-MPC2. Specifically, the total world model trains an encoder = h(s), a latent dynamics model z' = f(z, a), reward predictor R(2, a). and value function ((2, a). where denotes the latent state. Throughout the learning process, all states s are first encoded into latent representations 2. For notational simplicity, we continue to denote the inputs to the policy and value functions as although they implicitly refer to the encoded latent states. The components h, J. R, and Q are jointly trained using the TD loss:
H
Σ x" (1f(er)))))) sg(h(s))) CE(Re,)) (E(Qr. (1)
Lmodel = E Essaarss))o
where q: is the target value computed using the policy network म, sg denotes the stop-gradient operator and CE denotes the cross entropy loss function. Then following standard off off-policy RL, the policy is updated by maximizing predicted Q-values i.e., Spolicy -Es,,,,;'''' ΣH11 Q(s, xs)).
Despite being trained purely on off-policy data, this world model empowers the planner, such as MPPI, to perform effective predictive optimization. We refer to the resulting behavior policy executed at each timestep as 8, which can be understood as x +PPPII with world model. By scamlessly integrating model-based planning into the off off-policy learning pipeline, this approach yields high-quality training interactions and enhances sample efficiency.
However, this paradigm inevitably suffers from phenomenon known actor divergence. During data collection, the agent interacts with the environment using
mumner-augmenterr policy + MPPI, which refines policy actions using model rollouts. This behavior policy /3 may exhibit highly deterministic or modal behavior, resulting in a state-action distribution d3 d3, a) that may significantly diverges from that of the network policy и. This mismatch breaks the assumptions of standard off-policy learning and can lead to instability in training process. Specifically, such an actor divergence leads to following two major challenges:
3

---

Inétial solution
Bootstrapped policy learning
Environment interaction
Policy
Planner
Kin akL
Max QOe m(x)
Alligneent target
World model learning
Dynanics model
Reward
Value
nodel
Figure I: Overview of the propased BOOM algorithm. BOOM consists of three key components: policy, planner, and a world model. The policy and planner form bootstrap loop where the policy provides the planner with an initial solution, and the planner in turn guides the policy via alignment. The world model plays dual role: enables the planner to perform receding horizon control for collecting high-quality trajectories, and it allows the policy to utilize Q-values for effective performance improvement.
(1) Distribution shift in value learning. The value function is trained minimize the Bellman error over samples drawn from the behavior distribution: [((sa)=d [Q(s, a) T Q(s, a))² where T=Q(s, a) r(s, a) + (>))==( a')] denotes the Bellman backup under policy л. However, when d assigns little probability mass to regions where π places significant density, the value function is only optimized on narrow subset of the state-action space. This mismatch can result in biased and generally overconfident estimates in out-of-distributin regions, undermining the accuracy and reliability of value learning.
(2) Unreliable policy improvement. The policy x is typically optimized by maximizing the expected value under Q. However, due to the distribution shift mentioned above, Q may be inaccurate, especially in regions where assigns high probability but are poorly represented in the behavior distribution B. This bias will result in unstable policy updates and poor performance.
In summary, actor divergence inevitably occurs when online planning meets off-policy RL, which undermines the core assumption of distributional consistency in off-policy learning paradigm and degrades both model accuracy and policy improvement.
3.2 Bootstrap Off-policy with World Model
To mitigate actor divergence caused by collecting data with plamning-augmented behavior policy (e.g., 8- 8 ग + MPPI), we propose a simple yet effective framework: BOOM (Bootstrap Off-policy with World Model) a planner, and a world model. At the heart of BOOM a bootstrap loop: the policy provides an initialization for the planner, while the planner refines this initialization through model predictive optimization and in turn bootstraps the policy via behavior alignment. The world model, trained in the TD-MPC2 style, serves dual purpose: it enables the planner to simulate future trajectories for better control, and it supports the policy with value estimates for improvement.
shown in Figure I It comprises three tightly coupled components- policy,
The core of BOOM is the Bootstrap Alignment objective-a likelihood-free regularization term that encourages the policy π to align with planner-generatee actions without requiring the likelihoods of planner's noramaraitr distribution. This objective is further enhanced by a soft value-weighted mechanism that prioritizes high-return behaviors. We describe these two components in detail below.
Likelihood-ffer alignment metric. Since all actions stored in the replay buffer are generated by the planner policy B, we can directly imitate them to align planner behaviors without impairing training efficiency. However, 33 is non-parametric sasple-based planner whose action likelihood is intractable. This makes typical imitation learning metric, such as reverse KI. divergence, theoretically inapplicable, as they rely knowing 3(a s). To avoid this, adopt a likelihood-free approach by

---

minimizing the forward KL divergence:
((a 8) <aa s)
KL π) = Ba s) log
(()s( [log ((a
|s] - 1a- [log (a s)]
(2)
The first term depends only on 3 and is constant with respect to the parameters of x. Therefore, we discard during optimization, resulting the following simplified loss:
Calign *) [- log T(a s)]
which encourages π to assign higher probability to actions chosen by the planner, without requiring any access f(a s). This likelihood-fre formulation provides simple, principled mechanism
to distill the strengths of an online coanner-yillding non non-parametric action distributions inttiooonnn-ooooooooo parametric policy.
Soft Q-weighted mechanism. To further enhance policy learning, we introduce a value- guided alignment objective that prioritizes high-return behaviors, drawing inspiration from the planner's action selection principle-whh where candidate actions are first sampled and then selected based on their value-weighted probabilities. Given a batch of off-policy transitions {(s). ax) ] replayed from buffer, we define a soft target distribution over actions using the current Q-function: p x exp (Q/T), where T > 0 controls the distribution sharpness and is set by default. Normalization across the batch yields weights wi = exp(Qt/T)/E exp(Q./T). Now the soft Q-weighted alignment loss becomes
H-1 N
Lalign = (' Ja:
ΣΣ 1= ==
Σ
wi [ log ग (a; 8:)]-
This mechanism ensures that the policy assigns higher probability to high-value actions, effectively guiding it toward promising regions. Besides, since these actions stored in the replay buffer may vary in quality, this value-weighting ensures that the policy prioritizes the most beneficial experiences, thus accelerating learning while accommodating the variability in the planner's past actions. The Q function here can also be replaced by other critics, such as state value V or advantage A, here we think Q is the most convenient one access.
Bootstrapped policy objective. By combining the above two techniques, we integrate the alignment term into the standard policy loss to obtain the final bootstrapped policy objective:
Σ Q(ss + Aatign Calign
Cpolicy -s,,,,, s')oi JoiM
(5)
where Aatign tunable coefficient. This bootstrapped objective encourages the policy not only to improve with respect to its own value estimates (max Q) but also to stay aligned with the high-quality planner actions found in the replay buffer (min Lquatign). The complete pseudocode of our BOOM is presented in Algorithm I
Discussion on why the world model learning is improved. The reasons are two-fold. (I) Improved value learning. By aligning the policy with the planner, we reduce the discrepancy between the collected data and the actual policy behavior. This improves the distributional matching during training, enabling the value estimator to learn from more consistent and policy-relevant trajectories. As result, the predicted values used in planning become more accurate and reliable. (2) Improved representation, reward, and dynamics learning. We adopt TD-style learning objective that jointly optimizes the value function along with the other components of the world model as shown in (I). As the value predictions become more accurate due to better distributional match, the gradients flowing into the encoder, dynamics model, and reward predictor become more informative, leading to improved overall model quality.
3.3 Theoretical Analysis
We provide theoretical guarantees for bootstrap alignment in addressing actor divergence- the mismatch between the planner collected data and the policy actual behavior. By minimizing the divergence (T((B](T)) we establish theoretical bounds on the return gap and Q-value deviation, ensuring stable and efficient policy learning.
5

---

Algorithm 1 BOOM: Bootstrap Off-policy with World Model
Input: Policy no, encoder hig dynamics fuz, reward Rw, value p. planner P
Initialize: Te, hgg fg. Rws Q
1: // Warmup (World Model Pretraining)
2: Interact with environment using random actions: (r, done) <- env. .nv.step( stepp((rand
3: Store random transitions (s, Grand: T, s') into replay buffer DD
4: Update bg. fe. Rws Q by minimizing model loss moddal in
5: for each iteration do
6: // Data Collection (Using Planner)
7: Encode current observation:
hg(s)
8: Plan: a3 3 = P(Te, Jv, Rw, Q+. Interact with environment: (r,, 8' done) f- env. step(ag)
9:
10:
Store transition (s, (4,, 7,8 into replay buffer DD
11:
// World Model and Policy Learning
12: Replay rollout batch {(5t; (1. St.11) H-1 D
Update hę. fØ. Rw Qo by minimizing model loss maadel
14: Update Te by minimizing bootstrapped policy loss poliic in 5
end for
Theorem 1 (Bootstrap Alignment Controls Return Gap). Let B(a s) be the behavior policy, Tia | be the learned policy, and d8 (s) the state distribution induced by B. Assume the per-step reward satisfies [r(s, a)) S Ranax and the discount factor y E 1). Then for any state if (T] [)) S E, the following return gap bound holds:
(s)
S Rmax V2e.
[(B) [(T)-
1 ?
Proof. See Appendix A.2
The first theorem shows that bootstrap alignment ensures a small return gap between 33 and n, meaning staying close to the planner avoids performance drops from distribution mismatch.
Theorem (Bootstrap Aligmment Controls Q- V-luee Gap). Assume that for any state 8, the learned policy (a 8) and the planner 3(a | s) satisfy KL(3I)T) S E, and that (s, is Liissshciitz continuous in a. Then for any state s, the expected Q-value difference is bounded as:
Q((saa)) - Q(, az)) S LQ ||a a 22 S LQ D(E),
(7)
where D(s) is an upper bound on the 2-norm distance ||a ax 12 with ag 3(ss and an ~ T(s). In general, the action distribution of MPPI can be approximated by a Gaussian Mixture Model
(GMM) as B(x) = E W.N (1ss Σ.) with weights we 2 0, -i=1 Wi 1, and π(s) = N Nuuz E.z) is Gaussian policy. Denote the maximum eigenvalue of each Y, as A; = A(E;) and that of Enx as A(Ex). Then for any å E (0, 1), with probability at least - over a3 ~ B(s) and ag ~ n()), the deviation bound satisfies:
D(e) < min 2Vd, max (V2 log K6 + + 2= + 2A(E2) log
(8)
Here, d is the dimensionality of the normalized action space.
Proof. See Appendix A.3
The second theorem bounds the Q-value difference under bootstrap alignment, showing that value overestimation-cont coneiiaatiion-ommmllly seen when the policy strays into poorly covered regions-is effectively controlled by alignment. This avoids misleading policy updates and stabilizes training.
Together, these insights justify that bootstrap alignment mitigates actor divergence by keeping the learned policy close to the behavior policy, preventing large discrepancies in value estimates. It retains the benefits of off-policy learning without requiring access to behavior policy likelihoods, making it compatible with modern planners and applicable in complex, dighddimensional settings.

---

1802
3500
123
15
20
18
25
05
28
15
29
05
16
15
(a) AVG. DMC Suite
(b) Humanoid-staiie
(c) Humanoid-we
(d) Humanoid-rum
1822
3009
100
640
6/
200
102
meeaeis 191
(e) Dog-stand
(I) Dog-walk
(g) Dog- trot
(h) Dog-run
202
(i) AVG. Human. Bench
(j) H Handd-sttand
(k) Hlhaad-walk
(1) Hlhand-run
64
400
409
(m) Hlland-s
(n) Hlhand-slide
Hlhhandpoole
(p) HHhhnn-hurrih
Legend
SAC
DreamerV3
TD-MPC2
BMPC
BOOM (ours)
Figure 2: Training curves on benchmarks. The solid lines represent the mean, while the shaded regions indicate the confidence interval over three runs. The average performance curves for the two benchmarks appear the left corner of the 1st and 3rd rows, respectively, highlighted in bold.
Experiments
4.1 Experimental Setup
Baselines. We choose four representative online RL algorithms as our baselines: (1) SAC I
the state-of-the-art model-free off-policy RL algorithm under the maximum entropy framework; (2) DreamerV3 [16] the state-of-the-arr imagination-driven model-based RL algorithm that learns from imaginary rollouts generated by world model. To enable comprehensive comparison,
evaluate DreamerV3 under two different interaction budgets: 2M and 10M iterations; (3) TD-MPC2 [18]: clanning-driver model-basee RL algorithm that performs online planning via an MPPI planner; (4) BMPC 45]: an improved variant TD-MPC2, where the policy is trained solely to imitate the actions generated by the MPPI with an extra relabeling mechanism update the historical actions in the replay buffer with the latest planner.
Benchmarks. We evaluate our method on a challenging benchmark of 14 high- figh-dimensional locomo- tion tasks drawn from the DeepMind Control Suite (DMC) 41] and the recently proposed Humanoid Bench (H-Bench [38]. The chosen 7 DMC tasks feature two most complex agents-humanoii (67/21 state/action dims) and dog (223/38)-th) -tht demand sophisticated balance and coordination. The other H-Bench tasks raise the difficulty further with long-horizon, goal-directed tasks on the Unitree HIna robot (151 (/1/))). such as walking over slides, traveling over pole forest without collision, and continuously crossing hurdles. Detailed descriptions are listed in Appendix B.I
7

---

Implementation details. The detailed hyperparameters and reproducibility statement of other baselines are documented Appendix B.2
4.2 Experimental Results
All the training curves are shown in Figure and the detailed numerical results are listed in Table Our method, BOOM, consistently delivers the best Total Average Return (TAR) across all 14 high- dimensional locomotion tasks. These tasks pose significant challenges due to their large state and action spaces, yet BOOM demonstrates remarkable stability and effectiveness.
and
Results on the DMC Suite. Our BOOM achieves an average TAR of 877.7, outperforming the previous best BMPC (835.8) by a substantial +5.0% and exceeding TD-MPC2 by an even larger margin of +17.7% %. Notably, BOOM achieves new best results on all humanoid and dog tasks. In Humanoid-run, BOOM outperforms the second-best method by +9.7% %, and in Dog-run, it leads by staggering +21.8%. While SAC and DreamerV3 often fail these high-dimensional control tasks and achieve near zero performance, planning- based methods like TD-MPC2 and BMPC perform better but still suffer from instability and limited final returns. In contrast, BOOM learns faster and achieves much stronger final performance across all tasks.
Results the Humanoid Bench. Our BOOM again dominates with average TAR of 820.6. This marks a dramatic +47.7% improvement over D3eamerV3 (10M), which is (555.6), and an even more impressive +60.5% gain BMPC. BOOM sets new records every single task in the Humanoid Bench. For example, in H Hhand-slide BOOM improves over the second-best method by +110.5% %, in Hhhnn-pooe,, by +25.8 %, and in H by +121.0% ...
In summary, compared to SAC's difficulty in scaling to complex control, DreamerV3's limitations in sample efficiency, and BMPC/TD-MPC2's occasional instability, our BOOM demonstrates clear and consistent advantages across the board.
Table I: Total Average Return (TAR) on 7 DMC Suite tasks and 7 Humanoid Benchmark (H-Bench) tasks. Mean Std over seeds. Bold best, underlined second-best. Higher is better.
Task
SAC
Dreamer V ( 2M & 10M iters)
TD-MPC2
BMPC
BOOM (ours)
sumancid-stand
9.0 + 0.7
264.5 + 44.1
717.0 21.2
913.3 = 14.7
947,9 = 4.4
962.1 + 10.7
Humanoid-walk
173.8 +
251.5 35.2
755.6 25.2
884.8 = 8.3
935.1 4.0
936.1 +
Hlumanoid-rm
353.5 33.2
9.2
$31.2
42.0
26.0
Dog-stand
197.6 L 102.4
35.4 10.8
35.4 10.8
936.477.66
971.3 1 11.0
986.8 1 1.8
Dog-walk
24.7.11.33
9.1 0.6
9.1
885.0 74,8
942.9 9.6
965.4 1 0.3
Dog-trot
67.1
7.9 7.9::3
8.9
884.4
911.3
18.2
947.9 + 4.7
Dog-run
16.5
4.3 3.2
3.2
127.0
50.2 673.7
820.7 + 23.0
AVG. DMC Suite
58.8 1 57.7
87.9 4: 15.8
269.0 4: 13.6
745.6 1: 34.1
835.8
20.7
877.7 1: 20.3
Hthand-stand
74.1 17.5
220.3 73.5
845.4 + 27.3
728.7 + 121.9
780.0 65.8
926.1 + 19.2
H Whlkddwwlk
27.0 13.8
161.3 + 44.5
744.0 28.7
644.2 + 281.1
672.6 10.4
935.4 +
Hlhand-run
14.1 1.4
55.8 10.5
622.4 + 66.7
66.1 8.1
236.0 1: 53.9
682.2 + 120.6
Hlhand-sit
268.4 26.1
687.3 + 138.0
699.1 177.2
693.7 + 249.9
688.2 + 46.3
918.1 + 4.2
H1hand-slide
1.9
162.6 29.5
367.6 29.7
141.3
440.1 25.4
926.1 + 8.0
Hlhand-polc
122.5 33.5
$34.3:6551
577.4
207.5 35.6
739,9 18.0
930.5 + 18.9
Hlhand-hurdle
1.79
3.0
135.7
$9.0 19.3
197.1 12.1
435.6 29.8
AVG. H-Bench.
1..51.99.2
233.0 53.9
555.6 49.5
338.8 1 98.6
$11.7
820.6 31.0
4.3 Ablation Study
We conduct three ablation studies to assess the contribution of key components in our framework:
Bootstrap alignment metric. We compare the common reverse KL divergence with our proposed fxellhoodd-rre forward KI. As shown in Figure 3al forward KI. consistently shows higher returns, highlighting its strength capturing the planner's non-paraeetric distribution without requiring likelihood estimation. Reverse KL, in contrast, relies approximating the planner's likelihood,

---

1000
1000
300
300
500
TAR
400
x1 (ours)
200
(ours)
200
Q-weight, (ours)
200
(low ratio)
Reverse
WO G-weight
10 (high rabio)
5.5
2-3
1.3
20
3.3
1.3
10
5.5
3-3
1.3
20
Bienations
Barations (M)
Bleradions (H)
(a) Ablation study on the alignment metric
(b) Ablation study on the Q-weight mechanism
(c) Ablation study on the alignment coefficient
Figure 3: Ablation study curves. We select the Dog-run task (223/38 state/action dims) in DMC Suite with the highest dimensionality to perform all ablation experiments.
which is not directly accessible. We estimate it using Gaussian surrogate based on the value- weighted mean and variance of planner actions. The weaker performance also suggests that such approximations may be inaccurate and detrimental. Further discussion is provided in Appendix C.T
Soft Q-weight mechanism. We replace the soft Q-weighting with uniform weighting to assess the benefit of leveraging the learned Q-function. As shown in Figure (bb, incorporating Q-weights consistently accelerates training and improves final performance. This improvement stems from the ability of Q-weighting to handle the variability in action quality within the replay buffer by prioritizing high-value actions, thereby enabling more focused and efficient policy updates.
Alignment coefficient. We vary the alignment coefficient by testing 0.1 and 10x the default setting (Aalign = (im( 1100 for DMC and dim( /50 for Humanoid -ennch))... Results in Figure 3e show stable performance across this range, suggesting that BOOM is robust to this hyperparameter and does not require very sensitive tuning.
5 Related Work
Model-based RL, i.e., MBRL can be broadly categorized into planning-den driven and imagination-driven approaches, depending on how the learned model is utilized during training.
Planning-driven MBRL. Planning-driven methods use planner rather than policy itself to generate
high-quality actions for environment interaction [5]. Early work has shown that solely learning the
value and then combining with the planner can achieve good control performance B3 22 7 26). To further improve performance in high-dimensional tasks, recent approaches have combined online planning with policy learning, where the policy can provide a good initial solution to speed up planning 20 44 33 25) LOOP (39 takes SAC [11] as the backbone, and employs planner under the policy behavior constraint for collecting samples. TD-MPC family [19 [8] jointly learns model and value function through TD-learning achieving strong performance through both algorithmic innovations and implementation advances. It successfully delivers substantial gains over model-free baselines, particularly on complex benchmarks. However, these methods inevitably encounter actor divergence-a mismatch between the planner and the policy. Our approach, BOOM, addresses this challenge by tightly coupling planning and off-policy learning through a bootstrap loop that aligns the policy with the planner's 's non-parann non-parametric action distribution via a Q-weighted likelihood- free alignment loss, preserving distributional consistency. A recent method BMPC [45] simplifies the pipeline by discarding explicit policy optimization and directly imitating the planner to avoid actor divergence. However, BMPC ignores the Q-function during training, resulting in lower policy learning efficiency and sensitivity the variability of historical planner actions in the buffer. This leads to unstable learning, as reflected by its oscillatory training curves in our experiments. We acknowledge a concurrent and close work, TDM(PC)² (25), which similarly found that aligning the policy with the planner is beneficial. The major distinction lies in the design of the alignment objective: their approach follows TD3+BC style using reverse KI., whereas ours is closer AWAC using critic-guided weights and forward KL.

---

Imagination-driven MBRL. Imagination-drinn driven methods leverage a learned model to generate
synthetic rollouts for policy and value updates 440] Modern approaches such as SimPLe [2]].
14 13 train latent dynamics models to support actor-critic learning entirely in imagination. These methods offer fast test-time execution and relatively high sample efficiency, but their performance is often limited by compounding model errors over long imagined rollouts. Among them, DreamerV3 [[6] stands out as a leading representative, demonstrating strong performance across range of tasks. Unlike planning- -baeidd MBRL methods, DreamerV3 interacts by directly sampling actions from its learned policy, typically requiring more iterations to converge. For comprehensive comparison, iterations. While it improves with more interaction budget and outperforms TD-MPC2 and BMPC on certain tasks, our BOOM consistently outperforms all baselines across all tasks.
IRIS 28 IDMM 30), and the Dreamer family [12
evaluate DreamerV3 under 2M and 10M
Compared to Offline RL. One might notice the similarity between BOOM and offline RL; however, the fundamental paradigms and practical implications of these two settings differ significantly, (1) Leurning puradigm. Offline RL centers on fixed dataset, relying heavily on behavior cloning (BC) to restrict the policy within the dataset's support [TO]. To cautiously improve policy performance, offline RL relaxes BC for poor actions, but always aims to keep the policy close to known data to avoid extrapolation error (34, 35]. BOOM centers on the policy itself as the optimization target. It seeks to maximize Q-values and align with Saanner-generate actions, both aimed at improving policy performance. Ultimately, the policy and planner co-adapt and converge to the optimal solution through online interaction. (2) Learning objective. In offline RL, maximizing Q-values and BC often conflict 19: Q-values outside the dataset distribution cannot be accurately estimated, so maximizing Q risks pushing the policy toward unsupported actions; BC pulls the policy back toward known actions. This tension forces a conservative balance [46]. In BOOM, the two objectives are largely complementary: the planner generally produces higher-quality actions than the policy [51). Aligning policy with the planner improves performance and strengthens policy-planner consistency, which in turn leads to more accurate Q estimates. More accurate Q-values then enable better policy improvement and allow the planner to generate higher-quality actions. This positive feedback loop bootstraps policy and planner consistently toward faster convergence to the optimal solution.
6 Conclusion
We introduce BOOM, a model-based RL method that enhances the integration of planning and off- off-policy learning through bootstrap alignment. By leveraging the non- parameerttticc planner actions not only for environment interaction but also for bootstrapping policy behavior via a Q-weighted likelihood alignment loss, BOOM mitigates the inevitably actor divergence issue in planning- driven model-based RL methods, improving both training stability and final performance while maintaining high time-efficiency and flexibility of off-policy learning paradigm. Experiments on tens of high-dimensional locomotion tasks show that BOOM consistently outperforms existing planning- driven and imagination-drrver baselines. We believe BOOM establishes strong foundation with ample room for further improvement, such as more adaptive integration of max-Q and bootstrap alignment objectives, or the adoption of more expressive policy classes beyond diagonal Gaussians like diffusion models to unlock even higher performance and broader applicability.
Our work underscores two critical directions for advancing planning- MBRL. First, we em- phasize that the learned value function remains the fundamental bottleneck; reliable improvement in its estimation accuracy consistently translates to enhanced planning and superior final asymptotic performance. Second, a promising avenue for future research concerns the principled adjustment of the maximum entropy temperature coefficient (o). In planning-driven MBRL, the sampling is dictated by the planner's 's search, yielding an intricate, non-trivial relationship between the planner's 's intrinsic exploration entropy and the network policy's ' entropy. Developing a automatic tuning mechanism for based on these entropy signals is key to stabilizing the inherent tension between exploration and exploitation-a challenge that continues define the frontier of effective RL algorithm design.
7 Acknowledgment
This study supported by the Tsinghua University-Toyore Joint Research Center for AI Technology of Automated Vehicle, Beijing Natural Science Foundation (L257002), and SunRising Lab.
10

---

References
[1] Arthur Argenson and Gabriel Dulac-Arold. Model-based offline planning. arXiv preprint arXiv:2008.05556, 2020.
[2] Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforeement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 31, 2018.
[3] Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In Proceedings of the International Conference Learning Representations (ICLR), pages 6-9, 2019.
[4] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
[5] Sebastian Curi, Felix Berkenkamp, and Andreas Krause. Efficient model-based reinforcement learning through optimistic policy search and planning. Advances in Neural Information Processing Systems, 11144116-111177 2020.
[6] Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICMI-11), pages 465 465-472,77 2011.
[7] Mare Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data-efficient learning in robotics and control. IEEE transactions on pattern analysis and machine intelligence, 37(2):408 420,0 2013.
[8] Jingliang Duan, Wenxuan Wang, Liming Xiao, Jiaxin Gao, and Shengbo Eben Li. Dsac-t: Distributional soft actor-critic with three refinements. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[9] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 34:20132-20145, 2021.
[10] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062.
PMLR, 2019.
[11] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. Pmlr, 2018.
[12] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1911:: 2019.
[13] Danijar Hafner, Timothy Lillicrap. Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2565. PMLR, 2019.
[14] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
[15] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.0411 2023.
[16] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pages 1-7, 2025.
[17] Jessica B Hamrick, Abram L Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Witherspoon, Thomas Anthony, Lars Holger Buesing, Petar Veličković, and Theophane Weber. On the role of planning in model-based deep reinforcement learning. In International Conference on Learning Representations, 2020.

---

[18] Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023.
[19] Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.0490 2022.
[20] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
[21] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model- based reinforcement learning for atari. arXiv preprint arXiv:/903.000
2019.
[22] Jus Kocijan, Roderick Murray-Smith Carl E Rasmussen, and Agathe Girard. Gaussian process model based predictive control. In Proceedings of the 2004 American control conference, volume 3, pages 2214-2219. IEEE, 2004.
[23] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1 179- 1191, 2020.
[24] Shengbo Eben Li. Reinforcement Learning for Sequential Decision and Optimal Control. Springer Verlag, Singapore, 2023.
[25] Haotian Lin, Pengcheng Wang. Jeff Schneider, and Guanya Shi. Td-mpe?: Improving temporal difference mpe through policy constraint. arXiv preprint urXiv:2502.03550, 2025.
[26] Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. In International Conference on Learning Representations, 2019.
[27] Jared Markowitz, Jesse Silverberg, and Gary Lynn Collins. Avoiding value estimation error in off-policy deep reinforcement learning. In Can't I Believe It's Not Better Workshop: Failure Modes of Sequential Decision-Makiin in Practice (RLC 2024), 2024.
[28] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models, arXiv preprint arXiv:2209.00 005888, 2022.
[29] Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 16(1):1-118, 2023.
[30] Yao Mu, Yuzheng Zhuang, Bin Wang, Guangxiang Zhu, Wulong Liu, Jianyu Chen, Ping Luo, Shengbo Li, Chongjie Zhang, and Jianye Hao. Model-based reinforcement learning via imagination with derived memory. Advances in Neural Information Processing Systems, 1149493-9505, 2021.
[31] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference on robotics and automation (), pages 7559-7566. IEEE, 2018.
[32] Michal Nauman, Michat Bortkiewicz, Piotr Miloš, Tomasz Trzciski,, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In International Conference on Machine Learning, pages 37342-37364.
PMLR, 2024.
[33] Tung D Nguyen, Rui Shu, Tuan Pham, Hung Bui, and Stefano Ermon. Temporal predictive coding for model-based planning in latent space. In International Conference on Machine Learning, pages 8139. PMLR, 2021.
[34] Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
12

---

[35] Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. In International Conference on Machine Learning, pages 28701-28717. PMLR, 2023.
[36] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):044-077) 2020.
[37] Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, loannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. Advances in Neural Information Processing Systems, 01127580-27999 2021.
[38] Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Hu- manoidbench: Simulated humanoid benchmark for whole- body locomotion and manipulation. arXiv preprint arXiv:2403. 10506, 2024.
[39] Harshit Sikchi, Wenxuan Zhou, and David Held. Learning off-policy with online planning, In Conference on Robot Learning, pages 1622-1633, PMLR, 2022.
[40] Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 264):160-163, 1991.
[41] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 18011800000000 2018
[42] Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behav- iors through online trajectory optimization. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 4906-4913. IEEE, 2012.
[43] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster in starcraft using multi-agent reinforcement learning. nature, 575(7782):350-333 354, 2019.
[44] Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. arXiv preprint arXiv: 1906. 1906.08649, 2019.
[45] Yuhang Wang. Hanwei Guo, Sizhe Wang, Long Qian, and Xuguang Lan. Bootstrapped model predictive control. arXiv preprint arXiv:2503. 2025.
[46] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies an expressive policy class for offline reinforcement learning. International Conference on Learning Representations, 2022.
[47] Grady Williams, Andrew Aldrich, and Evangelos A Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, (002):34-3577, 2017.
[48] Guojian Zhan, Qiang Ge, Haoyu Gao, Yuming Yin, Bin Zhao, and Shengbo Eben Li. An explicit discrete-time dynamic vehicle model with assured numerical stability. Vehicle System Dynamics, pages 1-24, 2024.
[49] Guojian Zhan, Yuxuan Jiang, Shengbo Eben Li, Yao Lyu, Xiangteng Zhang, and Yuming Yin. A transformation-agrreaatii framework for state representation of autonomous driving systems. IEEE Transactions on Intelligent Transportation Systems, 25(7):7311-732 2024.
[50] Guojian Zhan, Yao Lyu, Shengbo Eben Li, Yuxuan Jiang, Xiangteng Zhang, and Letian Tao. Enhance generality by model-based reinforcement learning and domain randomization. In 2023 7th CAA International Conference on Vehicular Control and Intelligence ( (CVCT), pages 1 1-6. IEEE, 2023.
[51] Guojian Zhan, Xiangteng Zhang, Feihong Zhang, Letian Tao, and Shengbo Eben Li. Bicriteria policy optimization for high-aceuracy reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2025.
13

---

A Theoretical Analysis
A.1 Useful Lemmas
Lemma 1 (Triangle Inequality). For any vectors I, R", the triangle inequality states that the norm of the sum of two vectors is less than or equal to the sum of their norms:
2 + vll & ||||| + llvl:
where -| denotes the standard Euclidean norm.
Lemma 2 (Pinsker's Inequality). Let and q be two probability distributions measurable space F). Denote the total variation distance between and as
TV 4) sup (n(A) -
pp(x) q(x)) dx,
AEF
where and q are the probability density functions of p and &. respectively. The forward Kull- back-Leibler (KI.) divergence from p to q is defined as
| p(x(t) log p(x) dx.
KL(pllq)
q(x))
Then the total variation distance is upper bounded by the square root of the forward KL divergence:
TV S V KL(llg).
Lemma 3 (Total Variation Bound on Expectation Difference). Let p and be two probability densities over a common measurable space X. and let : + R be a measurable function such that || suPaex [=)] Then the difference expectations bounded by
Bx=p[((x)) - (f < 2ll TV(p,
Proof. We hegin by expressing the difference in expectations:
- [,,]]]]]]] 1 [(x))((z)) a(x)) dx
By the triangle inequality,
1L, -()))())-oo())
g(x)) dr (1(x))lo(())) a(x)) dr.
Using the fact that [f() s I1flloc, obtain
f [(()))p(())) - q(z)) de s llf / (p(z) - g(()) de.
Recalling the definition of total variation distance.
[ (m(x) - a(x)) de dr= 2TV(p)))
we conclude that
- S 2ll TV(p.q).
Lemma 4 (Concentration of Gaussian Policy Samples). Let 0r N( Ex,) be sample from multivariate Gaussian distribution with mean Rd and covariance matrix En E [Rdxd, Define
A(Ex) = sup v E
||||==1
as the largest directional variance (i.e., the spectral norm of Ex). Then, for any (0, 1), with probability at least - 8,
||a Hall2 S 2A((E)) log J.
14

---

Proof. Let 2 ~ (O,, Ia), and let L be matrix such that Σn LL written ax Mar + Lz, and we have
Then the policy sample can be
||,,
M 12 L2 Σx
Using a standard concentration bound for sub-Gau nub-Gaussian quadratic forms (e.g., the Saurent-Massare inequality), for any å E (0, 1),
Pr (= E R[= Ex 22 s 8.
Since B[2 E = Tr(Ex)) ), we obtain
Pr (Ila prll 2 Tr(E,) + (A,)), 8.
By omitting the trace term, get a looser but simpler bound:
|| - Mrll2 1 2A(Ex) log 3. with probability at least
Lemma 5 (Mean Bound via KL Between Gaussians). Let = N (1pp Ep) and q = =N(Wgg Eg) be Iwo multivariate Gaussian distributions of dimension d. Then,
KL(plla) 2 2 11 Molls,
Consequently, (LL(l)))) S 6, then
1|p Mall2 S 2cA(2ig)
where A(2xg) denotes the largest eigenvalue of Mig
Proof. The KL. divergence between Gaussians is given by:
1 tr tr(2., Exp) + (Mg - -1p) M.g (Mg pp) d + log det Σg det E.p
(T.(pll4) 2
Dropping the non-negative trace and log-determinant terms, we obtain the lower bound:
KL(plla) 2 2 11 Moll Mollz,"
Applying the spectral norm inequality ||u1|3 S A(E || we get:
1|pp 13ll3 < 2<A(E.)
This completes the proof.
A.2 Proof of Theorem
Proof. We aim bound the difference between the expected returns under two policies and B based on the divergence of their induced trajectory distributions.
By definition, the expected return under policy is:
J(x) E
Σ
where (sp, ag. 81, a1...... denotes full trajectory, and similarly for we obtain
00
J(B) E,,
Σ
(rr(
1=0
15

---

We compute the difference:
(VB)
(B (s az)] [x
1=0
Using the triangle inequality in Lemma] 1 this is upper bounded by
(r at) - (eeeaae)all (r(s), at]l
Assuming the reward function is uniformly bounded as a)( S Rmaxx Lemma 3 implies the following bound:
E (r ax)] Beer,e))~ (r(s, de)] S 2Rmax (V(P.Pp))
where S and pf denote the marginal distributions of as) under policies and 3, respectively. Substituting this into the previous expression, we obtain:
(J(B) - J(m) < 2
r TV
Now, applying Pinsker's inequality in Lemma 2 we further have
TV e pb) S ( lP77)
Assuming that at every timestep, the KL divergence is bounded as KL (1p7) S s, get:
E
2R
E
Rmax
- J S 2R
22.
2
7 2
1=
This completes the proof.
A.3
Proof of Theorem 2
Proof. Let an ~ (T) and 0, ~ ((s), we begin by applying the Lipschitz continuity of Q(s, a):
IQ(s, as) - Q(s, ax)) S LQ |lag - ax112-
The difference the RHS is typically bounded by the forward KL divergence between these two distributions, i... ||, a2 < D(e), where is the forward KL divergence bound, i.e.,
KL(B[[=) S e. This bound provides an upper limit on the discrepancy between the sampled actions from both policies, reflecting how much the policies differ in terms of their action distributions.
We consider the case that ((s) ΣX" and ΣK wi = Policy <*)) = W( Ex) is a Gaussian. Let A; == A(E,) denote the largest eigenvalue of each component covariance Σ, and let A(E,) denote the largest eigenvalue of Ez-
-:= W,,N(s, Ex), where the mixture weights satisfy 2 O
Consider the decomposition using triangle inequality in Lemma 1
||aş or < max (Iaa pilla + || (μ4-)=l12) + ||a fxx 1|2-
(1) Sampling deviation GMM: Conditional on sampling component w, the action sample ag ~ N(4,, Σ,). For each component, we apply Lemma to obtain the following bound:
|las pill2 S 22A, log K with probability at least - 8.
16

---

(2) Mean shift of GMM components: We now bound the deviation between the GMM component means Hi and the target policy mean Pr. From the definition of KL divergence between Gaussians:
KL (N ( Ex)) 1l M 11
Therefore, if we know that the total mixture KL satisfies KI(B11=) S then by invoking Lemma 5 we have: for each component i, it must be that
| ue - 2
1l - Mxlla S V/ /eA(Ex)/ws.
W2
Hence, plugging this into the deviation bound, we now obtain:
||a 22112 S max ( 22, log K 8 /2EA(En)/wz) || Mz 11 with high probability at least 1 -
(3) Sampling deviation of т: As N( En), by invoking Lemma agaai, we have:
||, Mrll2 S v 2 ( log J.
Putting everything together, with probability at least 1 - 6, we have:
||ag 22l I max (22A, + (2AA(E.)/ws)) + 2A(E,) log 8
Since both ag and a are supported on the normalized action space |-,, 1]9, the maximum possible distance is bounded by 2vd. d. Hence:
||ag ax a2 S min 2Vdd, max (V log $ + /2AA(Ix)/uu) + + 2A(xx)) log
Conclusion. For any å E (0, 1), with probability at least we obtain the bound:
D(e) I min 2vd, max (V2A, log K + /EA((E))/ + 2AA(E,)) log
This completes the proof.
17

---

B Environmental Details
B.1 Benchmark Introduction
We evaluate on the 7 most challenging tasks involving the dog and humanoid d agents. These tasks fall into two categories: (1) Standing tasks, where the agent must maintain upright balance, and (2) Moving tasks, which additionally require the agent to move at target velocity. For moving tasks, the reward is defined the product of the standing reward and the forward velocity reward, i.e., Reward = (Standing reward) x (Forward velocity reward).
Deep Control Suite.
Encourages the agent to maintain an upright posture.
Standing reward:
Forward velocity reward: Ensures the agent moves at the target speed (1 mm/s for dog-walk, m/s for dog-trot, m/s for dog-run, 1 m/s for humanoid walk and 10 m/s for humanoid-run).
Figure 4: Dog
Figure 5: Humanoid
Humanoid Bench. We consider typical locomotion tasks involving a Unitree H1hand robot. This robot is initialized to standing position, with random noise added to all joint positions during each episode reset. Their specific goals are presented below.
Objective. Maintain standing pose.
Reward: a) stable x (0.5 still, + 0.5 x stilly). where the still terms penalize non-zero velocities balance. stable favors maintaining stable and energy-efficient standing status.
encourage stationary
Termination. 1000 steps, or when apelvis < 0.2.
Figure 6: Stand
Objective.
Keep forward velocity close to 1 m/s without falling to the ground.
Reward: R(s, a) = stable tol(Uz, (1, oo), 1).
where tol encourages the agent to maintain forward velocity Vo above 1 mm/s, thereby promoting low-speed locomotion.
Termination. 1000 steps, when Epelvis 0.2.
Figure 7: Walk
Objective.
Keep forward velocity close to 5 m/s without falling the ground.
Reward: R(s, a) stable tol(Uz, (5, 00), 5), where tol encourages the agent to maintain forward velocity V, above 5 5m/s, thereby promoting high-speed locomotion.
Termination. 1000 steps, or when 2pelvis 0.2.
Figure 8: Run
18

---

Objective. Sit onto chair situated closely behind.
Reward: R(s,a) (0.5 sitting_z + 0.5 (ittinngrrttttinnn_xxxppiihtiiie Xposturex mean(still_x, still_y), where is an energy penalty term, sitting_x, sitting-y. and sitting_2 measure the robot's positional tolerance relative to the chair.
Termination. 1000 steps, or when apelvis 0.5.
Figure 9: Sit
Objective. Walk over iterating sequence of upward and downward slides at I m/s.
Reward: R(s,a) = x tol(vz. (1, too), x upright (foot_ x foot_right). where foot_let left and foot_rooseee
measure the vertical distance between the head and left/right foot respectively, ensuring proper foot positioning.
Termination. 1000 steps, when Eproj 0.6.
Figure 10: Slide
Objective. Travel forward over dense forest of high thin poles, without colliding with them.
Reward: R(s, Tcollisian (0.5 x stable + 0.5 X tol(Uz. (oo), 1)), where the collision penalty Voollision equals 0.1 the robot collides with pole, and otherwise.
Termination. 1000 steps, when Epelvis 0.6.
Figure 11: Pole
Objective.
Keep forward velocity close to 5 m/s without falling the ground.
Reward: R(s, a) = stable x tol (vas (5, oo), Toooliisoon which penalizes colliding with hurdle.
Termination. 1000 steps.
Figure 12: Hurdle
B.2
Reproducibility Statement & Detailed Hyperparameters
We base all our experiments on the released official TD-MPC2 codebase https: com/ wocklashansee/tddie tdmppc We adopt their hyperparameter settings without additional tuning and use the same configuration across all previously demonstrated tasks. The details are listed in Table 2 Our core algorithm file and video demos for the most challenging hurdle, pole and slide tasks are accessible lhttps://anonnnott wwt.:e..ppee.iieiioo.. www.aiineeee/eeeeSSS.SSS.. CB07
4open.
In this paper, we evaluate each algorithm for each tasks over three random seeds. The CPU used the AMD Ryzen Threadripper 3960X 24-Core Processor, and the GPU used is NVIDIA GeForce
19

---

RTX 3090Ti. Taking Dog-rim task in the DMC Suite as an example, the time taken to train 2M iterations is around 50 hours.
For SAC and DreamerV3, we report the baseline results released by TD-MPC2, which are obtained from the official repositories of http:://gtthtttb com dew.syarats/ hotteehhhpphtccchhcomom sac and https //github. com/ ctt/e//oo/aaaaaaaaareeaaaaeaaea httpaa/b..c////aaa////////ara/rr https://gi thub. com cww/ http//eertyylliffe//mmmnc and align the settings such number of iterations and evaluation protocol for a fair comparison.
For BMPC, we use the official implementation
Table 2: Hyperparameter settings.
Hyperparameter
Value
Hyperparameter
Value
Training
Learning rate
3 10
Target network update rate
0.5
Encoder learning rale
1 10
Discount factor (r)
0.99
Sample batch size
1
Gradient Clipping Norm Optimizer
20
Replay batch size
256
Adam
Buffer size
10000000
Loss norm
Moving (5%, 95%)
Steps
2_000_000
Sampling
Uniform
World Model
Reward loss coefficient (c.)
0.1
Dynamics loss coefficient (cy) Value functions esemble
20
Value loss coefficient (c²)
0.1
Number of value bins
101
Warmup steps
5000
Planner
MPPI Iterations
||| 20)
Minimum planner Maximum planner std Horizon
0.05
Population size
512
2
Number of clites
64
3
Policy prior samples
24
Actor
Minimum policy log std
-10
Entropy coefficient (o:)
1 x 10
Maximum policy log std
2
Architecture (around 5M parameters in total)
Encoder layers
2
Latent space dimension
512
Encoder dimension
2.66
Task embedding dimension
96
hidden layer dimension
Q function drop out rate
0.01
MLP activation
Mish
MIP Normalization
LayerNorm
C
Supplemental Results
C.1 Illustrative Discussion: Forward KL vs. Reverse KL
To empirically illustrate the differences between forward and reverse KL objectives in aligning an actor policy with a multimodal planner distribution, create toy ID problem for fitting mixture two Gaussians, which simulates a planner policy that captures multiple high- regions. We initialize a unimodal Gaussian policy q(x) ~Nxx o²), and optimize its parameters by minimizing either the forward KL divergence KL(pllg) or the reverse KL divergence (L(gllp).
As shown in Figure the policy trained with forward KL consistently adjusts its mean and variance cover both modes of the planner distribution, demonstrating a mode-covering behavior. In contrast, the reverse KL objective tends to converge only one mode of the distribution, often ignoring the global structure. This highlights mode-seeking nature and reveals the risk of it aligning with suboptimal peak, potentially converging to low-value region.
20

---

0.8
Target Forward
0.6
Densty
0.4
0
Figure 13: Comparison of forward KI. and reverse KI. on a fitting task. This example highlights key issue: when the planner is multimodal but the policy is unimodal, reverse KL may fail to sufficiently cover high-density regions the target distribution, still resulting in significant mismatch and poor performance. In contrast, forward KL provides more stable alignment strategy, promoting broader coverage of the target distribution and preventing premature collapse.
C.2 Visualizations
To demonstrate the effectiveness of BOOM in solving complex, high-dimensional locomotion tasks, provide visualizations of policy control process on three of the most challenging benchmarks in the Humanoid Bench: hurdle, pole, and slide as shown in the following Figure [4 These tasks require precise coordination across many degrees of freedom, long-houe long-horizon reasoning, and dynamic interaction with many objects. The visualization showcase that BOOM not only achieves task success also learns robust behaviors, highlighting strong capabilities difficult control scenarios.
(a) Frame 131
(b) Frame
(c) Frame
(d) Frame 161
Frame 140
(f) Frame 150
(g) Frame 160
Frame 170
(i) Frame 296
(j) Frame 301
(k) Frame 306
(1) Frame 311
Figure 14: Visualizations of BOOM solving three of the most challenging tasks in the Humanoid Bench Bench-burdle, hurdle, pole, and slide.
D Limitation and Future Work
While BOOM demonstrates strong performance in nigh-dimensional continuous control, several limitations remain. First, the use of planner during data collection introduces additional com- putational overhead, as generating and optimizing candidate actions through the world model significantly slower than direct sampling in model-free methods. This can limit overall training throughput, particularly when environment interaction time not the dominant bottleneck. Second, BOOM relies on reasonably accurate dynamics model throughout training. When the model suffers
21

---

from large prediction errors distribution shift, especially over longer rollouts, both planning quality and value estimation may degrade, potentially harming overall performance.
Future work may explore accelerating the computation of online planning process and incorporating uncertainty-aware mechanisms better handle multimodal unreliable planner outputs. Addition- ally, extending BOOM to sparse-reward or real-world robotics settings with noisy observations also presents promising and exciting direction.
E
Positive and Negative Social Impact
Our method, BOOM, enhances the integration of online planning and off-policy learning in RL, leading improved sample efficiency and final performance high-dimensional control tasks. This has positive implications for real-world applications such robotics and autonomous systems, where efficient and stable learning is crucial. By leveraging world models to reduce reliance on physical trials, our approach may also contribute to safer and more cost-effective training processes. However, like many RL technologies, BOOM could be misapplied in sensitive domains such as surveillance autonomous weapons. Additionally, improved simulation-bass basedd efficiency might encourage premature deployment in safety-critical settings. We recommend cautious evaluation and responsible use to ensure the technology is applied ethically.
22

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract clearly outlines the key contributions and scope of the paper. accurately presents (I) the motivation divergence between planner and policy in planning- driven model-based RL. (2) the proposed method-BOOM, which tightly integrates planning and policy learning via a bootstrap loop, and (3) the novel sechiqucs--lkelihooe orreffre divergence loss and soft value-weighted mechanism. It also specifies the evaluation domains (DeepMind Control Suite and Humanoid-Bennd
and the claimed results (state-of offie-of-rie saaee-of-theaette art performance in sample efficiency and final return). These elements are consistent with the core contributions described in the main paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper explicitly discusses several limitations of the proposed approach in Appendix D It acknowledges that the use of a planner during data collection increases computational overhead compared to model-free methods, potentially limiting training throughput. Additionally, the method's reliance on reasonably accurate dynamics model noted as iiiitatton-mode errors or distribution shift can degrade both planning and value estimation, affecting performance. These points demonstrate a clear and honest discussion of the method's constraints.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
23

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed
penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The paper provides two formal theorems that analyze the theoretical guarantees of Bootstrap Alignment. For each result, we clearly state the assumptions (e.g.. bounded reward, discount factor, forward KI. divergence bound, Lipschitz continuity of the Q- function), and the statements are mathematically precise. While the main proofs are deferred to the appendix, the presentation in the main paper outlines the core logic and implications of the theorems.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper includes all necessary details to reproduce the main experimental results: provides comprehensive descriptions of the benchmarks in Appendix [B]] and full hyperparameter settings in B.2 Our core algorithm file is accessible at lhttps:// anonymous , ww........iinice... www.scciiceeeeeeeeeee //Neur/Neuuuuueee IPS INe_BOMMS_BOMMCC5 that readers can independently verify and reproduce the main claims and conclusions, satisfying the reproducibility criterion.
These elements together ensure
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility also be provided via detailed
24

---

instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: This paper provides training curves, and numerical results (Section Addi- tionally, it offers a public repository https://githuu com wwwmmoollnnoooooocooooocmoccoooo taining the core implementation of the proposed algorithm. These elements together ensure that readers can independently verify and reproduce the main claims and conclusions, satisfying the reproducibility criterion.
ht/mmouuuu lumi tw/BO0t MBRL con-
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
jublic/guidess httol////iiddd//deeebbbmasssssss (osscyyyncccy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 7/nips. cc/ httcc/ubblccgguuuee wwwpc//ggeeeeseeee http:///SoooSSSsssiooeeeeeem
stiobbmiissiooooooic.. for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
25

---

Justification: The paper specifies the training and test setup in detail, including benchmark environments (Appendix B.I). full hyperparameter settings (Appendix [B2)). and training curves (Figure It clearly outlines choices such as the type of optimizer used, learning rates, horizon lengths for planning, and network structures. Hyperparameter values and their selection process are disclosed, allowing readers to understand and contextualize the reported performance.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
The paper includes training curves with error bars that reflect the variability across random seeds in Figure which is standard and appropriate for RL benchmarks. It also reports mean and standard deviation values for final performance metrics in Table D providing clear sense of the statistical reliability of the results. These practices offer sufficient information about the significance and robustness of the experimental findings.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper provides sufficient information the computational setup used for the experiments in Section including the type of compute workers (e.g., GPU/CPU), memory specifications, and total training time.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: :ttttp:///eurips.. caaerrssscccpppcc/eeeiee cc/pbllccttiiiiccuuddedleeeeeee
Answer: [Yes]
Justification: We made sure the code was anonymous
4opeen
Bcience/r//eur1159 BPBB0MMMCES
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix B our work.
we discuss the potential positive and negative social impacts
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
27

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The paper properly credits all external assets used, such as open- repositories including Pytorch-RL, DreamerV3 and TD-MPC2, by citing the original authors (Appendix B.2). These assets are used in accordance with their licenses and terms of use. Proper attribution ensures ethical reuse of resources and acknowledges the contributions of prior work, fulfilling this requirement.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, ppppppswpttteeee thoode com com/datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces new assets, including a core implementation file and demonstration videos three challenging tasks
4open. science/ //Neur IPS. IattiroSSSO--MM-- C587). These assets effectively showcase the method's capabilities and support the main experimental claims. Upon acceptance, will release the full codebase along with additional demos.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
28

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:///eerrrrieeec///////eeeees for what should should be described.
29


Reviews:
[{"review_id": "3dZUU0ei9H", "reviewer": "Reviewer_NmT1", "summary": "This paper studies off-policy learning. The paper uses model-based approach to solve the off-policy problem where the sample distribution of the behaviour policy does not match that of the target policy. The world model is a latent dynamic model, learned together with policy optimization. \n\nHowever, this class of planning-driven model-based RL algorithms inevitably suffers from a fun37\ndamental issue known as actor divergence: the data used for learning is collected by the planner,\n which acts as a different actor from the policy network [37].\n\nThis is just the off-policy leaning problem [Sutton& Barton, 1999]. Although not proposed by this paper, why do you need a new name for it? Actor divergence is a confusing name. \n\nthis issue leads to two problems: (1) Distribution shift in value learning: again this is the classical problem of off-policy learning, see [GTD, TDC, TDRC]\n\n\nInevitable Actor DivergenceWhen Off-policy RL Meets Online Planning: this is a confusing subsection title. \n\nSection 3.1: this latent dynamics model started from earlier research, e.g., ACE. There is no literature survey (not even discussions) on world models. This is one shortcoming of the paper. \n\n\nAnother weakness of the paper is that it only compares with DreamerV3, TD-MPC and one of its variants.", "questions": "have you surveyed other model-based DRL algorithms?", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Results on DMC (14 tasks): some domains are very strong, including Humanoid-run, Dog-run, H1hand-walk, etc. However, some domains the advantages are small or tied, such as dog-walk, dog-trot, Dog-stand. \nontinuous Control with Tree Search\nThis is a good baseline to compare against too.  \n\n\nIt’s surprising an off-policy learning missed gradient TD methods, which discusses the sample distribution mismatch, and importance sampling in early RL times. \nGTD:\nhttps://scholar.google.ca/citations?view_op=view_citation&hl=en&user=BXUTo4AAAAAJ&citation_for_view=BXUTo4AAAAAJ:qjMakFHDy7sC\n\nTDC:\n\nhttps://scholar.google.ca/citations?view_op=view_citation&hl=en&user=BXUTo4AAAAAJ&citation_for_view=BXUTo4AAAAAJ:u-x6o8ySG0sC\n\nTDRC:\nhttps://arxiv.org/abs/2007.00611\n\nACE: https://arxiv.org/pdf/1811.02696\nACE: An Actor Ensemble Algorithm for C", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed and constructive feedback. Below are our responses to your concerns.\n# 1. Clarify confusing names\n> This is just the off-policy leaning problem [Sutton& Barton, 1999]. Although not proposed by this paper, why do you need a new name for it? Actor divergence is a confusing name. This issue leads to two problems: (1) Distribution shift in value learning: again this is the classical problem of off-policy learning, see [GTD, TDC, TDRC]. (2) Inevitable Actor DivergenceWhen Off-policy RL Meets Online Planning: this is a confusing subsection title.\n\n**Regarding \"Actor divergence,\"** we understand that for experienced RL researchers, this term may appear redundant with known off-policy challenges. However, in our planning-driven MBRL setting, we face a unique form of divergence: between the policy and the planner—both of which serve as “actors” but generate data from different distributions. Our goal was to make this divergence intuitive to readers less familiar with classical theory. We will explicitly link this terminology to traditional off-policy learning problem to mitigate misleading.\n\n**For the \"Distribution shift in value learning,\"** we fully agree that this is a classical off-policy learning problem, extensively studied in foundational works such as GTD, TDC, and TDRC. We will cite the recommended literature to clarify the connection.\n\n**Finally, regarding the subsection title \"Inevitable Actor Divergence When Off-policy RL Meets Online Planning,\"** our goal was to highlight the intrinsic tension that arises when off-policy RL is combined with model-based planning. We also agree that the title could be clearer—something like “Distribution Shift in Planning-Driven Model-Based RL” may better reflect the core challenge being addressed.\n\n# 2. Discussion of GTD methods\n> It’s surprising an off-policy learning missed gradient TD methods, which discusses the sample distribution mismatch, and importance sampling in early RL times.\n\nWe acknowledge the importance of GTD, TDC, and TDRC for stable off-policy learning with linear function approximation. While these are foundational, our method uses deep RL with nonlinear networks, where accurate TD learning typically relies on empirical tricks such as target networks. We will include citations and briefly discuss these classical approaches for completeness and historical context.\n# 3. Survey on world models\n> Section 3.1: this latent dynamics model started from earlier research, e.g., ACE. There is no literature survey (not even discussions) on world models. This is one shortcoming of the paper.\n\nWe appreciate this point and will expand related works. Below, we categorize world models into three major types based on their objectives and roles in policy learning:\n\n**1. Information-preserving latent models.**\nThese methods aim to learn a fully informative latent representation of the environment by combining with auxiliary reconstruction losses. This ensures that the latent space retains all relevant features from high-dimensional observations, supporting accurate imagination.\nNotably, PlaNet [1] and Dreamer [2] use stochastic recurrent latent dynamics models (e.g., RSSMs), which are trained with an additional decoder to reconstruct the original observations. Although this decoder is discarded at test time, the reconstruction loss regularizes the latent space, improving model stability, generalization. These models are typically used in imagination-driven MBRL, where the policy and value functions are trained entirely on imagined rollouts.\n\n**2. Task-aligned latent models trained via one unified TD loss.**\nThe latent space in these methods is not required to be fully reconstructive; instead, it only needs to encode information task-aligned critical for decision-making, enabling greater efficiency and stronger performance in high-dimensional control. Early examples include ACE [3], VPN [4] and TreeQN [5], which perform latent tree search over value predictions without decoding observations. More recent approaches like TD-MPC [6] adopts a unified TD loss, where the encoder, dynamics, reward, and value functions are jointly optimized. This family of methods is well-suited for planning-driven MBRL, where latent rollouts are used for online planning.\n\n**3. Trajectory-level sequence models for offline RL.**\nA third category treats RL as a sequence modeling problem, without explicitly learning a policy. Instead, models such as Decision Transformer [7] and Trajectory Transformer [8] are trained offline to autoregressively predict future tokens (states, actions, rewards) conditioned on past trajectories and return-to-go.\nThese methods rely on powerful sequence models (e.g., Transformers) to capture behavioral patterns and temporal dependencies. They do not require a separately defined policy network—the Transformer itself acts as a conditional policy, directly outputting the next action given the desired return and current observations. Extending this trend to the pixel level, video generation models like Sora [9] predict future frames directly from visual inputs using large-scale spatiotemporal Transformers. While not trained as agents, they serve as high-capacity world simulators and open new avenues for embodied planning in visually complex domains.\n\n**Our approach** follows the 2nd category: we adopt a latent model from TD-MPC [6] and improve its learning by introducing a bootstrap alignment mechanism that explicitly encourages consistency between the policy and the planner. This consistency suppresses distributional shift and leads to more accurate model learning.\n# 4. Add tree-search baseline\n> Another weakness of the paper is that it only compares with DreamerV3, TD-MPC and one of its variants.\n\n> Continuous control with Tree Search. This is a good baseline to compare against too.\n\nFollowing ACE [3], we implement a multi-actor tree-search variant as a new baseline. Specifically, we rollout multiple candidate trajectories by alternately using multiple actors in a tree-unfolding manner, and select the best one before refinement with MPPI. Results are shown below:\n\n||Dog-run|\n|-|-|\n|BOOM|819.7 ± 2.9|\n|BOOM + Tree-search|817.4 ± 7.4|\n\n*Note we now use 5 seeds, so the result of BOOM is updated*\n\nAs shown, tree search does not bring significant improvements, likely for two main reasons: (1) modern deep RL often uses stochastic policies (e.g., diagonal Gaussian), so multiple actors produce behaviors with limited diversity; (2) even if minor differences exist among actors, the MPPI planner further refines and smooths the actions, resulting in similar outputs. Therefore, tree search yields minimal performance gain while adding additional computational overhead. For these reasons, We choose to adopt a single shared policy for simplicity and efficiency.\n\n# 5. Clarify small advantages on some domains\n> Some domains are very strong, including Humanoid-run, Dog-run, H1hand-walk, etc. However, some domains the advantages are small or tied, such as dog-walk, dog-trot, Dog-stand.\n\nIndeed, simpler tasks such as Dog-stand, Dog-walk, and Dog-trot have lower difficulty (specifically, lower speed requirements) and smaller performance gaps across methods. Baselines already achieve near-optimal returns (>900), leaving little headroom for improvement.\n\nNonetheless, our method maintains performance advantages in these tasks while providing more stable training. Importantly, on the most challenging high-speed run tasks, our approach yields significant gains, highlighting its robustness under harder control demands.\n\n# 6. Survey on other model-based RL methods\n> Have you surveyed other model-based DRL algorithms?\n\nBeyond imagination-based and planning-based methods, another emerging paradigm involves synthesizing training data. These methods generate additional trajectoried or transitions to support policy learning. For example, Diffuser [10] uses a diffusion model to iteratively denoise trajectories for planning. Synthetic Experience Replay (SER) [11] generates additional replay data via diffusion-based upsampling. Prioritized Generative Replay (PGR) [12] improves data efficiency by generating high-value transitions independent of the current policy. However, these approaches often rely on training large generative models and have mostly been applied to low-dimensional tasks (e.g., Cheetah, Reacher), without showing consistent gains over standard planning-based methods. For these reasons, we did not include them in our comparison, though we consider them complementary and promising directions for future work.\n\n[1] D. Hafner et al. *Learning latent dynamics for planning from pixels*. ICML, 2019.\n\n[2] D. Hafner et al. *Mastering diverse control tasks through world models*. Nature, 2025.\n\n[3] S. Zhang et al. *ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search*. AAAI, 2019.\n\n[4] J. Oh et al. *Value prediction network*. NeurIPS, 2017.\n\n[5] G. Farquhar et al. *TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep RL*. ICLR, 2018.\n\n[6] N. Hansen et al. *TD-MPC2: Scalable, Robust World Models for Continuous Control*. ICLR, 2024.\n\n[7] L. Chen et al. *Decision Transformer: Reinforcement Learning via Sequence Modeling*. Arxiv, 2021.\n\n[8] M. Janner et al. *Offline Reinforcement Learning as One Big Sequence Modeling Problem*. NeurIPS, 2021.\n\n[9] Brooks T et al. *Video generation models as world simulators*. OpenAI Blog, 2024.\n\n[10] M. Janner et al. *Planning with diffusion for flexible behavior synthesis*. ICML, 2022.\n\n[11] C. Lu et al. *Synthetic experience replay*. NeurIPS, 2023.\n\n[12] R. Wang et al. *Prioritized generative replay*. ICLR, 2025.\n\n# 7. Conclusion\nWe have clarified the confusing names, added the requested tree-search baselines, and more literature reviews on the world model development and other model-based methods. We hope our responses address your concerns and kindly request your consideration for a higher score."}, {"review_id": "T8C4xWd4qy", "reviewer": "Reviewer_HgMu", "summary": "This paper introduces BOOM (Bootstrap Off-policy with World Model), a framework designed to resolve the \"actor divergence\" problem in planning-driven model-based reinforcement learning (MBRL). The core issue is that the online planner used for data collection acts as a different, more powerful actor than the learned policy, leading to a distributional shift that degrades both value and policy learning. BOOM addresses this through a tight bootstrap loop where the policy initializes the planner, and the planner's refined actions are used to guide the policy via behavior alignment.", "questions": "1. In Figure 4(b), the authors investigated the importance of soft Q-weighting mechanism. Have the authors considered making ablation studies on alternative weighting schemes, such as using the advantage A(s, a), which might provide more stable learning signals?\n\n2. In equation 7, the bound of the expected Q-value depends on $L_Q$. Have the authors considered explicitly or implicitly controlling this value, such as through gradient clipping or a regularization term?\n\n3. Equation (13) in AWAC (https://arxiv.org/abs/2006.09359) proposes a weighted behavior cloning objective that appears similar to Equation (4) in this paper. Could the authors clarify whether these objectives are equivalent or highlight the key differences between them?", "limitations": "Yes, it's in Appendix D.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper provides a clear diagnosis of the actor divergence problem inherent in using online planners for data collection. The proposed solution, which utilizes a likelihood-free forward KL loss to align the policy with the planner's actions, is an elegant and effective way to bridge this gap without requiring the modeling of the complex, non-parametric planner distribution.\n- BOOM demonstrates significant performance gains over strong and recent baselines, including TD-MPC2, BMPC, and DreamerV3.\n\nWeaknesses:\n- The soft Q-weighting mechanism introduces a temperature hyperparameter τ. In similar softmax-based weighting schemes, this parameter can be highly influential, controlling the \"hardness\" of the weighting. The paper lacks an ablation study on τ, leaving its sensitivity unevaluated.", "was_revised": true, "final_justification": "My major concerns about not using the advantage function, but rather soft Q, and the difference to AWAC have been addressed.", "rebuttal": "We sincerely appreciate your detailed and constructive feedback. Below are our responses to your concerns.\n\n# 1. Ablation on the soft Q-weight temperature $\\tau$ (Weakness 1)\n> The soft Q-weighting mechanism introduces a temperature hyperparameter $\\tau$. In similar softmax-based weighting schemes, this parameter can be highly influential, controlling the \"hardness\" of the weighting. The paper lacks an ablation study on $\\tau$, leaving its sensitivity unevaluated.\n\nWe thank you for highlighting the importance of the temperature parameter $\\tau$, which controls the sharpness of the softmax distribution over Q-values. A smaller $\\tau$ results in a distribution closer to hard-max, while a larger $\\tau$ leads to more uniform weighting.\n\nWe conducted an ablation study on the Dog-run task by varying $\\tau$ from 0.1 to 10.0:\n\n|Temperature|TAR|\n|-|-|\n|$\\tau = 0.1$|781.2 ± 8.1|\n|$\\tau = 0.5$|806.3 ± 5.2|\n|$\\tau = 1.0$ (default)| 819.7 ± 2.9|\n|$\\tau = 2.0$|823.4 ± 2.7|\n|$\\tau = 10.0$|799.4 ± 4.6|\n\nThe results show that performance remains stable across a broad range of $\\tau$ values. **Notably, a slightly higher $\\tau$ (e.g., 2.0) can lead to marginal performance improvements. In contrast, setting $\\tau$ too low results in performance degradation due to overly selective.**\n\n# 2. Ablation on the gradient clipping norm (Q2)\n> In equation 7, the bound of the expected Q-value depends on $L_Q$. Have the authors considered explicitly or implicitly controlling this value, such as through gradient clipping or a regularization term?\n\nWe agree that gradient clipping is a practical and effective strategy to mitigate instability during Q-function updates. As suggested, we explicitly investigated its role in our setting. Following TD-MPC2, we initially adopted a clipping norm of 20, as listed in Table 2 of Appendix B.2.\n\nTo evaluate the sensitivity of our method to this choice, we conducted an ablation study on the Dog-run task by varying gradient clipping norm from 2 to 200:\n\n|Gradient clipping norm|TAR|\n|-|-|\n|20 (default)|819.7 ± 2.9|\n|2|813.4 ± 4.7|\n|200|805.6 ± 3.8|\n\nThese results reveal the following:\n- A moderate clipping threshold of 20 yields the best performance and aligns with the default setting in TD-MPC2.\n- A smaller norm (2) still maintains reasonable stability, suggesting some robustness to tighter control.\n- A large norm (200) weakens the effect of clipping, allowing larger gradients that potentially destabilize training, leading to performance degradation.\n\nOverall, this confirms that clipping is an effective and necessary component, and the default choice inherited from TD-MPC2 is already close to optimal.\n# 3. Ablation on the Advantage-weighting mechanism (Q1)\n> In Figure 4(b), the authors investigated the importance of soft Q-weighting mechanism. Have the authors considered making ablation studies on alternative weighting schemes, such as using the advantage A(s, a), which might provide more stable learning signals?\n\nYes, we have explored an alternative weighting scheme based on the advantage function $A(s,a) = Q(s,a) - V(s)$, which is commonly used in actor-critic methods to provide more stable gradients.\n\nSpecifically, we implemented an advantage-weighted variant of our method, where $V(s)$ is estimated by averaging multiple samples from the current policy (python code included for clarity):\n```\n# Approximation of advantage\n# Number of policy samples per state to estimate V more accurately\nnum_pi_samples = 10\n\nwith torch.no_grad():\n    qs = self.model.Q(zs, actions_beta)\n\n    vs_samples = []\n    for _ in range(num_pi_samples):\n        sampled_actions_pi = self.model.pi(zs)\n        vs_samples.append(self.model.Q(zs, sampled_actions_pi))\n    vs = torch.stack(vs_samples, dim=0).mean(dim=0)\n\n    adv = (qs - vs).detach()\n```\n\n|Method|TAR|Training time|\n|-|-|-|\n|ours Q-weight|819.7 ± 2.9|45.6h|\n|Advantage-weight|783.5 ± 4.2|46.7h|\n\nWhile advantage-weighted imitation can offer more stable gradients, we observed a slight ~5% performance drop and a ~2% increase in training time due to additional policy sampling. We hypothesize that this is because advantage emphasizes the relative quality of planner actions compared to current policy actions, leading to a more selective imitation signal. However, **our objective is not only to imitate actions that are strictly better, but to promote consistent alignment between the policy and the planner** over time.\n\nAs online RL progresses, both the policy and planner co-evolve toward optimality. Overly selective weighting can hinder this mutual alignment, especially in early phases where the planner and value are not perfect. This observation is consistent with our earlier ablation on the temperature ($\\tau$) of soft Q-weight, where a smaller $\\tau$ also led to degraded performance due to more hard-max style weighting.\n\nIn summary, **while advantage-weighted imitation offers a more selective filter, our results suggest that such over-selectivity can be counterproductive, leading to degraded performance.** In contrast, the soft Q-weight mechanism provides a better balance between stability and performance. Given its effectiveness and lower overhead, it serves as a practical and sufficient choice, making advantage-based filtering unnecessary in our setting.\n# 4. Comparison with Offline RL algorithm AWAC (Q3)\n> Equation (13) in AWAC (https://arxiv.org/abs/2006.09359) proposes a weighted behavior cloning objective that appears similar to Equation (4) in this paper. Could the authors clarify whether these objectives are equivalent or highlight the key differences between them?\n\n**(1) Conceptual differences:**  \nAWAC is mainly designed for offline RL, aiming to balance behavioral cloning with performance optimization under the constraint of a fixed dataset. By applying Lagrangian relaxation analysis, AWAC derives a theoretical result stating that the optimal policy satisfies: $\\pi^*(a|s) \\propto \\pi_\\beta(a|s)\\exp(\\frac{1}{\\lambda}A(s,a))$. Consequently, at the algorithmic level, AWAC directly relaxes the imitation objective using advantage-based weighting. **This approach constrains the learned policy within the dataset coverage while allowing relaxation of imitation on poor actions, enabling slightly better performance than pure behavioral cloning.**\n\n$$L_\\text{AWAC} = \\Sigma w_A \\text{KL}(\\beta || \\pi)$$\n\nIn contrast, BOOM operates in an online RL setting where fresh planner rollouts are continuously available. Both the policy and planner are jointly optimized toward the optimal solution over time, enabling BOOM to focus more on maintaining alignment, rather than solely on selectively imitating only the highest-advantage actions.\n\n$$L_\\text{BOOM} = Q(s,a_\\pi) + \\lambda_\\text{align} \\Sigma w_Q \\text{KL}(\\beta || \\pi)$$\n\n**(2) Empirical comparison:**  \nSince AWAC fundamentally employs an imitation-style loss similar to BMPC, and BMPC incorporates a re-planning technique that refreshes planner actions in the replay buffer, we extended the comparison by adding re-planning variants for both AWAC and our BOOM (denoted as AWAC + replan and BOOM + replan). This ensures a fair comparison regarding the use of up-to-date planner data for policy alignment.\n\nThe Python code is provided for clarity, with the `awac_lambda` value set to 1.0, following the original paper’s setting.\n\n```\n# AWAC-style weighted imitation loss\nweights = torch.exp(adv / self.cfg.awac_lambda)\nlog_pis_action = self.model.log_pi_action(zs, actions_beta)\npi_loss = (- weights * log_pis_action).mean()\n```\n\n| Method          | TAR       | Training Time    |\n|-----------------|-------------------|------------------|\n| BOOM (ours)     | 819.7 ± 2.9       | 37.2h            |\n| BOOM + replan   | 860.1 ± 2.5       | 56.3h (+51.3%)   |\n| AWAC            | 382.4 ± 37.8      | 39.6h            |\n| AWAC + replan   | 497.3 ± 27.5      | 59.1h (+49.2%)   |\n\n**(3) Results analysis:**\nAs shown in the results, AWAC performs significantly worse than BOOM (The performance gap is so large that we did not add more ablation experiments to the hyper-parameter `awac_lambda`). Although both methods benefit from re-planning—with AWAC exhibiting a larger relative improvement—AWAC still lags far behind BOOM in absolute performance. It is important to note that re-planning incurs approximately 50% additional training time due to the overhead of repeated planning. Remarkably, BOOM achieves strong performance even without re-planning. This advantage likely stems from BOOM’s combined use of Q-function loss and imitation loss, which better supports learning in the online RL setting compared to AWAC’s sole reliance on weighted imitation loss.\n\nIn summary, although AWAC and our BOOM share conceptual similarities, differences in their practical settings and algorithmic designs make AWAC less suitable for our online planning-driven MBRL framework. Notably, BOOM retains the original Q-function loss alongside its particularly designed bootstrap alignment, which enhances learning effectiveness. Therefore, BOOM offers a more effective and computationally efficient approach for maintaining policy-planner consistency and achieving superior performance.\n\nWe appreciate your insightful question and will clarify these distinctions in the revised manuscript.\n# 5. Conclusion\nWe sincerely appreciate your detailed and constructive feedback. We have added the requested experiments on soft Q-weight temperature, grad clipping norm, Advantage-based weights and comparison with AWAC. We hope our responses address your concerns and kindly request your consideration for a higher score."}, {"review_id": "grBIOIqGSw", "reviewer": "Reviewer_HqWU", "summary": "This paper seeks to address the problem of actor divergence in online planning with off-policy RL, because the data for value function update is collected by a behavior policy that is modified upon the control policy with MPPI -type planner. To this end, the authors propose a method called BOOM by jointly training a world model and the policy. The main design component is a policy regularization term for policy alignment which uses likelihood-free KL divergence for policy alignment and a soft Q-weighted mechanism to further change the weights in policy alignment toward high-value actions. Both theoretical and empirical results are provided to justify the effectiveness of the proposed approach.", "questions": "Please see the weaknesses above. While the performance looks promising, it is hard to think of some new technical contributions from this paper to the research community. I would consider to change my score if the novelty can be well justified.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. The idea of using world model for online planning is interesting.\n2. Both theory and experiments are provided.\n3. The presentation is clear.\n\nWeaknesses:\nMy main concern about this paper lies in its limited technical novelty, which can be articulated from the following perspects:\n\n1. World model is an important component proposed in this paper as it can be told from the title. However, it didn't receive much attention in the main body of this paper, and the way to use it seems like a direct plug-in to replace the standard dynamics model in model-based planning.\n2. The entire actor divergence problem is similar to the problem of distribution shift in offline RL. The proposed design is basically a policy regularization by following a very similar way as in offline RL where one tries to regularize the policy update to be closer to the behavior policy for the offline dataset. In particular, the likelihood-free alignment metric in eq.3 has been widely used in offline RL to directly use the offline data without modeling the offline behavior policy, see \n\n- Rasool Fakoor, Jonas Mueller, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. NeurIPS 2021.\n\nThe soft Q-weighted mechanism also shares a similar idea advantage weighted policy regularization in offline RL.", "was_revised": true, "final_justification": "The rebuttal has largely addressed my concerns. While I still have some reservations regarding the significance of the technical modifications, they are an improvement over direct applications of existing techniques. Combined with the contributions in identifying actor divergence and proposing a valid solution, I have updated my rating to 'borderline accept'.", "rebuttal": "We sincerely appreciate your detailed and constructive feedback. Below are our responses to your concerns.\n# 1. How world model learning is improved in BOOM (Weakness 1)\n> World model is an important component proposed in this paper as it can be told from the title. However, it didn't receive much attention in the main body of this paper, and the way to use it seems like a direct plug-in to replace the standard dynamics model in model-based planning.\n\nWe appreciate your comment and would like to clarify the *motivation*, *structure*, and the *improvements to the world model* in our framework.\n\n**Motivation:** Our work investigates the challenges and solutions in integrating off-policy RL with planning-driven world model frameworks. One key challenge is actor divergence: the mismatch between the policy and planner, leading the $(s,a)$ (from planner) and $(s',a')$ (from policy since $a'=\\pi(s')$) in the TD error follows different distributions, which easily leads to **value over-estimation and impairs policy performance.** (*note that value is a core component of our world model*).\nTo mitigate this issue, our proposed BOOM framework leverages a policy-planner alignment mechanism, which not only stabilizes policy learning but also improves the learning of the world model itself. \n\n**Structure:** Concretely, our world model consists of four components:\n- A latent encoder that encodes observations into a compact representation space\n\n- A dynamics model that predicts latent transitions\n\n- A reward predictor that provides rewards used for planning\n\n- A Q-value estimator that enables policy improvement and provides terminal value for planning.\n\n**Improvements:** The reasons why we improve the learning of these components are:\n\n**(1) Q-value learning is improved due to policy-planner consistency.**\nBy aligning the policy with the planner, we reduce the discrepancy between the collected data and the actual policy behavior. This improves the distributional matching during training, enabling the value estimator to learn from more consistent and policy-relevant trajectories. As a result, the predicted values used in planning become more accurate and reliable.\n\n**(2) Encoder, reward, and dynamics learning are all improved due to a unified TD loss.**\nWe adopt a TD-style learning objective that jointly optimizes the value function along with the other components of the world model. \n```\nTD-style world model loss = latent dynamics loss + reward loss + Q-value loss\n```\nAs the value predictions become more accurate due to better distributional match, the gradients flowing into the encoder, dynamics model, and reward predictor become more informative, leading to improved overall model quality.\n\nIn summary, although our world model is based on standard components, our framework improves its learning through designed bootstrap alignment mechanism. These enhancements contribute directly to the improved performance and stability observed in BOOM.\n# 2. Comparison with offline RL (Weakness 2)\n> The entire actor divergence problem is similar to the problem of distribution shift in offline RL. The proposed design is basically a policy regularization by following a very similar way as in offline RL where one tries to regularize the policy update to be closer to the behavior policy for the offline dataset. In particular, the likelihood-free alignment metric in eq.3 has been widely used in offline RL to directly use the offline data without modeling the offline behavior policy\n\nWe thank you for raising this important point regarding the similarity between the actor divergence problem in planning-driven MBRL and the distribution shift challenge in offline RL. While there are some surface-level resemblances—such as the use of policy regularization to keep the updated policy close to a behavior policy—the motivations and practical implications differ fundamentally between the two settings.\n\n**(1) Focus of algorithm design**\n- Offline RL centers on a fixed offline dataset, relying heavily on behavior cloning (BC) to restrict the policy within the dataset’s support. To improve policy performance, offline RL cautiously relaxes BC for poor actions, but always aims to keep the policy close to known data to avoid extrapolation error.\n\n- BOOM centers on the policy and seeks to maximize Q-values and align with planner-generated actions, both aimed at improving policy performance. Ultimately, the policy and planner co-adapt and converge to the optimal solution through online interaction.\n\n**(2) Relationship between two policy objectives** (*max Q and min KL*)\n- In offline RL, maximizing Q-values and BC often conflict: Q-values outside the dataset distribution cannot be accurately estimated, so maximizing Q risks pushing the policy toward unsupported actions; BC pulls the policy back toward known actions. This tension forces a conservative balance.\n\n- In BOOM, the two objectives are largely complementary: the planner generally produces higher-quality actions than the policy. Policy alignment with the planner improves performance and strengthens policy-planner consistency, which in turn leads to more accurate Q estimates. More accurate Q-values then enable better maximization by the policy and allow the planner to generate even higher-quality actions. This positive feedback loop bootstraps policy and planner consistently toward faster convergence to the optimal solution. \n\nThis distinction highlights that BOOM is not an offline method regularized around given dataset, but an active, planner-supervised online RL algorithm that enables consistent and progressive improvement. This is also supported by our ablation experiments on the alignment coefficient. In a large range, the performance is good and stable.\n\n**Concise summary for quick understanding**\n```\n## Offline RL\n1. (Fixed Dataset)\n  Static data collected from a behavior policy.\n\n2. (Policy Regularization) \n  Constraint the policy close to the dataset, while also leaving little room for improvement by maximizing Q-values.\n```\n```\n## BOOM (Online RL)\n1. (Policy)\nAlign with planner's higher-quality actions for better behavior than max-Q alone.\n\n2. (World Model)\nBetter distribution matching improves accuracy.\n\n3. (Planner)\nImproved model results in higher-quality actions for sampling.\n\n4. (Policy)\nHigh-quality data enhances policy training and completes the bootstrap loop.\n```\n\n# 3. Comparison with advantage weighted policy regularization in offline RL (Weakness 3)\n> The soft Q-weighted mechanism also shares a similar idea advantage weighted policy regularization in offline RL.\n\nFor the Advantage-weighting mechanism and offline RL algorithm AWAC, we have supplemented the comparative experiments, ablation analysis and discussion in the **response to (the 2nd reviewer HgMu )**. Results indicate that advantage-weighting is slightly less effective mainly due to over-selectivity. AWAC performs significantly worse, likely because it relies solely on weighted imitation loss, whereas our BOOM benefits from combining Q-value loss with alignment loss, which is better suited for online RL.\n# 4. Clarify contributions (Q1)\n> While the performance looks promising, it is hard to think of some new technical contributions from this paper to the research community. I would consider to change my score if the novelty can be well justified.\n\nThank you for your candid and constructive feedback. While our method builds upon existing techniques in MBRL and offline RL, we believe the primary contribution of our work lies in **clearly identifying and effectively addressing a critical yet under-explored issue in planning-driven MBRL — the divergence between the actor and the planner**. This problem, though fundamental, has not received adequate attention in prior research.\n\nOur contributions are justified as follows:\n\n**(1) Bottleneck identification of planning-based MBRL.**\nWe highlight the actor divergence as a under-explored challenge in planning-based MBRL. While similar to distributional shift issues in offline RL, this problem arises even in fully online and model-based contexts, and has distinct implications for model and policy learning.\n\n**(2) Simple yet effective solution.**\nWe propose a bootstrap mechanism that is easy to implement and integrates seamlessly with existing components, effectively aligning the policy with planner actions to improve stability and performance.\n\n**(3) Shift in focus.**\nRather than improving the structure and training loss of world models or adopting stronger planner, we focus on closing the gap between policy and planner while still maximizing Q-values. This complements existing directions in MBRL and offers a new angle for further performance improvement.\n\n**(4) Strong empirical evidence.**\nWe validate BOOM on multiple challenging benchmarks, where it consistently delivers stable performance gains. These results support our hypothesis that alignment is key to reliable planning-based learning.\n\n**Summary: A minimalist yet powerful approach for planning-driven MBRL.**\nOur method aims to improve planning-driven MBRL through a simple bootstrap alignment mechanism that requires minimal changes to existing components. This straightforward and easy-to-implement approach consistently delivers strong and stable performance gains without introducing complex modifications or additional computational overhead.\n\nWe believe this minimalistic yet effective strategy represents a promising direction for further research. **We genuinely hope this work can serve as a valuable contribution to the community, inspiring others to explore not only similarly simple yet impactful ideas, but also more sophisticated and powerful solutions in planning-driven MBRL.**\n\n# 5. Conclusion\nWe have clarified how model learning benefits from our method, elaborate the difference from offline RL and highlight the contributions. We hope our responses address your concerns and kindly request your consideration for a higher score."}, {"review_id": "sZFZimojqH", "reviewer": "Reviewer_H2KF", "summary": "The paper introduces BOOM, a model-based RL method that improves the integration of planning and off-policy learning. BOOM uses \"bootstrap alignment\" by leveraging non-parametric planner actions for both environment interaction and bootstrapping policy behavior via a Q-weighted likelihood-free alignment loss. This approach mitigates actor divergence, leading to enhanced training stability, and better final performance. Experimental results on various high-dimensional locomotion tasks demonstrate that BOOM consistently outperforms existing planning-driven and imagination-driven baselines.", "questions": "**Q1**: In Eq.(2), action a is extracted from a \"planning-augmented behavior policy,\" yet in Eq.(3) and Algorithm 1, it appears to be sampled from the replay buffer. These methods are distinct, especially as MPPI planning actions evolve throughout training. The author needs to clarify the difference between these two methods.\n\n**Q2**: Following Q1, due to the model bias in the initial world model training, the interaction data stored in the replay buffer is poor. Therefore, I have concerns about the effectiveness of using actions from the replay buffer for alignment. Why not adopt the re-planning approach used in BMPC?", "limitations": "No, see weaknesses.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**: \n\n(1) The paper is well-written, with clear explanations that make the methodology and contributions easy to follow. It also provides solid theoretical justification, which enhances the credibility of the approach.\n\n(2) The proposed method is conceptually simple and easy to implement. It demonstrates superior Total Average Return (TAR) across 7 tasks from the DeepMind Control Suite and 7 tasks from the Humanoid Benchmark. Compared to strong baselines such as DreamerV3 and TD-MPC2, the method consistently shows competitive or even superior performance, highlighting its practical effectiveness.\n\n\n**Weaknesses**: \n\n(1) Limited novelty: The key approach of imitating planner behavior to address actor divergence (Eq.2), as presented in this paper, appears to mirror similar methodologies already employed in BMPC (Eq.6 in BMPC paper). From my perspective, the novelty of the proposed method in this regard is not immediately apparent. The authors should delineate the core contributions of this work that distinguish it from the existing BMPC.\n\n(2) Limited Statistical Validation: Experimental results (Fig. 2, Table 1) based on only three random seeds are insufficient for robust validation. Ablation experiments are confined to a single task, limiting generalizability.", "was_revised": true, "final_justification": "After reviewing the authors’ rebuttal, and conducting my own reproduction experiments, I am maintaining my overall positive assessment with the following justifications:\n\n**Time Efficiency – Partially Addressed**\n\nThe authors clarified that while replanning introduces non-negligible overhead, the use of compiler-level optimizations (e.g., torch.compile) substantially mitigates this cost. As highlighted in my original comments, the training time was reduced from 23.6h to 9.9h (with replanning) and 6.8h (without), demonstrating that the method remains practical even when replanning is used. Although replanning remains an overhead, its impact is significantly reduced in the presence of modern compilation tools — making the cost more manageable in practice. \n\n**Simplicity and Generality — Resolved**\n\nI appreciate the authors' agreement with the clarification and their intention to revise the wording accordingly. This improves the clarity and general appeal of the method.\n\n\n**Performance Evaluation — Partially Resolved**\n\nI acknowledge the concern raised regarding reproduction of BMPC results. That said, I’d like to highlight that even under a reduced training budget (1M steps), the method achieves strong performance on challenging tasks such as Dog-run (731.3) and Humanoid-run (570), which are comparable to or better than those reported in the BMPC paper. While some variation in reproduction is to be expected, I encourage the authors to include more seeds for the BMPC baseline in the revised version to support a fair and robust comparison.\n\n**Novelty Compared to BMPC — Clarified and Accepted**\n\nThe authors clearly articulated their contribution in retaining max-Q for policy improvement, which distinguishes their method from BMPC. While the distinction is somewhat subtle, it is technically sound and worth emphasizing more prominently in the final version.\n\n**Remaining Issues:**\n\nNo major concerns remain unresolved. The few that persist (e.g., number of seeds, clarity of novelty) are relatively minor and do not significantly undermine the contribution.", "rebuttal": "We sincerely appreciate your detailed and constructive feedback. Below are our responses to your concerns.\n# 1. Comparison with BMPC (Weakness 1)\n> Limited novelty: The key approach of imitating planner behavior to address actor divergence (Eq.2), as presented in this paper, appears to mirror similar methodologies already employed in BMPC (Eq.6 in BMPC paper). From my perspective, the novelty of the proposed method in this regard is not immediately apparent. The authors should delineate the core contributions of this work that distinguish it from the existing BMPC.\n\nWe appreciate the opportunity to clarify the differences between our work and BMPC.\nWhile both methods leverage planner-generated actions for imitation in MBRL, our approach is conceptually distinct and practically more efficient in several key aspects:\n\n**1. Time efficiency**  \n- BMPC relies on frequent re-planning during training to obtain up-to-date planner actions for behavior cloning, which introduces significant computational overhead. \n- In contrast, our method employs a bootstrap alignment mechanism that aligns the actor with the stored planner actions, eliminating the need for explicit re-planning. This design shows a clear advantage in time efficiency. (see following **Table 6: Training time with re-planning**).\n\n**2. Simplicity and generality**  \n- BMPC incorporates several auxiliary designs beyond its core planning framework. For instance, it periodically alters `max_policy_std` to control exploration levels. While such heuristics can be effective in specific tasks, they require manual tuning and are sensitive to domain choices, limiting generality and robustness.\n\n- In contrast, our approach focuses on a minimal yet effective alignment mechanism that improves policy quality without introducing domain-specific tricks or altering core components. This simplicity not only makes our method easier to implement and integrate, but also more stable and broadly applicable across tasks.\n\n**3. Performance**  \nEmpirically, our method consistently outperforms BMPC across various tasks, particularly in high-dimensional settings like Dog-run and Humanoid-run. While BMPC is prone to early convergence and unstable updates, our method maintains steady progress and achieves higher final returns with greater learning efficiency.\n\n**Summary**  \n- BMPC reflects a transitional stage in the development of planning-driven MBRL—a design that gestures toward the right direction but remains incomplete. It relies on several loosely motivated heuristics, such as periodically adjusting max_policy_std and fresh planner actions in replay buffer, to promote exploration and maintain training stability. These handcrafted tweaks suggest that BMPC is still evolving and lacks the robustness needed for broader applicability.\n\n- In contrast, our BOOM offers a fully-formed, lightweight solution that seamlessly integrates into existing MBRL frameworks. Without introducing new components or tuning-sensitive mechanisms, we achieve consistent gains in stability and performance. We believe this kind of alignment-based strategy not only improves practical usability but also opens up a promising direction for more systematic advancement in planner-policy integration.\n# 2. Improved statistical validation (Weakness 2)\n> Limited Statistical Validation: Experimental results (Fig. 2, Table 1) based on only three random seeds are insufficient for robust validation.\n\nWe have increased the number of random seeds from 3 to 5 for our method on all benchmark tasks. Due to time constraints, we were only able to extend the seed count for our method rather than all baselines. The updated results are shown in the following table, and our method still consistently outperforms baselines with slightly reduced variance.\n \n**Table 1: Benchmark with increased number of seeds**  \n|Seeds|Humanoid-stand|Humanoid-walk|Humanoid-run|Dog-stand|Dog-walk|Dog-trot|Dog-run|AVG.DMCSuite|H1hand-stand|H1hand-walk|H1hand-run|H1hand-sit|H1hand-slide|H1hand-pole|H1hand-hurdle|AVG.H-Bench.|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|3|962.1 ± 10.7|936.1 ± 3.3|582.8 ± 26.0|986.8 ± 1.8|965.4 ± 0.3|947.9 ± 4.7|821.5 ± 3.8|876.7 ± 8.4|926.1 ± 19.2|935.4 ± 7.3|682.2 ± 120.6|918.1 ± 4.2|926.1 ± 8.0|930.5 ± 18.9|435.6 ± 29.8|820.6 ± 27.7|\n|5|963.5 ± 9.2|938.3 ± 2.8|578.6 ± 20.4|988.2 ± 1.4|964.1 ± 0.2|949.2 ± 3.8|819.7 ± 2.9|877.4 ± 6.0|928.2 ± 15.8|936.8 ± 6.1|684.4 ± 95.2|919.3 ± 3.5|927.4 ± 6.3|932.1 ± 15.6|417.4 ± 32.7|818.9 ± 26.1|\n\nNotably, the only task where performance dropped was the hurdle, which requires the humanoid to continuously jump over multiple hurdles. We observed that some seeds resulted in the agent missing 1–2 jumps and falling prematurely, leading to relatively low scores around 350–400, while successful seeds achieved 450–500. The overall average across 5 seeds is 417, which, despite some instability, still significantly outperforms prior baselines—BMPC achieves only 197.1 ± 12.1, and Dreamer (10M) only 135.7 ± 6.1.\n# 3. Add ablation study (Weakness 2)\n> Ablation experiments are confined to a single task, limiting generalizability.\n\nWe conduct additional ablation experiments on the challenging H1hand-slide task from Humanoid-Bench, which complements the original ablation on Dog-run.\n\n**Table 2: Ablation on the alignment metric**  \n||Dog-run|H1hand-slide|\n|-|-|-|\n|Forward (default)|819.7 ± 2.9|927.4 ± 6.3|\n|Reverse|631.1 ± 29.4|860.0 ± 24.1|\n\n**Table 3: Ablation on the Q-weight mechanism**  \n||Dog-run|H1hand-slide|\n|-|-|-|\n|W/ Q-weight (default)|819.7 ± 2.9|927.4 ± 6.3|\n|W/O Q-weight|738.2 ± 17.0|840.0 ± 32.5|\n\n**Table 4: Ablation on the alignment coefficient**  \n||Dog-run|H1hand-slide|\n|-|-|-|\n|×1 (default)|819.7 ± 2.9|927.4 ± 6.3|\n|×0.1|760.6 ± 31.3|875.0 ± 17.2|\n|×10|728.7 ± 44.1|932.5 ± 5.0|\n\nAs shown, the H1hand-slide involves higher-dimensional action space and thus requires relatively larger alignment coefficient. Overall, the trend still aligns well with our original claim about the effectiveness of the alignment mechanism. We believe these additional ablation study strengthen the empirical robustness of our claims.\n\n# 4. Clarify the action in Eq.(2) and Eq.(3) (Q1)\n>  In Eq.(2), action a is extracted from a \"planning-augmented behavior policy,\" yet in Eq.(3) and Algorithm 1, it appears to be sampled from the replay buffer. These methods are distinct, especially as MPPI planning actions evolve throughout training. The author needs to clarify the difference between these two methods.\n\nTo clarify: Eq.(2) reflects our theoretical design — in an ideal setting, every training sample from the replay buffer would be paired with a freshly re-planned action generated using the current world model and policy. This corresponds to what we refer to as the \"planning-augmented behavior policy.\"\n\nHowever, in practice, continuously re-generating planner actions for all replay samples during every policy update introduces significant computational overhead. To address this, we adopt a practical compromise: when collecting experience, we store the MPPI-generated planner action into the replay buffer alongside the transition. As shown in Eq.(3), during training, we directly imitate these stored actions instead of re-planning on the fly.\n\nWhile this may lead to some degradation due to action staleness, we mitigate this by applying a soft Q-weight mechanism. This allows to focus more on high-quality historical behaviors, reducing the impact of suboptimal actions.\n\nWe apologize for the confusion and will add a dedicated “Practical Implementation Details” section in the revision to make this distinction explicit.\n# 5. More experiments and discussions on the re-planning mechanism (Q2)\n> Following Q1, due to the model bias in the initial world model training, the interaction data stored in the replay buffer is poor. Therefore, I have concerns about the effectiveness of using actions from the replay buffer for alignment. Why not adopt the re-planning approach used in BMPC?\n\nWe fully agree that actions stored in the replay buffer may be suboptimal, particularly due to model bias in the early stages. To evaluate the benefit of using up-to-date planner actions, we incorporated a re-planning mechanism from BMPC into our BOOM framework. Specifically, during every \\(k=10\\) policy updates, we select \\(b=20\\) samples from each training batch (batch size 256), and regenerate their corresponding planner actions using the current policy and world model. The re-planned action will replace the old action in the buffer.\n\n\n**Table 5: TAR with re-planning**  \n||Dog-run|H1hand-slide|\n|-|-|-|\n|ours|819.7 ± 2.9|927.4 ± 6.3|\n|w/ re-plan|860.1 ± 2.5 (+4.5%)|939.5 ± 5.4 (+1.3%)|\n\n**Table 6: Training time with re-planning**  \n||Dog-run|H1hand-slide|\n|-|-|-|\n|ours|37.2h|45.6h|\n|w/ re-plan|56.3h (+51.3%)|71.9h (+57.7%)|\n\nThis introduction of re-planning led to a consistent improvement in performance—from 819.7 to 860.1 TAR on the dog-run task—suggesting that aligning with fresh actions from the current model-planner pair can indeed enhance training. However, we also observed a notable trade-off: the overall training time increased by approximately 50% percent due to additional rollouts and planning.\n\nWe appreciate your suggestion, as it helped us identify a meaningful performance gain. However, we believe the increased time cost may limit the practicality of frequent re-planning in larger-scale or real-time applications. **Given the relatively modest improvement and the added computational cost, we believe re-planning is not strictly necessary. That said, if performance is the top priority, it remains a reliable trick worth considering.**\n# 6. Conclusion\nAfter extensive effort, we have added the requested more experiments with more seeds, clarified the difference from BMPC, and add experiments employing the re-planning mechanism from BMPC. We hope our responses address your concerns and kindly request your consideration for a higher score."}]

Author rebuttal:

Meta-review: The paper introduces BOOM (Bootstrap Off-policy with World Model), a model-based reinforcement learning (MBRL) framework designed to mitigate the actor divergence problem in planning-driven RL. Actor divergence refers to the mismatch between the behavior data collected using a planner (often more powerful than the learned policy) and the policy itself, leading to distribution shift and degraded policy/value learning. BOOM addresses this by (i) bootstrapping the planner from the current policy, and (ii) aligning the policy toward the planner through a likelihood-free KL regularization, with additional soft Q-weighting that prioritizes high-value actions. The authors provide theoretical analysis supporting their formulation and demonstrate empirical results across tasks from DeepMind Control Suite and Humanoid benchmarks. The method shows competitive or superior performance against strong baselines such as DreamerV3, TD-MPC2, and BMPC.

**Strengths** The reviewers agree that the paper provides a clear articulation of the mismatch between planning and policy learning, a central issue in planning-driven MBRL. The proposed bootstrap alignment mechanism—initializing the planner from the policy and then aligning the policy back toward planner actions—is conceptually simple, easy to implement, and empirically effective in improving stability and performance. In addition, the paper includes a theoretical analysis of the policy regularization design, which strengthens the overall contribution.

**Weaknesses** Several reviewers noted that the approach shows strong parallels to offline RL methods, particularly regularized policy learning with likelihood-free metrics and advantage weighting, as well as to BMPC, which also aligns the policy toward planner behavior. This raises questions about the degree of incremental novelty. The submission also lacks sufficient discussion of connections to advantage-weighted mechanism in AWAC. Experimental evaluation was limited in scope, with comparisons restricted to only a few baselines (DreamerV3, TD-MPC2, BMPC) while omitting other relevant algorithms such as continuous control with tree search. Additional concerns included the use of only three random seeds, the absence of hyperparameter sensitivity analysis, and the design choice regarding whether planner actions were sampled during training or drawn from the replay buffer for alignment.

The authors addressed many of these concerns during the rebuttal. They clarified the distinction between their use of soft Q-weighting and advantage functions, as well as the differences from AWAC. They also added comparisons with a multi-actor tree-search baseline and explained why BOOM outperforms. The number of random seeds was increased from 3 to 5 across all benchmark tasks to validate the statistical significance of the results. Additional ablations were provided, including an analysis of the temperature parameter $\tau$ in soft Q-weighting and a comparison of using replay buffer actions versus replanning for alignment, which alleviated the concerns of the novelty claim relative to BMPC.

There is broad consensus that the paper is technically solid, clearly written, and supported by strong empirical results. For the revision, the authors are encouraged to better situate the contribution relative to prior work (particularly BMPC and AWAC) and expand experimental validation.
Decision: Accept (poster)