Title: Bootstrap Off-policy with World Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: Model-based Reinforcement Learning

Online Planning
Abstract: Online planning has proven effective in reinforcement learning (RL) for improving sample efficiency and final performance. However, using planning for environment interaction inevitably introduces a divergence between the collected data and the policy's actual behaviors, degrading both model learning and policy improvement. To address this, we propose BOOM (Bootstrap Off-policy with WOrld Model), a framework that tightly integrates planning and off-policy learning through a bootstrap loop: the policy initializes the planner, and the planner refines actions to bootstrap the policy through behavior alignment. This loop is supported by a jointly learned world model, which enables the planner to simulate future trajectories and provides value targets to facilitate policy improvement. The core of BOOM is a likelihood-free alignment loss that bootstraps the policy using the planner’s non-parametric action distribution, combined with a soft value-weighted mechanism that prioritizes high-return behaviors and mitigates variability in the planner’s action quality within the replay buffer. Experiments on the high-dimensional DeepMind Control Suite and Humanoid-Bench show that BOOM achieves state-of-the-art results in both training stability and final performance. The code is accessible at \url{https://github.com/molumitu/BOOM_MBRL}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces BOOM, a model-based RL method that improves the integration of planning and off-policy learning. BOOM uses \"bootstrap alignment\" by leveraging non-parametric planner actions for both environment interaction and bootstrapping policy behavior via a Q-weighted likelihood-free alignment loss. This approach mitigates actor divergence, leading to enhanced training stability, and better final performance. Experimental results on various high-dimensional locomotion tasks demonstrate that BOOM consistently outperforms existing planning-driven and imagination-driven baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: \n\n(1) The paper is well-written, with clear explanations that make the methodology and contributions easy to follow. It also provides solid theoretical justification, which enhances the credibility of the approach.\n\n(2) The proposed method is conceptually simple and easy to implement. It demonstrates superior Total Average Return (TAR) across 7 tasks from the DeepMind Control Suite and 7 tasks from the Humanoid Benchmark. Compared to strong baselines such as DreamerV3 and TD-MPC2, the method consistently shows competitive or even superior performance, highlighting its practical effectiveness.\n\n\n**Weaknesses**: \n\n(1) Limited novelty: The key approach of imitating planner behavior to address actor divergence (Eq.2), as presented in this paper, appears to mirror similar methodologies already employed in BMPC (Eq.6 in BMPC paper). From my perspective, the novelty of the proposed method in this regard is not immediately apparent. The authors should delineate the core contributions of this work that distinguish it from the existing BMPC.\n\n(2) Limited Statistical Validation: Experimental results (Fig. 2, Table 1) based on only three random seeds are insufficient for robust validation. Ablation experiments are confined to a single task, limiting generalizability.\n\nQUESTIONS:\n**Q1**: In Eq.(2), action a is extracted from a \"planning-augmented behavior policy,\" yet in Eq.(3) and Algorithm 1, it appears to be sampled from the replay buffer. These methods are distinct, especially as MPPI planning actions evolve throughout training. The author needs to clarify the difference between these two methods.\n\n**Q2**: Following Q1, due to the model bias in the initial world model training, the interaction data stored in the replay buffer is poor. Therefore, I have concerns about the effectiveness of using actions from the replay buffer for alignment. Why not adopt the re-planning approach used in BMPC?\n\nLIMITATIONS:\nNo, see weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper seeks to address the problem of actor divergence in online planning with off-policy RL, because the data for value function update is collected by a behavior policy that is modified upon the control policy with MPPI -type planner. To this end, the authors propose a method called BOOM by jointly training a world model and the policy. The main design component is a policy regularization term for policy alignment which uses likelihood-free KL divergence for policy alignment and a soft Q-weighted mechanism to further change the weights in policy alignment toward high-value actions. Both theoretical and empirical results are provided to justify the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The idea of using world model for online planning is interesting.\n2. Both theory and experiments are provided.\n3. The presentation is clear.\n\nWeaknesses:\nMy main concern about this paper lies in its limited technical novelty, which can be articulated from the following perspects:\n\n1. World model is an important component proposed in this paper as it can be told from the title. However, it didn't receive much attention in the main body of this paper, and the way to use it seems like a direct plug-in to replace the standard dynamics model in model-based planning.\n2. The entire actor divergence problem is similar to the problem of distribution shift in offline RL. The proposed design is basically a policy regularization by following a very similar way as in offline RL where one tries to regularize the policy update to be closer to the behavior policy for the offline dataset. In particular, the likelihood-free alignment metric in eq.3 has been widely used in offline RL to directly use the offline data without modeling the offline behavior policy, see \n\n- Rasool Fakoor, Jonas Mueller, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. NeurIPS 2021.\n\nThe soft Q-weighted mechanism also shares a similar idea advantage weighted policy regularization in offline RL.\n\nQUESTIONS:\nPlease see the weaknesses above. While the performance looks promising, it is hard to think of some new technical contributions from this paper to the research community. I would consider to change my score if the novelty can be well justified.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces BOOM (Bootstrap Off-policy with World Model), a framework designed to resolve the \"actor divergence\" problem in planning-driven model-based reinforcement learning (MBRL). The core issue is that the online planner used for data collection acts as a different, more powerful actor than the learned policy, leading to a distributional shift that degrades both value and policy learning. BOOM addresses this through a tight bootstrap loop where the policy initializes the planner, and the planner's refined actions are used to guide the policy via behavior alignment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper provides a clear diagnosis of the actor divergence problem inherent in using online planners for data collection. The proposed solution, which utilizes a likelihood-free forward KL loss to align the policy with the planner's actions, is an elegant and effective way to bridge this gap without requiring the modeling of the complex, non-parametric planner distribution.\n- BOOM demonstrates significant performance gains over strong and recent baselines, including TD-MPC2, BMPC, and DreamerV3.\n\nWeaknesses:\n- The soft Q-weighting mechanism introduces a temperature hyperparameter τ. In similar softmax-based weighting schemes, this parameter can be highly influential, controlling the \"hardness\" of the weighting. The paper lacks an ablation study on τ, leaving its sensitivity unevaluated.\n\nQUESTIONS:\n1. In Figure 4(b), the authors investigated the importance of soft Q-weighting mechanism. Have the authors considered making ablation studies on alternative weighting schemes, such as using the advantage A(s, a), which might provide more stable learning signals?\n\n2. In equation 7, the bound of the expected Q-value depends on $L_Q$. Have the authors considered explicitly or implicitly controlling this value, such as through gradient clipping or a regularization term?\n\n3. Equation (13) in AWAC (https://arxiv.org/abs/2006.09359) proposes a weighted behavior cloning objective that appears similar to Equation (4) in this paper. Could the authors clarify whether these objectives are equivalent or highlight the key differences between them?\n\nLIMITATIONS:\nYes, it's in Appendix D.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies off-policy learning. The paper uses model-based approach to solve the off-policy problem where the sample distribution of the behaviour policy does not match that of the target policy. The world model is a latent dynamic model, learned together with policy optimization. \n\nHowever, this class of planning-driven model-based RL algorithms inevitably suffers from a fun37\ndamental issue known as actor divergence: the data used for learning is collected by the planner,\n which acts as a different actor from the policy network [37].\n\nThis is just the off-policy leaning problem [Sutton& Barton, 1999]. Although not proposed by this paper, why do you need a new name for it? Actor divergence is a confusing name. \n\nthis issue leads to two problems: (1) Distribution shift in value learning: again this is the classical problem of off-policy learning, see [GTD, TDC, TDRC]\n\n\nInevitable Actor DivergenceWhen Off-policy RL Meets Online Planning: this is a confusing subsection title. \n\nSection 3.1: this latent dynamics model started from earlier research, e.g., ACE. There is no literature survey (not even discussions) on world models. This is one shortcoming of the paper. \n\n\nAnother weakness of the paper is that it only compares with DreamerV3, TD-MPC and one of its variants.\n\nSTRENGTHS AND WEAKNESSES:\nResults on DMC (14 tasks): some domains are very strong, including Humanoid-run, Dog-run, H1hand-walk, etc. However, some domains the advantages are small or tied, such as dog-walk, dog-trot, Dog-stand. \nontinuous Control with Tree Search\nThis is a good baseline to compare against too.  \n\n\nIt’s surprising an off-policy learning missed gradient TD methods, which discusses the sample distribution mismatch, and importance sampling in early RL times. \nGTD:\nhttps://scholar.google.ca/citations?view_op=view_citation&hl=en&user=BXUTo4AAAAAJ&citation_for_view=BXUTo4AAAAAJ:qjMakFHDy7sC\n\nTDC:\n\nhttps://scholar.google.ca/citations?view_op=view_citation&hl=en&user=BXUTo4AAAAAJ&citation_for_view=BXUTo4AAAAAJ:u-x6o8ySG0sC\n\nTDRC:\nhttps://arxiv.org/abs/2007.00611\n\nACE: https://arxiv.org/pdf/1811.02696\nACE: An Actor Ensemble Algorithm for C\n\nQUESTIONS:\nhave you surveyed other model-based DRL algorithms?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}