Title: World Models as Reference Trajectories for Rapid Motor Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Motor Adaptation

Model-based RL

Rapid Adaptation

Model Reference Control

Data Efficiency

Naturalistic Motor Control
Abstract: Learned control policies often fail when deployed in real-world environments with changing dynamics. When system dynamics shift unexpectedly, performance degrades until models are retrained on new data. We introduce Reflexive World Models (RWM), a dual control framework that uses world model predictions as implicit reference trajectories for rapid adaptation. Our method separates the control problem into long-term reward maximization through reinforcement learning and robust motor execution through reward-free rapid control in latent space. This dual architecture achieves significantly faster adaptation with low online computational cost compared to model-based RL baselines, while maintaining near-optimal performance. The approach combines the benefits of flexible policy learning through reinforcement learning with rapid error correction capabilities, providing a theoretically grounded method for maintaining performance in high-dimensional continuous control tasks under varying dynamics.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Reflexive World Model (RWM), a model-based control method designed to rapidly adapt to perturbed dynamics. \nRWM trains a residual correction online by minimizing the model prediction error (finding an action that better aligns the predicted next state with the \"intended\" next state).  \nWhen applied on top of TD-MPC2, RWM demonstrates improved adaptation performance under dynamic perturbations in the Walker2D environment. \nAlso, the authors provide a theoretical analysis on the error bound of the corrected policy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n**S1. Novelty**\n* To the best of my knowledge, the proposed approach largely differs from prior methods, which typically focus on adapting the world model, rely on historical context or domain randomization. \n* In contrast, RWM’s focus on directly adjusting the action to achieve a desired next state simplifies the adaptation process and efficient than re-learning a new model and policy.\n\n**S2. Simplicity**\n* The motivation and implementation is clean and simple.\n* It does not require significant overhead during online adaptation.\n\n**S3. Theoretical Analysis**\n* Section 4 provides a theoretical bound on the error of the corrected policy, supporting the effectiveness of the proposed method.\n\n&nbsp;\n\n**Weaknesses**\n\n**W1. Limited Evaluation**\n* The method is mainly evaluated on the Walker2D environment. \n* Additional experiments on other control benchmarks (e.g., HalfCheetah or more challenging settings like Humanoid) would strengthen the paper’s claims and demonstrate broader applicability.\n\nQUESTIONS:\n**Q1. More evaluation**\n* Could you include results of environments (e.g., Cheetah or Humanoid) for Figure 2 and Figure 3(a)? \n* This would provide a clearer picture of how RWM scales to high-dimensional, challenging systems.\n\n**Q2. Clarification on Figure 5**\n* In Figure 5, in the region without pertubation, the graph on the right shows that TD-MPC2 with domain randomization substantially outperforms RWM, whereas the performance appears similar in the left. \n* Could you clarify the discrepancy between these two views?\n\n**Q3. Mismatch of the line colors in the caption of Figure 2, 5**\n* It appears that the line colors described in the captions for Figure 2 and Figure 5 do not match the colors shown in the graphs. \n* Could you double-check and correct this?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n### Summary\n\nThis paper makes the following key contributions:\n\n1. It proposes Reflexive World Models (RWM), a method designed to enable model-based control policies to rapidly adapt to specific types of perturbations.\n\n2. It provides a theoretical error analysis that offers insight into the reliability and limitations of the proposed approach.\n\n3. It conducts a series of experiments demonstrating the effectiveness and robustness of RWM across various scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The paper provides a theoretical analysis to support its claims.\n\n2. The core idea is simple and practical, while also exhibiting a degree of novelty.\n\n---\n\n### Weaknesses\n\n1. The paper lacks a formal problem definition. Aside from a brief description around line 85, there is no rigorous formulation of the task. It would be helpful to include a dedicated problem definition section before Section 3. Moreover, since the proposed method focuses on optimizing a policy to match the world model's predictions, it is unclear what the exact objective is. Is the goal to train a policy to align with the outputs of a reliable, pre-trained world model? Or is it to maximize the accumulated reward in the traditional reinforcement learning sense?\n\n2. Several variables and functions are used without prior definition or clear explanation, including but not limited to: $V(\\cdot)$ in line 87, the base policy $\\pi_0$ in line 105, $\\theta_c$, $\\eta_c$, and $a_c$ in Equation (4); and $L$, $\\alpha$, $P$, $\\epsilon$, and $f$ in Assumption 4.1. In Figure 1, $Q_t$ and $s^*$ also appear undefined. While most of these can be inferred from context or domain conventions, the lack of explicit definitions significantly hinders readability.\n\n3. Some of the notation choices may lead to confusion. For example, $e$ is used to denote an encoder, but $e_t$ later refers to a loss value. It is unclear why the loss is denoted as $e_t$ in Algorithm 1 when $\\mathcal{L}$ has already been defined in Equation (3), and why a different notation $e(t)$ appears again in line 175. Unifying or clarifying these symbols would improve clarity.\n\n4. Beyond the mathematical content, the overall narrative of the paper could be clearer. If the main goal is to eliminate noise introduced by action perturbation, then the claim in line 143 that the approach “differs fundamentally from standard practice” seems overstated. In control literature, using neural networks to filter out noise is quite common. The contrast with model-based RL methods seems intended to highlight the novelty, but this distinction could be explained more precisely.\n\n5. The experimental evaluation lacks persuasiveness. Only one type of perturbation is considered—a simple multiplicative factor. Furthermore, the target world model is assumed to be pre-trained and near-optimal, which simplifies the learning objective: $a_c$ essentially only needs to approximate $a_c = -a_0 / (1 + p)$. The comparison baselines are also limited: one is a naive “no adaptation” baseline, and the other is continued RL training. The latter comparison, in particular, may be unfair, as it contrasts a supervised adaptation method with a sample-inefficient RL retraining baseline. As such, the empirical results offer limited insight into the real advantages of the proposed method.\n\nQUESTIONS:\n1. According to the definition in Equation (2), is your forward model $F$ only capable of predicting hidden states that evolve under $\\pi_0$? Or is it trained to fit a general transition model $F(z, a)$ independent of the specific policy?\n\n2. In line 100, you claim that a certain quantity is positive definite. However, it appears to be only semi-positive definite. Additionally, I found the narrative in Section 3.1 somewhat difficult to follow. The logical flow from line 100 to 102 is not very clear. While I can accept that $\\Delta z$ is small, if $z$ is already assumed to be near the optimum, it’s unclear why this optimization problem remains necessary.\n\n3. A minor formatting issue: none of the section or equation references in the paper appear as hyperlinks, and all references to the appendix lack section numbers. This makes navigation and reading somewhat inconvenient.\n\n4. A key underlying assumption for the validity of Equation (4) seems to be that $F(z_t, a)$ and $f(z_t, a) = e(P(s, a))$ are sufficiently close. Perhaps Theorem 4.2 is intended to justify this, but due to missing definitions, it is difficult to fully understand the statement and implications of Theorem 4.2.\n\n5. In Algorithm 1, the loop appears to run indefinitely. What is the intended stopping criterion? Is it a fixed number of epochs, a loss threshold, or perhaps a condition on gradient magnitude?\n\nLIMITATIONS:\n1. As noted in the weaknesses, the experimental setting is overly simplistic, which limits the generalizability and depth of the evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Reflexive World Models (RWM), a dual-controller framework where learned world model predictions are used as reference trajectories for adaptive control. A Taylor decomposition of the value function motivates the split between slow policy learning and fast motor correction. The method is theoretically grounded and well-motivated, bridging adaptive control and reinforcement learning.\nHowever, the experimental section is underwhelming in scope and clarity. Key claims made in the introduction—particularly regarding generalization to nonstationary environments and robustness across tasks—are not adequately supported by the experiments. As a result, while the theoretical contribution is strong, the empirical validation does not match the ambition of the paper’s framing, and some of the broader claims appear overstated.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n- The core idea of treating world model rollouts as reference trajectories for rapid online correction is novel and conceptually clean.\n- The use of a second-order Taylor expansion to decompose the value function provides a solid theoretical foundation.\n- The approach avoids costly retraining and replanning and is lightweight at inference time.\n\n### **Weaknesses**\n- **Experimental section is disorganized** and hard to follow. It lacks a clear introduction, grouping, or interpretation of results. Readers are left piecing together the setups from fragmented descriptions (e.g., point mass in §5.2 appears without prior context).\n- **Limited environments**: Results are only shown on a basic 2D point mass and Walker2D and humanoid. Why no more tasks? For a claim about general-purpose adaptation in high-dimensional continuous control, this is too narrow.\n- Claims about generality (e.g., handling nonstationary dynamics or high-dimensional tasks) are not well supported.\n- **No comparison to alternative adaptation methods** (e.g., residual policy learning, meta-RL, or even classic MRAC). Only comparisons are to a TD-MPC2 baseline and no-adaptation (such as Online fine-tuning or Meta-RL baselines). MRAC baseline should be there as you mention it in the related work.\n- **Misleading claims** about nonstationarity: the only variation tested is **multiplicative noise on actuators**. This is not equivalent to structural changes like terrain variation (e.g., slippery surfaces), morphology changes, or sensor shifts.\n- Table 1 is unclear: it's not evident what \"TD-MPC2 (Full Training)\" refers to in contrast to \"No Adaptation,\" nor how the timing aligns with pretraining vs adaptation phases. While you do clarify this in the appendix, the main text should do a better job at surfacing this distinction. Without it, the implication that RWM is universally cheaper is misleading.\n- In the introduction. \"enabling learned policies to maintain performance under changing dynamics without requiring specific robustness procedures or architectural constraints.\". I don't agree with this. You show only robustness to noise on actuators.\n- Authors claim \"avoid costly replanning\". But it is not clear in the manuscript if there is a performance drop (see questions). Are you disabling planning in TDMPC2?\n- The experiments compare RWM (using a frozen TD-MPC2 policy with no online planning) to TD-MPC2 with full MPC. While RWM performs better under perturbations, this comparison is not entirely fair, since TD-MPC2 is not equipped with an adaptive controller. It would be more informative to evaluate RWM against TD-MPC2 with adaptive fine-tuning, or robust planning variants.\n-  In §5.2 they refer to a \"point-mass system\" without any prior introduction.\n\nQUESTIONS:\n- TD-MPC2 configuration unclear: The experiments compare RWM against a TD-MPC2 baseline, but it is not stated clearly whether TD-MPC2 is run with online planning or not. Since planning is a central component of TD-MPC-style algorithms, this distinction is critical. If the baseline is run without planning, its performance may be substantially handicapped. If it is run with planning, this should be stated more clearly in the main text and captioned in relevant figures.\n- What exactly is \\pi_0? As I understand it, \\pi_0 is the frozen policy from TD-MPC2 trained offline without planning (i.e., a reactive policy head). Can the authors confirm this? And is it accurate that this feedforward policy alone is used as the baseline in “No Adaptation”?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}