World Models as Reference Trajectories for Rapid Motor Adaptation
Authors: Carlos Stein Brito
Daniel C McNamee
Venue: NeurIPS 2025 poster ()
Keywords: Motor Adaptation
Model-based RL
Rapid Adaptation
Model Reference Control
Data Efficiency
Naturalistic Motor Control
Primary area: reinforcement_learning
TL;DR: Dual control framework that uses world model predictions as implicit reference trajectories for rapid adaptation

Abstract: Learned control policies often fail when deployed in real-world environments with changing dynamics. When system dynamics shift unexpectedly, performance degrades until models are retrained on new data. We introduce Reflexive World Models (RWM), a dual control framework that uses world model predictions as implicit reference trajectories for rapid adaptation. Our method separates the control problem into long-term reward maximization through reinforcement learning and robust motor execution through reward-free rapid control in latent space. This dual architecture achieves significantly faster adaptation with low online computational cost compared to model-based RL baselines, while maintaining near-optimal performance. The approach combines the benefits of flexible policy learning through reinforcement learning with rapid error correction capabilities, providing a theoretically grounded method for maintaining performance in high-dimensional continuous control tasks under varying dynamics.

World Models as Reference Trajectories for Rapid Motor Adaptation
Carlos Stein Brito
NightCity Labs, Champalimaud Centre for the Unknown
Lisbon, Portugal
carlos. steinonightcityylabe
Daniel C. McNamee
Champalimaud Centre for the Unknown
Lisbon, Portugal
daniel. manamee@researcch fcw.mpalcchhhaaccccccccccc imaud. org
Abstract
Learned control policies often fail when deployed in real-world environments with changing dynamics. When system dynamics shift unexpectedly, performance degrades until models are retrained on new data. We introduce Reflexive World Models (RWM), a dual control framework that uses world model predictions as implicit reference trajectories for rapid adaptation. Our method separates the trol problem into long long-term reward maximization through reinforcement learning and robust motor execution through reward-free rapid control in latent space. This dual architecture achieves significantly faster adaptation with low online computa- tional cost compared model-basee RL baselines, while maintaining near-optimal performance. The approach combines the benefits of flexible policy learning through reinforcement learning with rapid error correction capabilities, providing a theoretically grounded method for maintaining performance in high-dimensional continuous control tasks under varying dynamics.
Introduction
Model-based reinforcement learning has significantly learned world models with policy optimization [Hafner et al., 2021 Hansen et al., 2022). These methods use neural networks predict future states, enabling both efficient planning through trajectory sampling and stable policy improvement through value estimation. Yet the challenge of maintaining performance under unexpected dynamic whifts- from environmental variation, physical wear, or other sources-remain critical. When dynamics change, planning and value computation degrade, often necessitating costly retraining specific online adaptation mechanisms Peng et al. 2018 Kumar et al. 2021].
I advanced continuous control by integrating
Control theory provides robust and formally grounded methods for handling changing dynamics through adaptive control, offering stability guarantees through Lyapunov analysis (Slotine and 1991. These methods maintain performance by continuously adjusting control parameters based tracking errors between desired and actual trajectories. However, classical control approaches rely on explicit reference trajectories and engineered cost functions, limiting their application to problems with well-defined objectives and structured dynamics models [Narendra and Annaswamy 2012). This contrasts with reinforcement learning's ability to learn flexible policies from abstract rewards and high-dimensional observations [Sutton and Barto 2018 Recht 2019
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We present Reflexive World Models (RWM), a framework that repurposes world model predictions as dynamic control targets for rapid adaptation while preserving learned policy behavior. reinforee- ment learning module determines optimal trajectories in latent space, which the world model predicts forward in time to serve as references for a control module that maintains performance through trajectory tracking. This architecture is formalized through analysis of value functions, showing how they decompose into slow learning and trajectory stabilization components. Our approach provides a novel mechanism for rapid adaptation by transforming world model predictions into reference trajec- tories, enabling learned policies to maintain performance under changing dynamics without requiring reward signals, specific robustness procedures, architectural constraints. In continuous control tasks including locomotion under varying dynamics, this achieves significantly faster adaptation than standard methods while maintaining performance. This computational efficiency arises from RWM's online mechanism, which uses lightweight controller updates driven by latent predictions from the world model. Such direct adjustments bypass the substantial per-step operational costs associated with planning rollouts or the extensive model retraining common in other adaptive strategies.
RWM offers a bridge between classical adaptive control (such as Model Reference Adaptive Control, or MRAC) and modern reinforcement learning. While classical methods provide stability guaran- tees, their common reliance on pre-defined reference models and state representations can limit their application, especially in complex systems where these are difficult to hand-engineer. RWM addresses this by leveraging learned world model to implicitly derive reference dynamics from high- high-dimensional observations. Furthermore, its operation in a learned latent space allows adaptation based on task-relevant features discovered by the RL agent, rather than fixed, pre-specified ones. By thus deriving references directly from world model predictions, our approach aims to combine the robustness of adaptive control with the flexibility of learned policies, with the value function decom- position establishing performance bounds under varying dynamics. This two-level policy learning discover optimal behaviors and rapid error correction by the adaptive controller to maintain execution fidelity-natural aligns with hierarchical control architectures. Our experiments demonstrate this separation of policy learning and rapid error correction leads more efficient adaptation to dynamic changes compared to approaches where these distinct functionalities are not explicitly modularized and managed by dedicated components.
saproach-slowwr
2 Background
Modern model- baseed reinforcement learning integrates several components through learned world models. Given observations of system state, these models learn compressed latent representations where planning and control occur [Hafner et al. [02T]. TD-MPC2 exemplifies this approach through a normalized latent space that enables stable trajectory sampling and value estimation [Hansen et al., 2023). Previous work has explored different approaches to adaptation Deep Model Reference
Adaptive Control [Joshi et al., 2019) combined neural networks with MRAC but required complex dual architectures, while Rapid Motor Adaptation (Kumar et al., [02T] and Residual Policy Learning [Silver et al. 2019] demonstrated online adaptation but required either specific architectural choices or limited adaptation particular types of system changes.
Adaptive control provides formal stability guarantees through Lyapunov analysis (Slotine and Li 1991]. achieving millisecond-scaa adaptation by adjusting parameters based on trajectory tracking errors. However, these methods require explicit reference trajectories, structured dynamics models, and engineered cost functions [Narendaa and Annaswamy 2012). In contrast, reinforcement learning learns flexible policies directly from rewards and high- digendimmnnional observations (Sutton and Barto 2018), but sacrifices adaptation speed and stability guarantees. Recent work meta-learning Finn ct al. 2017] and domain randomization [Tbiin et al., 2017| [Tn et al. 2018 Peng et al. 2018 has improved robustness but still requires extensive offline training.
Previous attempts to bridge these approaches have either restricted policies to specific forms amenable to control theory or limited adaptation to particular types of system changes [Recht] 2019]. A general framework for combining the flexibility of learned models with the rapid adaptation of control theory has remained elusive. Our work addresses this gap by leveraging predictive outputs to guide low-level controller, enabling classical control techniques while maintaining the benefits of learned policies.
2

---

3 Model design: implicit latent trajectory for adaptive control
3.1 Problem Formulation
We consider a continuous control problem formulated as a Markov Decision Process (MDP), defined by the tuple (S, A, P. r),), where S is the state space, A is the action space, P( a) is the transition probability, R(s,a) a) is the reward function, and 7 is the discount factor. Our goal is to learn policy {a]s) that maximizes the expected cumulative discounted reward. We operate in setting where the true system dynamics, represented by the transition function P, can change unexpectedly from the training conditions.
Our approach leverages a learned world model operating in latent space Z. The model consists of: an encoder @ S - Z where 2 o(s:), latent dynamics model F Z A + Z where 241 = F2,, ax), and a base policy To Z - A where an = xo xx))). The objective of our Reflexive World Model (RWM) is to learn second, adaptive control policy He 2 A, which outputs a corrective action ag- The total action is 0 ag + Og- The objective for Te is to minimize the discrepancy between the world model's one-step prediction (under the base policy) and the observed outcome, thereby making the system behave as predicted by the reference model under the base policy.
Consider a continuous control MDP with learned policy mo operating through a world model in latent space o(s). We assume that the latent space captures task-relevan dynamics from observations s. with (z) = (o(s)). A world model F predicts future latent states conditioned on the current state and policy actions.
3.2 Decomposition of the value objective
Locomotion involves fundamentally distinct learning processes operating at different timescales. Policy learning gradually discovers behaviors that maximize long- pergterm reward. This process requires extensive exploration but develops robust policies for diverse tasks. contrast, rapid adaptation maintains performance under changing dynamics without modifying the underlying policy. This process responds quickly to errors but operates within the framework of existing behaviors.
This functional separation suggests decomposing motor learning into complementary objectives. The policy learning system should discover behaviors that maximize expected value across tasks, while the adaptation system should maintain stable execution under perturbations. We formalize this approach by linking value-based learning with rapid error correction, decomposing the Taylor expansion of the value function around optimal trajectories in a task-relevant latent space 2:
(z++1 + Az) = ((++1) 1 = +
where H = (244)) is negative definite the optimum. This decomposition suggests separat- ing the problem into maximizing mean value through policy optimization and minimizing deviations through rapid adaptation.
3.3 Forward model predictions as references
Our Reflexive World Models (RWM) framework implements the functional separation through a dual architecture. The reinforcement learning module learns base policy, do To(2), that optimizes the mean value, as in classic RL models. We operate within a continuous, normalized latent space = o(s), where Ø is encoder (further details in Section 5 and Appendix (A). We opt for a continuous latent space, akin to TD-MPC2 [Hansen et al. 20233 as directly supports the differentiability required for our -rased adaptive controller (Section 3.3). This choice offers simpler pathway for encoding states for continuous control tasks compared to discrete representations (c.g., [Hfnnr al. 2021) which would necessitate specialized techniques for gradient propagation. Normalization ensures that all dimensions of the latent space have comparable scales, which is crucial for balanced contribution to the error computation discussed below. We maintain a forward model F predicting future latent states:
2+11 F(ze: mo(:))
The world model F and the policy To (and by extension, the encoder ()) are assumed to be differen- tiable with respect their inputs. This is common assumption in many model-based RL approaches
3

---

that use gradient- (Section 3.3).
gradient-based based learning and is prerequisite for our adaptive control gradient computation
Our framework builds on model-based reinforcement learning but changes how world model predic- tions drive behavior. A conceptual novelty is that we interpret the forward model predictions as target states. Both the forward model and controller share the same error function measuring discrepancy between predicted and actual states:
L | - Z++1 |?
Following approaches like TD-MPC2 (Hansen al. 2023] use the Mean Squared Error (MSE)
as the discrepancy measure. Given the normalized latent space, MSE provides a well-distribule measure of prediction emror across all latent dimensions. This choice contrasts with scale-invariant metrics such as Kl.-divergence, utilized in some world models (e.g.. [Hafner et al. 202T)) D) we found empirically that KL-divergence yields worse adaptation performance in our experiments. scale-invariant loss can disproportionately weight or neglect certain latent dimensions whose precise tracking might be critical for fine-grate fine-grained motor adaptation. However, the forward model and control module minimize this error in opposing ways. The forward model adapts its predictions to match observations, following the standard gradient to improve predictions. contrast, the control module adapts actions to make the system behave as predicted.
3.4 Adaptive control gradients
This approach inverts the standard relationship between models and control. Classical adaptive control assumes reference trajectories and adapts a controller to track them. Model-based RL learns models that predict actual outcomes and uses them for planning. Our framework generates reference trajectories directly from world model predictions while adapting control to maintain their validity under changing dynamics.
The control policy updates follow modified gradient computation that reflects this inversion. Rather than updating predictions to match observations, we update actions to make observations match predictions:
(-2ce
OC
duc 00.
0. f 0c- 7e
Bag
The update function leverages gradients through the world model to determine how actions should change to reduce prediction error, inverting the standard approach to model learning. This differs fundamentally from standard practice where gradients flow from predictions to parameters. The control module instead treats predictions as fixed targets and adapts actions to achieve them.
The total action combines the base policy with these corrections:
(5) Critically, this update requires only prediction error-no reward signal is needed for adaptation. Operating in the world model's latent space provides benefits. First, it ensures the control module focuses adaptation on task-relevant features captured by the learned representation. Second, provides an interface between the RL policy operating on compressed latent states and the control module maintaining prediction consistency.
a, = (2) + =e(2))
Theoretical Guarantees
A notable feature of the RWM framework is that this decoupling of long-term policy optimization from rapid, reference-tracking adaptive control makes the system particularly amenable to rigorous control-theoretic analysis. For many contemporary model-based RL agents (c.g.. [affnrr et al. 2021 Hansen et al., 2022), deriving formal guarantees on overall task reward or value under dynamic perturbations is exceptionally challenging. This difficulty often arises not just from the inherent complexity of these comprehensive, end-to-end learned systems. also stems from the fact that the general, nigh-dimensional value functions they optimize not easily subjected to direct stability or error- brund analysis from classical control theory.
In contrast, RWM's specific formulation of the adaptive control
the latent prediction error | - 2++1|l2 I|2 (Equation 3) to ensure the system tracks the world model's predic- tions presentss more constrained and tractable problem. This focus on well-defined error signal

---

Algorithm Reflexive World Models (RWM)
Require: Trained policy no. encoder o, world model F, learning rate Tec
Ensure: Adapted control policy Te
Initialize
for each episode do
Ft < c(st)
5 <=) Up +- Te(Za)
Execute at Up des observe 81+1
+++1 < (({++1)
B++1 F(t;,0))
e - 2++1 24+1
0g - 0e To
Ølle.1
Baz Bao 89.
end for
for the adaptive controller allows us to apply established can establish theoretical limits on this control emrrr and, crucially, link these to bounds on the overall value function (Theorem 4.3). This provides formal insights into expected performance degradation under changing dynamics, bridging gap between flexible learning and provable robustness.
tools. Consequently,
The following theorems formalize these guarantees:
Assumption 4.1 (System Propertics). The system satisfies:
1. |0F/Oal| < L
(Lipschitz control)
2. Jmin (OF/Oa)) 2 a LP
(control authority)
3. [[ f(2,a))
(model accuracy)
4. [p((t)| P
(bounded perturbation)
where P bounds external perturbations.
Theorem 4.2 (Control Error). Under Assumption 4.1 the control law a. -n0F//aa e(()) achieves:
p2
|e(())| S (e()) + Ve
(6)
a²
+ nL?) where y = -1-na2
< 1 for < 1/L².
Theorem 4.3 (Value Bounds). the error bound is sufficiently small, the value function satisfies:
V(z") V(z) HM
p2
2
a²
where HM bounds the eigenvalues of V near optimal trajectories.
These results show how world model accuracy (e), control authority (a), and perturbation magnitude (P) determine performance bounds, with quadratic scaling reflecting the natural structure of value functions around optimal trajectories. The proof' use standard Lyapunov techniques (see Appendix F
5 Simulations
Our experiments address three questions: (1) Can world model predictions serve as effective ref- erence trajectories for rapid adaptation? (2) Does minimizing control error-without access to acched- robust adaptation across diverse tasks? (3) How does performance scale from simple to high-dimensional systems?
We evaluate across multiple continuous control environments from the DeepMind Control Suite TTssa et al., 2018): a 2D point-mase mass system (direct state observations, no encoder), standard locomo- tion (Walker- Run, Cheetah-Run, Hopper-Hop), manipulation (Ball-in-Cup), and high-dimensional coordination Hamlanodd-Walk, 17 actuators). Adaptation is tested under actuator perturbations simulating online miscalibration, comparing RWM against frozen baseline policies and reward-base fine-tuning.
5

---

A
B
Beff
(ap + ac)
. (1))(11)))) p)
Reversal effect
RL
Control
Q)(2,a)
ac
-Ōa
4
ag
Sitk
oa
Deviation
RC
a
RELA
St
ap P
.0o2
0.
11
ac
St
Figure I: (A) Network architecture of Retlexive World Models (RWM), showing the reinforcement learning policy (blue) and adaptive control modules (green), with interface variables in orange. Each transformation is implemented adaptive control mechanism for a 2D pointmass task (without encoder, 2 = s). When actuators are perturbed, the trajectory deviates from the predicted future states $1+* under the base policy actions an- This error triggers an update to generate corrective actions ag. (C) Under alternating directional perturbations (red), RWM corrects deviations from the optimal trajectory, exhibiting characteristic after-effects when perturbations are removed.
two-hidden-layer MLP. (B) Illustrative simulation of the
5.1 Addressing Policy Saturation and Inaction via Thresholded Action Cost
Our adaptive controller Te requires the base policy mo to provide differentiable, non-saturated actions. However, standard RL policies in continuous control can exhibit action saturation (leading to vanishing gradients for Te, see Fig. C) or policy inaction (the "dead problem" from naive action costs, Fig. A,). Both issues hinder effective adaptation.
To ensure suitable base policy, we incorporate specific thresholded quadratic action cost, ΛE, (max(0, |a, - c))?, during To pre-training. This simple modification to standard action penalization encourages =o operate within smoother, non-saturated region responsive to T., while also mitigating policy inaction. The formulation, rationale, and illustrative effects (Fig. 44D) are detailed in Appendix C
5.2 Dual-model Design
Our Reflexive World Models (RWM) approach builds on model-based reinforcement learning methods such Dreamer and TD-MPC2 [affer et al. 2021 Hansen et al. 2023), primarily to leverage their task-related encoders. In particular, TD-MPC2 uses latent, reward and value predictions to learn the encoder and forward models. We reuse these pre-traine pre-triined representations focus the novel control mechanism. simple environments where an encoder is unnecessary. such as the point-mass mass task, direct state observations (2, sz). In these cases, SAC any alternative policy, including non-RL-bbaed controllers, could be used as the baseline policy, and the forward model is learned through latent prediction.
5.3 Perturbation Experiments
We evaluate adaptation by introducing perturbations p(() the action space, with an effective actuator O.ff = (an + a.)) (1 + p). and measuring the controller's ability to compensate for them (Fig. B). Step perturbations involve sudden changes in the action signal at specific intervals, while slow perturbations introduce gradual, non-stationary shifts that mimic actuator miscalibration. These perturbations allow us to analyze how the controller reacts to both abrupt and progressive deviations.
Our primary baseline uses the TD-MPC2 reactive policy trained without the online planner, with continued online fine-tuning. This configuration was chosen avoid conflating the use of the world model for both planning and our control mechanism, ensuring a cleaner experimental comparison.
The original TD-MPC2 work [Hansen et al. 2023) demonstrated that the online planner provides substantial benefits primarily on complex tasks, while the reactive policy remains highly performant on standard locomotion-s finding we confirm empirically in Appendix where both configurations achieve nearly identical performance Walker, Cheetah, and Hopper tasks. The exception is the

---

Humanoid environment, where the reactive policy alone is insufficient for stable locomotion, so we use TD-MPC2 with its planner for this task only.
The experiment consists of two phases. In the first phase, a policy is trained or provided without perturbations, and forward model is learned using its trajectories. This phase establishes the baseline model of the system's 's behavior under normal conditions. In the second phase, perturbations are introduced while simultaneously activating the controller. The controller uses the learned forward model to adjust its outputs in response to the deviations introduced by the perturbations.
To assess adaptation performance, we measure the drop in task performance caused by the perturba- tions and track the forward model error. The latter serves as an implicit measure of latent trajectory deviation, reflecting how well the system follows the predictions of the forward model.
The prerequisite pre-training baseline model common all and takes approximately 16- 16-17 hours. Execution times for the experimental setups are detailed in Table 1 pre-trained components and lightweight controller updates, is notably more computationally efficient than approaches requiring full model retraining.
RWM's online phase, which uses
Table I: Typical execution times for the online adaptation phase (1 million environment steps under perturbation) single NVIDIA Tesla T4 GPU.
Experimental Setup
Approximate Execution Time
TD-MPC2 (Full Training)
16.5 hours
RWM (Online Adaptation)
1.8 hours
No Adaptation (Inference)
1.4 hours
The computational efficiency of RWM's online adaptation (Table I stems from using lightweight controller updates based on forward-model predictions, avoiding the computational overhead of planning horizons or full model retraining typical in model-based approaches.
5.4 Correction for Trajectory Deviations
The point mass system demonstrates the core interaction between policy and adaptation modules. Under angular perturbation, the base policy's 's trajectories systematically deviate from target while the world model maintains predictions intended paths (Fig. B). The control module uses these predictions as references generate corrective actions, recovering performance without modifying the underlying policy.
The system exhibits characteristic aftereffects when perturbations are removed (Fig. CC). Initial overcorrection in the opposite direction indicates adaptation through internal model formation rather than reactive control. This validates the method's ability to learn and compensate for systematic changes in dynamics while preserving the original policy.
5.5 Robust Motor Control
The Walker2D environment demonstrates how adaptation can operate effectively in learned latent space. Under step perturbations to actuator gains, the control module rapidly reduces the error between predicted and actual latent states (Fig. 2). The right column of Figure [ presents these metrics aggregated over perturbation cycles, with values normalized within each cycle relative to the Noo Adaptation' agent's 's performance range to highlight relative improvements. As the latent prediction error decreases (shown in the control error plots), task performance improves correspondingly (shown in the reward plots), validating that world model predictions in latent space provide effective references for adaptation.
Table 2 presents results across multiple environments under step perturbations (full results in Ap- pendix B}. Critically, RWM achieves robust adaptation without access reward signals- miinmmzzin latent control error alone is sufficient to recover substantial task performance. On Walker, RWM achieves 94.3% reward recovery, substantially outperforming fine-tuning's 66.3%, demonstrating that the reward-free objective can match or exceed reward-driven adaptation. The Humanoid environment particularly revealing for complex, high-dimensional tasks: RWM's controller successfully operates on the 17-actuator system, while fine-tuning reward causes the policy to deviate significantly from
7

---

370
Nommlizzdd Roward
350
350
ca-
Reward
340
G4
230
320
Mipptation
ca
No Adeepaaion HWM MPCC
RWH
MD-
0.0010
Nomalized Medien Centrol Error
3.24
2.0
ca-
0.0006
0.0004
0.0000
Episodes 120
Episodes
Figure 2: RWM adaptation performance under step motor perturbations. The plots show Reward and Control Error over 800 episodes (left column); shaded areas indicate perturbation periods. The right column displays Normalized Median Reward and Control Error, aggregated across perturbation cycles. Shaded areas the right plots represent the 95% confidence interval of the median (bootstrapped), RWM (green line) consistently maintains higher reward and lower control error compared to No Adaptation (gray line) and the TD-MPC2 (no planner) baseline (blue line), demonstrating effective and rapid recovery from perturbations.
its original motor program (58.6% increase in control error). In contrast, RWM preserves the core motor pattern (24. % decrease control error). This illustrates central narrow task-specific metrics (e.g., forward velocity). while control error captures fidelity to the entire learned motor program. complex tasks, these objectives can diverge: maximizing reward may degrade the underlying coordination pattern, suggesting that preserving trajectory integrity through control error minimization can provide a more robust foundation for adaptation.
rewward reflects
Table 2: Adaptation performance on representative environments under step perturbations. Reward
Drop and Control Error measure degradation from baseline; Improvement (%) shows recovery relative to frozen policy. See Appendix B for complete results across all environments.
Environment
Method
Reward Drop
Control Error
Improvement (%) Reward Control
Frozen Policy
108.40
28.09
Walker-Run
Fine-tuning
36.50
8.87
66.3
68.4
RWM (ours)
6.20
7.43
94.3
73.6
Frozen Policy (w/ planner)
28.10
0.58
Humanoid-Wail
Fine-tuning (w/ planner)
12.90
0.92
54.1
-58.6
RWM (ours, w/ planner)
13.60
0.44
51.6
24.1
A crucial test for deployment in real-world systems is the ability to handle nonstationary dynam- ics-scenarios where system properties gradually change over time due wear, temperature fluctua- tions, miscalibration. Such scenarios are particularly challenging because they cannot be addressed through fixed robustness strategies, demanding continuous adaptation. To evaluate this capability, we introduced continuously varying perturbations by applying filtered noise to actuator gains, simulating gradual deterioration and drift common in physical hardware (Fig. A The results suggest strong potential for real-world applications: RWM maintains a performance of 360.56, substantially outperforming both TD-MPC2 (no planner) (311.67) and the fixed baseline policy which degrades to 233.42. This performance advantage is accompanied by systematically lower control error for

---

RWM compared to both alternatives, indicating more efficient adaptation to continuously changing dynamics. The world model predictions provide a stable reference for ongoing adaptation even as dynamics evolve unpredictably, enabling rapid corrections without reward signals or policy retraining that would be impractical in deployed systems.
A
P
B
1.3.99
1.0-0M
0.0-900
Epioodes
Figure Nonstationary perturbations and high-dimensional coordination. (A) Walker2D under continuous filtered noise perturbations to actuator gains, following sinusoidal pattern p (top). Time series (left) and averaged performance (right) show RWM (green) achieves the highest reward (360.56). followed by TD-MPC2 (no planner) (blue) (311.67), with No Adaptation (gray) performing worst (233.42). Control error measurements (bottom) demonstrate that RWM maintains systematically lower eπror throughout adaptation compared to both alternatives. (B) Analysis of the 17-actuator Humanoid environment showing coordinated movement patterns maintained by RWM even under perturbations.
5.6 High--Dimensionl Coordination and Control-Theorettic Perspective
Beyond nonstationary dynamics, examined high-dimensional coordination in the 17-actuator Humanoid environment B)) where successful locomotion requires synchronized movements across multiple joints. When perturbed, the unadapted system exhibits severely degraded coordination between joints (compare the normal orange gait with the gray affected movement). RWM's primary advantage is revealed in the trajectory density visualization: by tracking latent predictions that capture coordinated multi-joint relationships, it maintains coherent movement patterns even under challenging conditions.
This separation of concerns between RL and control modules provides a framework to analyze differ- ent sources of variability: intentional variability from exploration and decision decision-making (RL module) versus unwanted variability from imperfect execution and external perturbations (control module). This distinction has potential applications in understanding biological motor control, where similar separations between voluntary movements and reflexive corrections exist. The system demonstrates sophisticated adaptation where perturbations individual actuators trigger compensatory adjustments across multiple joints simultaneously, preserving overall balance and performance.
5.7 Comparison with
Baselines
To isolate the benefits of RWM's 's online adaptive mechanism, we compared its performance against bascline agents (TD-MPC2 (no planner) and No Adaptation) that were pre-trained with exposure to the same randomized actuator perturbations subsequently used during evaluation. This form of domain randomization during pre-training aims to enhance the inherent robustness of the baseline policies. The comparison then assesses whether RWM's dedicated online adaptation still provides substantial advantages beyond this enhanced baseline robustness. The results of this comparison are presented in Figure 5 (see Appendix D).
This comparison investigates whether pre-training the baseline TD-MPC2 (no planner) policy with exposure to randomized actuator perturbations-a form of domain randomization-con could achieve robustness comparable to RWM's online adaptation. While such pre-training does enhance the base- line's 's performance when dynamics nominal (i.e., perturbation OFF in Figure (see Appendix D

---

where this specialized TD-MPC2 (no planner) agent achieves higher reward than RWM), the critical test is performance under active, unmodelled dynamic shifts. During these perturbation phases. RWM's advantage is evident: Table shows that RWM maintains significantly higher reward and lower control error compared to the TD-MPC2 (no planner) agent pre-trained with perturbations. This outcome underscores that while domain randomization can improve baseline robustness to some extent (as the pre-trained No Adaptation policy still performed worst, with a median reward of 0.1 and control error of 0.9676 during perturbation), it does not eliminate the performance degradation caused by the perturbations encountered during evaluation. Consequently, this form of domain randomization does not eliminate the need for an explicit online adaptation mechanism like RWM, which provides more effective real-time compensation for unmodelled dynamics than pre-training alone.
Table 3:
Comparison with Domain- Randomized Baselines: Performance Metries during Perturbation ON (Baselines Pre-trained with Perturbations)
Metric
RWM
TD-MPC2 (no planner)
Median Normalized Reward
0.6003
0.3600
Median Normalized Control Error
0.7307
1.4162
6 Discussion
This work introduces Reflexive World Models (RWM), demonstrating how predictive models can be cast as sources of desired state trajectories for rapid adaptation. RWM distinctively inverts the standard model-based RI. paradigm: rather than using world models for planning, we enable the control policy Te to directly minimize prediction errors in latent space. This mechanism reframes the learning objective-instead of the model learning to track reality, the agent acts to make reality track the model. This contrasts with typical model-based RL methods where world models primarily serve for planning or data augmentation. Unlike Model Predictive Control, which plans action sequences over a horizon, RWM uses single-step predictions as instantaneous references, enabling rapid responses with low computational overhead ideal for real-world deployment. A notable advantage of this approach is that adaptation requires no reward signal, making it applicable when rewards are sparse, delayed, unavailable in deployed systems.
The inherent dual timesalle operation in RWM- policy learning with wo discovering optimal behaviors and faster eиror correction via Te maintaining execution fidelity-is a central aspect of its effectiveness. This separation of concerns, where Te rapidly compensates for dynamic shifts based on To's is intended trajectory (as predicted by the world model), allows the system to adapt without costly retraining of the entire policy or world model. These theoretical guarantees (Section 4) rely on assumptions such as sufficient control authority and model accuracy, which may not always be fully met in highly complex or underactuated scenarios. Nevertheless, our empirical results demonstrate RWM's 's practical effectiveness across challenging benchmarks.
6.1 Limitations and Future Work
Future work includes several directions. First, learning latent representations specifically optimized for RWM's adaptive control could enable application to arbitrary pre-trained policies. Second, while our phased training approach simplifies analysis, end-to-end joint learning of the base policy, world model, and adaptive controller presents interesting challenges around preventing controller exploitation of model inaccuracies. Third, extending RWM beyond actuator perturbations to handle morphological changes, environmental shifts, or sensor noise would broaden applicability. This may require incorporating uncertainty estimation into the world model or developing methods to disambiguate different error sources.
The principle of reward- free adaptation through world model predictions offers a promising foundation for robust, learned behaviors in real-world deployments where reward signals may be unavailable. Reflexive World Models (RWM) build on this by integrating a rigorous theoretical framework with comprehensive empirical support, facilitating the development of adaptive agents that can sustain high performance under rorldworld dynamic changes.
10

---

Acknowledgments and Disclosure of Funding
This work supported by the Champalimaud Foundation and the Google Cloud Research Credits program with the award GCP398030901.
References
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model- meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1136-1155, PMLR, 2017.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Mastering atari with discrete world models. International Conference on Learning Representations, 2021.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In Proceedings of the 39th International Conference on Machine Learning, pages 8163- 8174. PMLR, 2022.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint UrXiv:2310.16822 2023.
Girish Joshi, Jasvir Virdi, and Girish Chowdhary. Deep model reference adaptive control. In AIAA SciTech Forum. AIAA, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference Learning Representations (ICLR), 2015.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. In Robotics: Science and Systems, 2021.
Kumpati S Narendra and Anuradha M Annaswamy. Stable Adaptive Systems. Courier Corporation, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high sesh-peffommncce deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3803-3810, 2018
Benjamin Recht. A tour of reinforcement learning: The view from continuous control. Annual Review of Control, Robotics, and Autonomous Systems, 2:253-279, 2019.
Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling. Residual policy learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 331-343 2019.
Jean-Jacques E Slotine and Weiping Li. Applied Nonlinear Control. Prentice Hall, 1991.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press,
Cambridge, MA, 2 edition, 2018.
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. In Robotics: Science and Systems (RSS), 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden. Abbas Abdolmaleki, Josh Merel, Arnoud Lefrancq, et al. Deepmind control suite. Journal of Machine Learning Research, 19(38):1-22 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In IEFF/RSJ International Conference Intelligent Robots and Systems (IROS), pages 23-30. IEEE, 2017.
11

---

A Implementation Details
Our main simulations are conducted using the DeepMind Control Suite [Tssa et al. 2018 with tasks such as humanoid d- walk and walker-walk (with an action repeat of 2), and are implemented in PyTorch [Paszke et al. 2019]. Experiments were run computer with single NVIDIA Tesla T4 GPU (16 GB GDDR6). For experimental runs of million environment steps, typical execution times on this GPU were approximately 1.5-2 hours for the 'No Adaptation' and RWM setups. These setups primarily involve inference using pre-trained components, with RWM additionally performing lightweight online update for its controller. In contrast, full TD-MPC: training run of million steps, which includes its comprchensive learning cycle of model updates, planning, and policy/value optimization, typically took around 16-17 hours.
Baseline TD-MPC2 Agent. We pre-train a TD-MPC2 agent [Hansen et al., 2023 for 1 million environment steps to provide the state encoder é, the base policy To, and the forward dynamics model hidden layers with 256 units each, while other MLPs within the TD-MPC2 agent (such as those for the policy and dynamics model) have 2 hidden layers with 512 units each. These networks use Mish activations. The encoder output is normalized using SimNorm with 8 dimensions [Hansen al. 2023). The base policy To is pre-trained with thresholded quadratic action cost (Section 5.1] with full details and illustration in Appendix C The environments use hard action bounds of 2-2. and the threshold for this cost is set to 0.5, with the penalty coefficient À = 0.2.
F. The encoder Φ consists
Forward Model Training. The forward model F is trained as a general transition model during the pre-traing mre-training phase, exposed to wide distribution of states and actions including those from early, suboptimal policies-not just trajectories from the final converged policy. This broad training enables the model to make reasonable predictions even for states off the optimal trajectory, which is crucial for providing corrective signals when perturbations occur.
Episode Length and Reward Scale. Our experiments use maximum episode length of 400 steps, compared to 1000 steps in the original TD-MPC2 paper [Hansen et al. 2023). This results in maximum achievable rewards of approximately 400 for locomotion tasks, rather than 800+. Our reported rewards of ~880 are therefore near-optimal for this episode length. The thresholded action cost is applied only during policy training to ensure sufficient action headroom for the adaptive controller.
For comprehensive details on the TD-MPC2 architecture and its training, we refer to the original publication [Hansen et al. 2023
RWM Controller. The Reflexive World Model controller, Tes is MLP with 2 hidden layers and 512 units per layer, using ReLU activations. During RWM adaptation, the parameters of the pre-trained TD-MPC2 components (encoder o, base policy no, and forward model F)) are frozen. The RWM controller n. is then trained online over a multi-step horizon of 3 to minimize the squared error between the forward model's latent state predictions (based on To's 's actions) and the observed next latent states, as per Algorithm[] T This update involves backpropagating the gradient of this prediction error with respect the base policy action Og and then transferring this gradient (with inverted sign, as detailed in Algorithm I and Equation 4 to update the RWM controller т.. The learning rate for म. is 3 10-4
To evaluate adaptation, we introduce multiplicative step perturbations to actuator gains, where the effective action for each actuator j becomes deffJ (oo. +aag) ( (1 (1++p)) p,). For each perturbation instance, the value Pj for each actuator j is sampled uniformly from the range 0-0.5, 0.5]. These perturbations alter actuator outputs for 10,000 to 15,000 environment steps, followed by an equal duration without perturbation. This cycle repeats throughout an experimental run, which extend up million steps for comprehensive experiments including pre-training and adaptation phases.
Perturbation Protocol.
All neural networks, including are optimized using Adam (Kinmma and Ba 2015 A simplified reference implementation of the RWM controller is provided in the supplementary material. The full codebase will be made available upon publication.
12

---

B Results Across All Environments
Table 4 presents adaptation performance across all evaluated environments under step perturbations.
Table 4: Adaptation performance across all environments under step perturbations. Reward Drop and Control Error measure degradation from baseline; Improvement (%) shows recovery relative to frozen policy. RWM achieves adaptation through reward-free free control error minimization.
Environment
Method
Reward Drop
Control Error
Improvement (%)
Reward Control
Frozen Policy
1.00
9.05
Ball-in-Cup
Fine-tuning
0.00
2.61
100.0
71.2
RWM (ours)
1.00
2.09
0.0
77.0
Frozen Policy
6.30
3,21
Cheetah-
Fine-tuning
5,60
2.44
11.1
24.0
RWM (ours)
1.40
1.70
77.8
47.0
Frozen Policy
15.40
5.67
Hlopee-Hoop
Fine-tuning
6.20
2.92
59.7
48.5
RWM (ours)
4.50
3,20
70.8
43.6
Frozen Policy
108.40
28.09
Walker-Run
Fine-tuning
36.50
8.87
66.3
68.4
RWM (ours)
6.20
7.43
94.3
73.6
Frozen Policy (w/ planner)
28.10
0,58
Humanoid-We Walk
Fine-tuning (w/ planner)
12.90
0.92
54.1
-58.6
RWM (ours, w/ planner)
13.60
0.44
51.6
24.1
C Details the Thresholded Action Cost
A base policy To that provides differentiable, non-saturated actions is crucial for the effectiveness of our adaptive controller, Re- Standard continuous control policies, however, can suffer from two common issues: (i) action saturation at the boundaries of the allowed range, which nullifies gradients needed by ne (Fig. C), and (ii) policy inaction or the "dead problem' (Fig. A.B), where naive quadratic action costs can overly penalize any movement, leading to minimal activity.
To address these, employ thresholded quadratic action cost, E, (max(0, |a,] ())l- during the pre-training of To. Here, a; is an action component, e is a threshold (e.g., c = 0.5), and a coefficient (e.g., À 0.2). This cost is simple modification of standard quadratic penalty but only penalizes action magnitudes |a,| exceeding
This encourages To to operate primarily within a smoother, non-saturated region (Fig. 44D),, preserving differentiability for ne. Simultaneously, by penalizing actions within [-c, c). it mitigates the dead problem, allowing the agent to achieve adequate task performance. While other techniques might yield suitably bounded actions, this approach offers straightforward way to obtain a responsive baseline for adaptation.
D Figure for Comparison with
Baselines
Figure illlstrates performance comparison for the comparison with domain-randomized base- lines discussed in Section 5.7 where baseline policies were pre-trained with exposure to actuator perturbations.
13

---

Figure 4: Addressing challenges in baseline policy actions for effective adaptation. (A) A humanoid agent exhibiting "dead" behavior due a simple quadratic action in its RL objective, leading to inaction. (B) The norm of actions for the simulation in (A), demonstrating decay towards zero over training episodes as the agent minimizes the naive action cost. (C) Action component values over time for standard TD-MPC2 policy (without the thresholded cost) in the Humanoid task, showing frequent saturation at the boundaries [-1, 1]. which impedes gradient flow for the adaptive controller. (D) Smoother and bounded action values from a TD-MPC2 policy trained with the proposed thresholded quadratic action cost, maintaining differentiability and responsiveness.
370
2.0
300
Reiaard
340
Normalized Median Control Mommal Median
3.0
.550
0.5
320
Adeptation
Adaptaaioon
330
RWM
0.5
300
0.0008
3.5
0.000e
Control
3.0
0.0002
8o
0.0000
Episodes
Cycle
Figure 5: Comparison with Randomz-addd Baselines: Impact of pre-training with perturbations. Comparison of No Adaptation, RWM, and TD-MPC2 (no planner) when baseline policies for No Adaptation and TD-MPC2 pre-trained with exposure actuator perturbations. (Left column) Reward and Control Error over episodes. (Right column) Normalized median reward and control error within perturbation cycles. While pre-training with perturbations improves the baseline, RWM (green line) still demonstrates superior adaptation capabilities in terms of reward and control error compared the pre- pre-trained TD-MPC2 (no planner) (blue line) and the pre-trained No Adaptation policy (gray line).
E Baseline Comparison: TD-MPC2 With and Without Planner
We use the TD-MPC2 reactive policy (trained without the online planner) as our primary baseline. This appendix provides empirical validation that this configuration is appropriate for the locomotion tasks evaluated.
Table[]]show: that for standard locomotion (Walker, Cheetah, Hopper), the reactive policy achieves nearly identical performance to the full agent with planning, both in final reward and when handling perturbations. This is consistent with the original TD-MPC2 work [Hansen et al. 2023]. which showed the online planner provides substantial benefits primarily on complex tasks like Humanoid. Importantly, TD-MPC) without the planner still leverages the world model for latent imagination

---

during policy learning (similar to Dreamer [Hatner et al. 2021 making it capable model- bbaee baseline.
Table 5: TD-MPC2 performance with and without online planner on standard locomotion tasks.
Environment
Configuration
Final Reward (IM steps)
Reward Under Perturbation
Without Planner
380.2
342.1
Walker-Run
With Planner
381.4
345.3
Without Planner
364.5
358.2
Cheetah-Run
With Planner
362.8
360.1
Without Planner With Planner
373.1
367.4
Hopper-Hop
372.6
368.2
Table 6 compares against SAC, standard model-free baseline. TD-MPC2 (no planner) substantially outperforms SAC.
Table 6: Comparison of TD-MPC2 (no planner) against model-free SAC baseline.
Environment
SAC
TD-MPC2 (no planner)
Cheetah-Run 244.0
364.5
Walker-Run 249.7
380.2
Hopper-Hop 19.2
373.1
These results validate our choice: perturbation and substantially outperforms model-free methods. This configuration enabled a clean comparison by avoiding conflation of world model use for both planning and our control mechanism.
the reactive policy performs comparably to the full agent under
15

---

Theoretical Analysis
We provide proofs for the main theoretical results, showing how control error bounds lead value function guarantees.
The proofs rely Assumption 4.1 from Section 4
Theorem F.1 (Control Error Bounds). Under Assumption 4.1 the control law oe = e(t)
with < 1/L2 achieves:
[(((()) |>'1|e(0)]| + v
pa a²
(8)
where 7 = na² + n221 < 1.
Proof.
The error evolves as:
e(t + 1) ((,, F(2e: ao) + F(z,, - fze, at) +p(t)
(9)
control effect
model error
For the control effect:
- -F + = - e(t) (first order)
(10)
[(F//8|[0F/8a)]|
22a a² (by min singular value)
(11)
[| 0g + 0.) ao), $ -naa nT. (1²)) (Lipschitz)
(12)
The model error satisfies:
[F(2, f(2, c Lllaoll sct LP
(13)
a
Therefore:
|le(1 +
[|]|+ 1)11 S 7|1(((1 +
(14)
By condition (2) of Assumption 4.1 and 1/122
7 1-11a + nnL² < 1 (LP)2 + nL2 <1
(15)
The bound follows from solving this recurrence, using the fact that for positive a, b:
(a+ < 2(a² + b²)
(16)
Theorem F.2 (Performance Guarantees). If 2 +2
where bounds the region of
quadratic approximation for V, then:
HM
2 p2
V(=*) V(2) -
(17)
2
a
where H bounds the eigenvalues of -VV.
Proof. Around optimal trajectories, Taylor expansion gives:
V(=" A=) = V(=*) - - R(A=)
(18)
where [[(Az) < | for some C 0.
The prediction error directly bounds state deviation:
p2
A | =11 < |e(t)]l s e a²
(19)
When this is less than 8, the quadratic term dominates since:
R(Az)] < C||A2|| 0
(20)
A
The bound follows from Amaze (H) -H and the error bound.
16

---

These results establish quantitative bounds linking world model accuracy (é), control authority (a), and perturbation magnitude (P) performance. The quadratic scaling reflects the natural structure of value functions around optimal trajectories.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's contributions regarding a dual control framework for rapid motor adaptation using world model predictions as reference trajectories. The claims align with the presented methods and results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
The paper discusses limitations and future work in section 6 covering aspects such as encoder learning, end-to-end training, and the scope of perturbation handling.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
17

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
section 4joutlines the theoretical results and their assumptions. The Appendix
(Appendix F provides detailed proofs and restates the assumptions for these theoretical
guarantees.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Appendix A (Implementation Details, lines 601-6300 describes the environ- ments, model architectures, hyperparameters, and perturbation protocol, providing sufficient details to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
18

---

(d)
The authors should provide scripts to reproduce all experimental results for the new reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While the full code not released at this time, Python- based reference implementation of the core Reflexive World Model (RWM) controller mechanism, as described in Algorithm I and Subsection 3.4] is provided in the supplemental material. This allows for inspection of the key algorithmic contributions.
Guidelines:
The answer NA means that paper does not include experiments.
Please see the NeurlPS code and data submission guidelines (https://nips.co
Sublic/guidess https//oodsunnnnsssooooo....
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/pips.ccc/publc//guue http://udiieseeeeeeeee / http:///ddeeuboiisoooooooooooeee http://pwwwppceeiioooooooeaaaaaa
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Apppendix A (Implementation Details) specifies necessary training and test details, including environments, model architectures, optimizer settings, hyperparameterrs and the perturbation protocols used for evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
19

---

Justification: For the main results concerning adaptation to perturbations (Figures 2 and - the paper reports median rewards and control errors, with shaded areas indicating variability. estimated over perturbation cycles.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix A (Implementation Details) specifies the GPU type (including memory), the scale of experiments in terms of environment steps, and information computing time, providing necessary details on compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research was conducted in accordance with the NeurIPS Code of Ethics. The work focuses algorithmic advancements for adaptive control in simulated environ- ments and does not involve human subjects, sensitive data, or direct societal deployments that would raise immediate ethical concerns under the code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
20

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses positive societal impacts through its contributions to the robustness of autonomous systems (c.g., in inssccioon (sbseectin 5.5). More robust and adaptive control can lead to more reliable and safer autonomous systems in various appli- cations. Potential negative societal impacts, such the misuse of advanced autonomous systems, are general concerns for Al but are not uniquely amplified by this specific method- ology, which focuses foundational control mechanism.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymmett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The work presents a control algorithm and its evaluation in simulated environ- ments. No large-scale pretrained models datasets with immediate high risk for misuse are being released, so specific safeguards for responsible release are not applicable this context.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
21

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper credits original research via citations. Key software frameworks and libraries used (e.g.. PyTorch, DeepMind Control Suite, TD-MPC2) mentioned in Appendix A Their standard open-source licenses and terms of use have been respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The single reference file, provided in the supplemental material to illustrate core RWM control algorithm, is documented with inline comments. This serves to illustrate the key algorithmic ideas.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The research does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research does not involve human subjects, so IRB approval is not applica- ble.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: Large Language Models (LLMs) were not used component of the methods or experiments in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:::///eerrpss..ccooeeeeeeeee for what should or should not be described.
23


Reviews:
[{"review_id": "ysy0AAeHhD", "reviewer": "Reviewer_Mvgi", "summary": "The paper introduces Reflexive World Models (RWM), a dual-controller framework where learned world model predictions are used as reference trajectories for adaptive control. A Taylor decomposition of the value function motivates the split between slow policy learning and fast motor correction. The method is theoretically grounded and well-motivated, bridging adaptive control and reinforcement learning.\nHowever, the experimental section is underwhelming in scope and clarity. Key claims made in the introduction—particularly regarding generalization to nonstationary environments and robustness across tasks—are not adequately supported by the experiments. As a result, while the theoretical contribution is strong, the empirical validation does not match the ambition of the paper’s framing, and some of the broader claims appear overstated.", "questions": "- TD-MPC2 configuration unclear: The experiments compare RWM against a TD-MPC2 baseline, but it is not stated clearly whether TD-MPC2 is run with online planning or not. Since planning is a central component of TD-MPC-style algorithms, this distinction is critical. If the baseline is run without planning, its performance may be substantially handicapped. If it is run with planning, this should be stated more clearly in the main text and captioned in relevant figures.\n- What exactly is \\pi_0? As I understand it, \\pi_0 is the frozen policy from TD-MPC2 trained offline without planning (i.e., a reactive policy head). Can the authors confirm this? And is it accurate that this feedforward policy alone is used as the baseline in “No Adaptation”?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### **Strengths**\n\n- The core idea of treating world model rollouts as reference trajectories for rapid online correction is novel and conceptually clean.\n- The use of a second-order Taylor expansion to decompose the value function provides a solid theoretical foundation.\n- The approach avoids costly retraining and replanning and is lightweight at inference time.\n\n### **Weaknesses**\n- **Experimental section is disorganized** and hard to follow. It lacks a clear introduction, grouping, or interpretation of results. Readers are left piecing together the setups from fragmented descriptions (e.g., point mass in §5.2 appears without prior context).\n- **Limited environments**: Results are only shown on a basic 2D point mass and Walker2D and humanoid. Why no more tasks? For a claim about general-purpose adaptation in high-dimensional continuous control, this is too narrow.\n- Claims about generality (e.g., handling nonstationary dynamics or high-dimensional tasks) are not well supported.\n- **No comparison to alternative adaptation methods** (e.g., residual policy learning, meta-RL, or even classic MRAC). Only comparisons are to a TD-MPC2 baseline and no-adaptation (such as Online fine-tuning or Meta-RL baselines). MRAC baseline should be there as you mention it in the related work.\n- **Misleading claims** about nonstationarity: the only variation tested is **multiplicative noise on actuators**. This is not equivalent to structural changes like terrain variation (e.g., slippery surfaces), morphology changes, or sensor shifts.\n- Table 1 is unclear: it's not evident what \"TD-MPC2 (Full Training)\" refers to in contrast to \"No Adaptation,\" nor how the timing aligns with pretraining vs adaptation phases. While you do clarify this in the appendix, the main text should do a better job at surfacing this distinction. Without it, the implication that RWM is universally cheaper is misleading.\n- In the introduction. \"enabling learned policies to maintain performance under changing dynamics without requiring specific robustness procedures or architectural constraints.\". I don't agree with this. You show only robustness to noise on actuators.\n- Authors claim \"avoid costly replanning\". But it is not clear in the manuscript if there is a performance drop (see questions). Are you disabling planning in TDMPC2?\n- The experiments compare RWM (using a frozen TD-MPC2 policy with no online planning) to TD-MPC2 with full MPC. While RWM performs better under perturbations, this comparison is not entirely fair, since TD-MPC2 is not equipped with an adaptive controller. It would be more informative to evaluate RWM against TD-MPC2 with adaptive fine-tuning, or robust planning variants.\n-  In §5.2 they refer to a \"point-mass system\" without any prior introduction.", "was_revised": true, "final_justification": "The rebuttal’s additional results and clarifications make the experiments more convincing, addressing most fairness concerns. The empirical case for RWM is now stronger, though broader evaluation would still improve the work.", "rebuttal": "We sincerely thank you for your detailed and insightful review. We are encouraged that you found our core idea \"novel and conceptually clean,\" with a \"solid theoretical foundation.\" We appreciate the constructive feedback and would like to address the identified weaknesses and questions.\n\n1. On Limited Environments and Generality Claims:\nYou raise a valid point regarding the specificity of our claims on nonstationarity. In the revised manuscript, we will be more precise, framing our results as demonstrating robustness to dynamics arising from actuator drift and gradual miscalibration.\n\nTo better demonstrate the method's generalizability, we have also extended our experimental evaluation. We now have results on Cheetah, Hopper, Ball-in-Cup, and Humanoid, with a similar profile to the Walker simulations: stronger adaptation than the comparisons, though with a smaller effect for the Humanoid environment. Furthermore, to show the method's potential beyond actuator effects, our preliminary experiments on body mass perturbations in the Cheetah environment show the characteristic decrease in control error during perturbations. All new findings will be integrated into the final version of the paper.\n\n2. On Baselines and Comparison to Alternative Methods:\nThank you for these important points. Our goal was to test our specialized adaptation mechanism against relevant and strong baselines. \n\n- TD-MPC2 Baseline Configuration: We thank the reviewer for this question, which allows us to clarify a critical aspect of our experimental design. Our primary goal is to compare different reactive adaptation mechanisms, as online planning is often computationally prohibitive. A key consideration is that when a policy is trained with an online planner, it learns to provide actions that serve as a good starting point for the planner, not to solve the task on its own; the policy and planner become entangled. Using such a policy without its planner during evaluation would be an unfair comparison against a crippled agent. Conversely, keeping the planner active would introduce a confounding variable, making it difficult to isolate the performance of the underlying adaptation mechanism. Therefore, to ensure a clean comparison, our \"TD-MPC2\" baseline uses the reactive policy head (trained without the planner) and continues to fine-tune online. Both this baseline and RWM rely on the world model for learning — TD-MPC2 for its policy and value functions, and RWM for its corrective controller. The exception is the Humanoid environment, where a reactive policy alone is insufficient to produce stable locomotion. For this challenging task only, the TD-MPC2 baseline uses its full online planner to achieve a competent level of performance. We will clarify this configuration in the revised manuscript. To provide even stronger comparisons, we will also finalize our ongoing simulations using TD-MPC2 with planning, as appropriately suggested in your feedback, and include them in the revision.\n\n- Model Reference Adaptive Control (MRAC): Directly applying classical controllers like MRAC to high-dimensional tasks with learned, black-box dynamics models is generally not feasible. These methods require pre-defined, analytical reference models and full state observability. Furthermore, they are designed for tracking problems, typically with quadratic cost functions defined around a pre-specified reference trajectory. This is fundamentally different from the deep RL paradigm, which optimizes behavior based on general (and often complex or sparse) reward functions where the optimal trajectory itself must be discovered through interaction. A core contribution of our work is precisely bridging this gap: we introduce a mechanism that achieves MRAC-like reference tracking, but does so within a modern deep RL framework by using the world model's own predictions as an implicit, emergent reference trajectory.\n\n- Residual Policy Learning: This brings up a useful comparison. Residual Policy Learning often starts with a classical controller as a stable base policy and adds a flexible, learned RL policy as the residual to handle complex, unmodeled dynamics. Our work can be seen as a conceptual inversion of this design. We start with a flexible, general-purpose RL policy ($\\pi_0$) and add a corrective controller ($\\pi_c$) that is conceptually closer to classical control, as it performs deterministic error correction based on the world model's reference signal. This architectural similarity and distinct design choice will be clarified in our related work section.\n\n- Meta-RL: Our work addresses continuous online adaptation to unforeseen dynamic shifts, whereas Meta-RL learns an adaptation strategy from a pre-defined distribution of training environments. We therefore focused our comparison on online fine-tuning, which more directly aligns with our core research question.\n\n3. On Experimental Clarity and Organization:\nWe agree that the experimental section can be improved. In our revision, we will restructure Section 5 to first clearly define the experimental questions, then introduce all environments, and finally present the results systematically. We will also clarify Table 1 and properly introduce the point-mass system.\n\nWe thank you again for your time and effort. We are confident that these clarifications will significantly strengthen the manuscript."}, {"review_id": "s778b1VKQv", "reviewer": "Reviewer_o3ek", "summary": "### Summary\n\nThis paper makes the following key contributions:\n\n1. It proposes Reflexive World Models (RWM), a method designed to enable model-based control policies to rapidly adapt to specific types of perturbations.\n\n2. It provides a theoretical error analysis that offers insight into the reliability and limitations of the proposed approach.\n\n3. It conducts a series of experiments demonstrating the effectiveness and robustness of RWM across various scenarios.", "questions": "1. According to the definition in Equation (2), is your forward model $F$ only capable of predicting hidden states that evolve under $\\pi_0$? Or is it trained to fit a general transition model $F(z, a)$ independent of the specific policy?\n\n2. In line 100, you claim that a certain quantity is positive definite. However, it appears to be only semi-positive definite. Additionally, I found the narrative in Section 3.1 somewhat difficult to follow. The logical flow from line 100 to 102 is not very clear. While I can accept that $\\Delta z$ is small, if $z$ is already assumed to be near the optimum, it’s unclear why this optimization problem remains necessary.\n\n3. A minor formatting issue: none of the section or equation references in the paper appear as hyperlinks, and all references to the appendix lack section numbers. This makes navigation and reading somewhat inconvenient.\n\n4. A key underlying assumption for the validity of Equation (4) seems to be that $F(z_t, a)$ and $f(z_t, a) = e(P(s, a))$ are sufficiently close. Perhaps Theorem 4.2 is intended to justify this, but due to missing definitions, it is difficult to fully understand the statement and implications of Theorem 4.2.\n\n5. In Algorithm 1, the loop appears to run indefinitely. What is the intended stopping criterion? Is it a fixed number of epochs, a loss threshold, or perhaps a condition on gradient magnitude?", "limitations": "1. As noted in the weaknesses, the experimental setting is overly simplistic, which limits the generalizability and depth of the evaluation.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The paper provides a theoretical analysis to support its claims.\n\n2. The core idea is simple and practical, while also exhibiting a degree of novelty.\n\n---\n\n### Weaknesses\n\n1. The paper lacks a formal problem definition. Aside from a brief description around line 85, there is no rigorous formulation of the task. It would be helpful to include a dedicated problem definition section before Section 3. Moreover, since the proposed method focuses on optimizing a policy to match the world model's predictions, it is unclear what the exact objective is. Is the goal to train a policy to align with the outputs of a reliable, pre-trained world model? Or is it to maximize the accumulated reward in the traditional reinforcement learning sense?\n\n2. Several variables and functions are used without prior definition or clear explanation, including but not limited to: $V(\\cdot)$ in line 87, the base policy $\\pi_0$ in line 105, $\\theta_c$, $\\eta_c$, and $a_c$ in Equation (4); and $L$, $\\alpha$, $P$, $\\epsilon$, and $f$ in Assumption 4.1. In Figure 1, $Q_t$ and $s^*$ also appear undefined. While most of these can be inferred from context or domain conventions, the lack of explicit definitions significantly hinders readability.\n\n3. Some of the notation choices may lead to confusion. For example, $e$ is used to denote an encoder, but $e_t$ later refers to a loss value. It is unclear why the loss is denoted as $e_t$ in Algorithm 1 when $\\mathcal{L}$ has already been defined in Equation (3), and why a different notation $e(t)$ appears again in line 175. Unifying or clarifying these symbols would improve clarity.\n\n4. Beyond the mathematical content, the overall narrative of the paper could be clearer. If the main goal is to eliminate noise introduced by action perturbation, then the claim in line 143 that the approach “differs fundamentally from standard practice” seems overstated. In control literature, using neural networks to filter out noise is quite common. The contrast with model-based RL methods seems intended to highlight the novelty, but this distinction could be explained more precisely.\n\n5. The experimental evaluation lacks persuasiveness. Only one type of perturbation is considered—a simple multiplicative factor. Furthermore, the target world model is assumed to be pre-trained and near-optimal, which simplifies the learning objective: $a_c$ essentially only needs to approximate $a_c = -a_0 / (1 + p)$. The comparison baselines are also limited: one is a naive “no adaptation” baseline, and the other is continued RL training. The latter comparison, in particular, may be unfair, as it contrasts a supervised adaptation method with a sample-inefficient RL retraining baseline. As such, the empirical results offer limited insight into the real advantages of the proposed method.", "was_revised": true, "final_justification": "With the authors’ clarifications and additions, I now have a more complete understanding of the proposed method. As a result, I am willing to upgrade my assessment of the paper's originality, significance, and quality. The core idea and algorithmic insights are indeed interesting.\n\nHowever, from a theoretical standpoint, I find the contribution to be rather modest — the main theoretical results follow in a fairly straightforward and unsurprising manner. Thus, the theoretical part cannot be considered a highlight of the work. The key difficulty, in my opinion, lies more in conceptual interpretation than in technical depth. I would even suggest moving the formal mathematical results to the appendix, and keeping only the informal versions of Theorem 4.2 and 4.3 in the main text to improve readability and reduce unnecessary conceptual overhead.\n\nAs for the experimental part, my current impression is relatively lukewarm. Many of the key experiments were added only during the rebuttal phase, and there seems to be a lack of proper seeding or variance reporting — which undermines the reliability of the conclusions. I therefore cannot offer a strong endorsement at this time.\n\nOverall, I believe this paper has the potential to be significantly improved through a major revision. However, my primary concern lies in clarity. The issues raised by all three reviewers — including myself — largely stem from unclear wording, inconsistent organization and the lack of hyperlinks. I believe this is a fundamental problem that cannot be fully addressed through discussion alone.\n\nTaking all of the above into account, I believe this paper warrants a major revision. As such, I am willing to raise my score slightly to a 3.", "rebuttal": "We thank you for your exceptionally thorough feedback. You have provided a clear roadmap for significantly improving the paper's clarity and rigor. We acknowledge that our initial presentation had shortcomings, and we will address every point.\n\n1. On Lack of Formalism and Confusing Notation:\n\nWe sincerely apologize for the lack of clarity. We will add a new subsection to formally define the problem setting and a glossary to the appendix. To address the notational ambiguity you correctly identified, we will refer to the encoder as $\\text{enc}(\\cdot)$ throughout the text, distinguishing it from the error term $e_t$.\n\nHere are new formalism sections to be added before the model design, in addition to a Glossary to be added to the appendix: \n\n> We consider a continuous control problem formulated as a Markov Decision Process (MDP), defined by the tuple $(\\mathcal{S}, \\mathcal{A}, P, R, \\gamma)$, where $\\mathcal{S}$ is the state space, $\\mathcal{A}$ is the action space, $P(s'|s,a)$ is the transition probability, $R(s,a)$ is the reward function, and $\\gamma$ is the discount factor. Our goal is to learn a policy $\\pi(a|s)$ that maximizes the expected cumulative discounted reward. We operate in a setting where the true system dynamics, represented by the transition function $P$, can change unexpectedly from the training conditions.\n\n> Our approach leverages a learned world model operating in a latent space $\\mathcal{Z}$. The model consists of: an encoder $z_t = \\text{enc}(s_t)$, a latent dynamics model $\\hat{z}_{t+1} = F(z_t, a_t)$, and a base policy $a_0 = \\pi_0(z_t)$. The objective of our Reflexive World Model (RWM) is to learn a second, adaptive control policy, $\\pi_c$, which outputs a corrective action $a_c$. The total action is $a_t = a_0 + a_c$. The objective for $\\pi_c$ is to minimize the discrepancy between the world model's one-step prediction (under the base policy) and the observed outcome, thereby making the system behave as predicted by the reference model $F$ under the base policy $\\pi_0$.\n\n2. On Confusing Narrative and Experimental Design:\n\nYou raised several important points about the narrative and the experimental choices.\n\nOn Novelty and the Goal of the Method: We will clarify that the main goal is not merely to \"eliminate noise.\" The goal is to maintain the intended behavior of the base policy ($\\pi_0$) in the face of any dynamic shift that causes a deviation from the world model's prediction. Action perturbation is just one clean and measurable way to test this core principle. The novelty lies in re-purposing the world model's one-step prediction as an implicit reference trajectory for a controller, a general mechanism for adaptation that goes beyond simple noise filtering.\n\nOn Experimental Persuasiveness: You noted that our chosen perturbation, $a_{eff}=a \\cdot (1+p)$, simplifies the learning objective for our controller. We agree that our method is well-suited to this type of perturbation. However, we see this as a strength of the design, not a weakness of the evaluation. This form of perturbation is a realistic model for actuator degradation or miscalibration. The fact that our method provides a principled and direct solution to this common problem is a key contribution. Furthermore, the difficulty that the strong TD-MPC2 baseline has in compensating for this \"simple\" perturbation demonstrates that the task is non-trivial and requires a specialized mechanism like RWM for effective, rapid adaptation.\n\n3. On Technical Questions:\n\n- Q1 (Forward Model $F$): This is an excellent question. The forward model $F(z,a)$ is trained as a general transition model during the pre-training phase. This means it is exposed to a wide distribution of states and actions, including those from early, suboptimal policies, not just trajectories from the final converged policy $\\pi_0$. This broad training allows it to make reasonable predictions even for states that are off the optimal trajectory, which is crucial for providing a corrective signal when perturbations occur. We will clarify this in the revision.\n\n- Q2 (Hessian): Thank you. You are correct. The Hessian of the value function, $H = \\nabla^2 V$, should be negative definite near a maximum. We will correct this typo.\n\n- Q3 (Formatting): We will fix all hyperlink and referencing issues.\n\n- Q4 (Theorem 4.2): This is a key point. You are correct that the model $F$'s predictions will be wrong post-perturbation. However, our method does not require the absolute prediction to be correct. It relies on a weaker, more plausible assumption: that the local gradient of the model remains informative. While the perturbation shifts the dynamics, making $F(z,a)$ inaccurate, the relationship between an action change $\\Delta a$ and the resulting state change $\\Delta z$ is assumed to be roughly preserved. The model still \"knows\" that increasing a certain action will move the latent state in a particular direction. This is what the gradient $\\partial F/\\partial a$ captures, and as long as this gradient points in a useful corrective direction, our controller can successfully reduce the error. We will expand this explanation in the paper.\n\n- Q5 (Algorithm 1): We will clarify that Algorithm 1 describes the online update rule applied at each timestep of an episode until its end.\n\nWe are grateful for your diligence and are confident these revisions will resolve the noted issues."}, {"review_id": "IdN9j5qak8", "reviewer": "Reviewer_PGW8", "summary": "This paper introduces Reflexive World Model (RWM), a model-based control method designed to rapidly adapt to perturbed dynamics. \nRWM trains a residual correction online by minimizing the model prediction error (finding an action that better aligns the predicted next state with the \"intended\" next state).  \nWhen applied on top of TD-MPC2, RWM demonstrates improved adaptation performance under dynamic perturbations in the Walker2D environment. \nAlso, the authors provide a theoretical analysis on the error bound of the corrected policy.", "questions": "**Q1. More evaluation**\n* Could you include results of environments (e.g., Cheetah or Humanoid) for Figure 2 and Figure 3(a)? \n* This would provide a clearer picture of how RWM scales to high-dimensional, challenging systems.\n\n**Q2. Clarification on Figure 5**\n* In Figure 5, in the region without pertubation, the graph on the right shows that TD-MPC2 with domain randomization substantially outperforms RWM, whereas the performance appears similar in the left. \n* Could you clarify the discrepancy between these two views?\n\n**Q3. Mismatch of the line colors in the caption of Figure 2, 5**\n* It appears that the line colors described in the captions for Figure 2 and Figure 5 do not match the colors shown in the graphs. \n* Could you double-check and correct this?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n**S1. Novelty**\n* To the best of my knowledge, the proposed approach largely differs from prior methods, which typically focus on adapting the world model, rely on historical context or domain randomization. \n* In contrast, RWM’s focus on directly adjusting the action to achieve a desired next state simplifies the adaptation process and efficient than re-learning a new model and policy.\n\n**S2. Simplicity**\n* The motivation and implementation is clean and simple.\n* It does not require significant overhead during online adaptation.\n\n**S3. Theoretical Analysis**\n* Section 4 provides a theoretical bound on the error of the corrected policy, supporting the effectiveness of the proposed method.\n\n&nbsp;\n\n**Weaknesses**\n\n**W1. Limited Evaluation**\n* The method is mainly evaluated on the Walker2D environment. \n* Additional experiments on other control benchmarks (e.g., HalfCheetah or more challenging settings like Humanoid) would strengthen the paper’s claims and demonstrate broader applicability.", "was_revised": true, "final_justification": "The concern regarding the baseline TD-MPC was resolved.\n\nThis paper shows a novel approach in RMA methods, using world models instead of history-conditioned policies or domain randomization-based methods, which will be impactful in areas such as real-world RL.", "rebuttal": "We thank you for your very positive and encouraging review. We are delighted that you recognized the novelty, simplicity, and theoretical grounding of our work.\n\nQ1. More evaluation (Cheetah or Humanoid):\nThank you for this excellent suggestion. We have now evaluated our method on Cheetah, Hopper, and Ball-in-Cup, and have new results for Humanoid under perturbation, which confirm the generality of the results obtained for the Walker environment. We will integrate these new results into the final manuscript to better showcase the method's scalability and strengthen our claims.\n\nQ2. Clarification on Figure 5:\nThis is a great observation that points to a key distinction. The experiment in Figure 5 is different from Figure 2: the baselines in Figure 5 were pre-trained with domain randomization on the same perturbation distribution used for testing, whereas the baselines in Figure 2 were not.\n\nThe domain-randomized TD-MPC2 agent outperforms RWM in the \"OFF\" phase (no perturbation) for two reasons:\n1. This \"average\" condition matches its training distribution, making it an expert for that specific setting.\n2. The agent had not fully converged during pre-training and continues to slowly fine-tune its policy for this average case.\n\nHowever, the crucial test is performance during active perturbations (\"ON\" phase). Here, the pre-trained agent still suffers a significant performance drop, whereas RWM's online mechanism maintains higher reward. This demonstrates that pre-training on a known noise distribution is not a substitute for a true online adaptation mechanism like RWM. We will clarify this narrative in the main text.\n\nQ3. Mismatch of the line colors in Figure 2, 5:\nThank you for spotting this. We apologize for the error and will correct the figure captions to match the plots in the final version.\n\nWe thank you once again for your strong support and helpful feedback."}]

Author rebuttal:

Meta-review: This paper introduces Reflexive World Models (RWM), a novel and elegant dual-control framework for rapid motor adaptation. The core idea is to leverage a pre-trained world model's predictions as reference trajectories for a fast, low-level controller, effectively decoupling long-term reward maximization from immediate error correction. This approach is well-motivated by a theoretical analysis based on a Taylor expansion of the value function. The reviewers unanimously recognized the work's primary strengths: its conceptual novelty, simplicity, and practical potential for enabling lightweight adaptation in dynamic environments without costly online re-planning or retraining.

Initial reviews raised valid concerns, primarily regarding the clarity of the presentation and the scope and fairness of the experimental evaluation. However, the authors engaged in an extensive and productive discussion, providing additional experiments and data that largely addressed the most critical points, particularly concerning the TD-MPC2 baseline. Following the rebuttal, all reviewers positively updated their assessment of the work, acknowledging that their main concerns had been resolved. While the paper will still benefit greatly from a thorough revision of its presentation and the graceful integration of the new results as promised by the authors, the core contribution is strong and well-supported. Given the clear novelty of the approach and the significant improvements made during the review period, I recommend acceptance. I'm confident the authors will incorporate the excellent feedback from the reviewers to produce a polished and impactful final paper.
Decision: Accept (poster)