Title: Improving planning and MBRL with temporally-extended actions
Venue: NeurIPS 2025
Accepted: yes
Keywords: macro-actions

temporally-extended actions

model-based reinforcement learning

reinforcement learning

planning
Abstract: Continuous time systems are often modeled using discrete time dynamics but this requires a small simulation step to maintain accuracy. In turn, this requires a large planning horizon which leads to computationally demanding planning problems and reduced performance. Previous work in model-free reinforcement learning has partially addressed this issue using action repeats where a policy is learned to determine a discrete action duration. Instead we propose to control the continuous decision timescale directly by using temporally-extended actions and letting the planner treat the duration of the action as an additional optimization variable along with the standard action variables. This additional structure has multiple advantages. It speeds up simulation time of trajectories and, importantly, it allows for deep horizon search in terms of primitive actions while using a shallow search depth in the planner. In addition, in the model-based reinforcement learning (MBRL) setting, it reduces compounding errors from model learning and improves training time for models. We show that this idea is effective and that the range for action durations can be automatically selected using a multi-armed bandit formulation and integrated into the MBRL framework. An extensive experimental evaluation both in planning and in MBRL, shows that our approach yields faster planning, better solutions, and that it enables solutions to problems that are not solved in the standard formulation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of learning temporally extended actions in an MDP setting, for both planning and reinforcement learning. The proposed approach involves learning a transition model and reward estimates corresponding to repeating the same action over multiple time steps. The duration of each decision is also learned from experience using a non-stationary multi-armed bandit model. This method reduces search time in decision-making tasks by decreasing the number of decisions needed to reach a goal. The approach is evaluated across several benchmark domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The model-based representation of temporally extended actions is intuitive and well-motivated.\n2. The use of a non-stationary multi-armed bandit to model time step selection is innovative.\n3. The method is supported by a comprehensive empirical analysis.\n\nWeaknesses:\n1. The distinctions and similarities between the proposed model and other temporal frameworks (e.g., semi-MDPs, co-MDPs, PDDL2.1) should be articulated more clearly.\n2. The approach does not support fully continuous-time modeling.\n\nQUESTIONS:\n1. Does the model support concurrency, as in other frameworks (e.g., Mausam and Weld, 2008)? If not, could it be extended to handle concurrent actions?\n2. The temporal setting is not entirely clear to me. If I understand correctly, the model does not operate in fully continuous time but instead allows durations that are natural multiples of a standard time step. This seems more like an aggregated discrete-time setting than a true temporal one—is that correct?\n3. Assuming the previous point is accurate, does this design eliminate the need to integrate over stochastic effects? In other words, are the resulting stochastic outcomes simply the sum of effects over the individual execution steps?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the challenge of modeling continuous time systems using discrete time dynamics, which often requires small simulation steps and large planning horizons. This can lead to computationally demanding planning problems and reduced performance. The authors propose an approach that involves controlling the continuous decision timescale directly by using temporally-extended actions. This allows the planner to treat the duration of the action as an additional optimization variable, alongside standard action variables.\n\nThe authors applied the approach to both planning and model-based RL. in the MBRL setting, the authors learn a dynamic model that is conditioned on the action and its duration. \nthe author also propose to select the range of action duration dynamically over training using a multi-arm bandit method.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- the paper is well motivated and the approach is well presented. overall, it is easy to read the paper.\n- the authors provide various experimental results and that illustrate the effectiveness of their approach in various environments, including both planning and MBRL.\n\nWeakness: \n- MAB solution doesn't seem to improve over the simple fixed range. it converges much slower.\n- planning over temporally extended action can indeed reduce the number of search space but we can lose control granularity. the author focus primarily on successful experiments, it would be helpful to discuss some failure cases.\n- the experimental section focuses mainly on short-horizon tasks (locomotion continuous control tasks). do the authors expect that their method would work in really long-horizon task: for example navigation in big mazes.\n- The authors compare only to planning with primitive actions. could the authors includes results with model-free RL? especially I am interested in there is any gain of the proposed method in term of sample complexity.\n- Moreover, the sample complexity dimension is not discussed. I am not sure in in figure 2 the training iteration means environment interaction or gradient updates ?\n\nQUESTIONS:\nsee questions in weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel planning framework that dynamically adjusts the length of action repeats. The method automatically selects the optimal maximum repeat length with Multi-Armed Bandit (MAB) problem. Experimental results demonstrate that integrating dynamic action repetition into planning-based methods significantly reduces the number of decision points, thereby accelerating planning, while also achieving better performance compared to no action repetition baselines. MAB-based selection of maximal action repeat range also performs reasonably well in practice.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n**S1. Simplicity**\n* The proposed method is conceptually straightforward and easy to implement.\n\n**S2. Reduced Hyperparameter Tuning**\n* An important hyperparameter, the maximum action duration ($\\delta t_{\\text{max}}$), is automatically tuned through the MAB-based approach, reducing the need for manual tuning.\n* Moreover, the performance is robust to other parameters, such as discount factors (they indeed have a sweet-spot, but they do not have significant degradation on the overall performance ).\n\n**S3. Thorough Analysis**\n* The paper includes an in-depth analysis of various aspects of the proposed method, such as the effect of planning horizon, discount factor, and the use of model ensembles.\n\n&nbsp;\n\n**Weaknesses**\n\n**W1. Scalability to Complex Tasks**\n* While the proposed method shows promise, its scalability to more complex tasks remains uncertain (experimental results are given for simple tasks). \n* Specifically, in planning, two main sources of error—model prediction error and optimization error—can accumulate over long horizons. \n* The paper does not fully address how these challenges are mitigated in high-dimensional or long-horizon tasks.\n\nQUESTIONS:\n**Q1. Training Iteration vs. Number of Steps**\n* What is the update-to-data (UTD) ratio used during training?\n\n**Q2. Training of $F_{\\text{TE}}$**\n* How is the transition estimator ($F_{\\text{TE}}$) trained? \n* Training such a model requires data that includes action durations. How is this data collected, and what supervision signal is used for training $F_{\\text{TE}}$?\n\n**Q3. $F_{\\text{TE}}$ vs. $F_{\\text{IP}}$**\n* Can the authors provide a comparison of the prediction error between $F_{\\text{TE}}$ (transition estimator with duration) and $F_{\\text{IP}}$ (instantaneous predictor)? \n* This would help clarify the impact of modeling action durations for model prediction accuracies.\n\n**Q4. Comparison with Fixed $\\delta t$**\n* In many cases, a fixed action repetition value is used (as also mentioned in the introduction). \n* How does the proposed dynamic approach compare quantitatively against models using a well-tuned fixed $\\delta t$? Is dynamic repetition consistently better, or are there trade-offs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach for planning (and model-based reinforcement learning) that treats action duration as a decision variable. Instead of only issuing a primitive action at every simulator time-step, the agent samples both an action and how long to repeat it. To support this, the authors learn a temporally-extended dynamics model that predicts state-reward transitions over variable time gaps. They then frame the choice of the maximum admissible duration as a multi-armed bandit, where each arm corresponds to an upper-bound $\\delta t_{max}$. The resulting “action-plus-duration” space is then embedded inside a cross-entropy method (CEM) planner from prior works. Experiments on several continuous-control benchmarks (classic control, MuJoCo, Dubins car, Mountain Car) compare this temporally-extended agent ($A_{TE}$) against a baseline that plans with single-step actions ($A_{STD}$). $A_{TE}$ often reaches similar or higher episodic returns while requiring far fewer decision points and achieving notable wall-clock speed-ups. Impressively, $A_{TE}$ is also able to solve tasks which are unfeasible for $A_{STD}$ given compute and memory constraints.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n### Quality\n- The motivation is sound with a clear goal. The authors provide simple neat method to address the practical burden of long horizons by letting the planner adjust its temporal resolution online.\n- The paper delivers an end-to-end pipeline (bandit + ensemble world-model + CEM planner) and evaluates it in both pure-planning and learned-model settings.\n- Across many domains $A_{TE}$ achieves equal-or-better return with fewer decisions and significantly faster training. The authors also provide an impressive number of hypermeter ablations. The ones on the effect of the different scale discount factors was particularly interesting.\n\n### Clarity\n- The high-level intuition (trading planning depth for breadth via action durations) is easy to follow, and the algorithmic components are standard.\n- Visualisations of decision frequency and horizon length make the speed-up tangible.\n\n### Significance\n- A generic, task-agnostic way to obtain temporal abstractions could benefit robotics and MBRL practitioners.\n- The work reconnects classical durative-action planning ideas with modern learned models.\n\n### Originality\n- Optimising action duration inside a sampling-based planner and adapting the duration range via a bandit scheme appears novel.\n- Sparks theoretical questions about search-space complexity under temporal abstraction.\n\n# Weaknesses\n\n- The paper repeatedly frames itself as “MBRL” even though no policy is learned. Policy learning is a key key component of RL (see MBRL works like Dreamer for example). This work is closer to model-based planning with a learned simulator. This distinction needs to be explicit from the outset.\n\n- There's a general lack of preciseness. For example, heavy reuse of “=” for definitions and equalities (esp. Sec. 3) hurts readability; “:=” or “≜” is recommended. It is also unclear how the update in step 9 of Algorithm 1 is done (Algorithm A1 references also hyperlinks to Algorithm 1). Section 4 is imprecise in general. \n  - Primitive dynamics function (f) and temporally-extended dynamics function (F) are not precisely defined. What are their domains and ranges?  \n  - The iterative primitive dynamics function ($F_{IP}$) in line 151 is not defined nor explained in the main text despite being referred to throughout. \n  - It is unclear what the statement $\\hat F_{TE} \\approx F$ means. Does that sentence (line 159-160) mean the authors only use near-perfect models in the experiments? \n  - It is still unclear how the neural network is trained precisely. What is the input space, and what is the output space? What is the set of arms (is it $\\\\{\\delta t_{min}, 2\\delta t_{min}, ..., m\\delta t_{min}\\\\}$)? What is \"candidate i\" in line 180 (is it a bandit arm i)? What's the informal justification for equation 3?\n  - It seems like the approach also requires a suitable choice for m (for the number of arms and the upper-bound action duration arm), but this is not discussed.\n\n- The claim \"Using temporally-extended actions helps to reduce the space of possible trajectories\" is false (line 192). It strictly increases the search space, since the search space now also includes the selection of action durations (and this is not even including the selection of $\\delta t_{max}$). For $A_{TE}$, the search space is at least $\\bigcup\\limits_{k=1}^{m} (k|\\mathcal{A}|)^{H/k}$. Similarly, the claim \"$A_{TE}$ needs to optimize less variables than $A_{STD}$\" is false (line 197).\n\n- Hence one would expect $A_{TE}$ to take longer to plan due to the increase in the search space, but lead to shorter plans than $A_{STD}$ due to the temporally extended actions. However, this is not what is observed in the experiments. In fact, even when the authors demonstrate results in environments where temporally extended actions are not appropriate (Pusher and Reacher in Fig 3), $A_{TE}$ still performs similarly to $A_{STD}$ despite the much larger search space (and the fact that the dynamics model that is being learned is also more complex for $A_{TE}$). This suggests a potentially serious hyperparameter bias. That's $A_{TE}$ planning may be getting enough sequence samples, planning iterations, and model training batch-size to account for the increase in the search space size. Can the authors run ablations of these hyperparameters for $A_{TE}$?\n\n- The approach uses several models in it's ensemble (for example a separate model for each bandit arm). It is unclear if the authors used the same total number of models for the baselines. Additionally, the use of only 5 seeds hurts the significance of the demonstrated results.\n\nQUESTIONS:\nPlease refer to my weaknesses above. Mainly:\n\n- Define all functions and spaces. Provide explicit domain/range (e.g. in terms of the state and action spaces) for f, F, $F_{IP}$, $\\hat F_{TE}$. \n- Please comment on the point regarding the search space claims. If my interpretation is correct, please revise the claims in the paper accordingly.\n- Ablations for fairness: \n  - Match the total number of ensemble models across ASTD and ATE, or show that model count does not drive performance.\n  - Can the authors run ablations of these hyperparameters for $A_{TE}$: number of sequence samples, planning iterations, and model training batch-size (using smaller values than the ones used currently for the experiments)? Or provide a discussion that addresses my concern here.\n\nLIMITATIONS:\nAn analysis or discussion of the failure cases of the proposed approach is not done. That is, settings where temporally extended actions are not desirable in the large majority of the state space. For example, the experiments are only in deterministic environments, even though the approach is presented to be more general. How well does the approach handle stochastic dynamics (e.g. the U-map navigation with increasing noise on the dynamics)? Also safety-critical environments, for example the cave-mini map modified such that hitting walls leads to failure.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}