Improving planning and MBRL with temporally-extended actions
Authors: Palash Chatterjee
Roni Khardon
Venue: NeurIPS 2025 poster ()
Keywords: macro-actions
temporally-extended actions
model-based reinforcement learning
reinforcement learning
planning
Primary area: reinforcement_learning

Abstract: Continuous time systems are often modeled using discrete time dynamics but this requires a small simulation step to maintain accuracy. In turn, this requires a large planning horizon which leads to computationally demanding planning problems and reduced performance. Previous work in model-free reinforcement learning has partially addressed this issue using action repeats where a policy is learned to determine a discrete action duration. Instead we propose to control the continuous decision timescale directly by using temporally-extended actions and letting the planner treat the duration of the action as an additional optimization variable along with the standard action variables. This additional structure has multiple advantages. It speeds up simulation time of trajectories and, importantly, it allows for deep horizon search in terms of primitive actions while using a shallow search depth in the planner. In addition, in the model-based reinforcement learning (MBRL) setting, it reduces compounding errors from model learning and improves training time for models. We show that this idea is effective and that the range for action durations can be automatically selected using a multi-armed bandit formulation and integrated into the MBRL framework. An extensive experimental evaluation both in planning and in MBRL, shows that our approach yields faster planning, better solutions, and that it enables solutions to problems that are not solved in the standard formulation.

Improving planning and MBRL with (emporally-extendel actions
Palash Chatterjee Indiana University palchatt@iu. edu
Roni Khardon Indiana University rkhardon@iu edu
Abstract
Continuous time systems are often modeled using discrete time dynamics but this requires a small simulation step to maintain accuracy. In turn, this requires a large planning horizon which leads computationally demanding planning problems and reduced performance. Previous work in model-free reinforcement learning has partially addressed this issue using action repeats where a policy is learned determine discrete action duration. Instead we propose to control the continuous decision timescale directly by using temporally-extended actions and letting the planner treat the duration of the action as an additional optimization variable along with the standard action variables. This additional structure has multiple advantages. It speeds up simulation time of trajectories and, importantly, it allows for deep horizon search terms of primitive actions while using shallow search depth in the planner. In addition, in the model-based reinforcement learning (MBRL) setting, it reduces compounding errors from model learning and improves training time for models. We show that this idea is effective and that the range for action durations can be automatically selected using multi-armed bandit formulation and integrated into the MBRL framework. An extensive experimental evaluation both in planning and MBRL, shows that our approach yields faster planning, better solutions, and that it enables solutions to problems that are not solved in the standard formulation.
Introduction
Many interesting real life systems evolve continuously with time, but the dynamics of such systems are often modeled using a discrete-time approximation. In these models, time evolves in discrete steps of 6, called the timescale of the system. Simulators used in RL or robotics often use such models to capture physical systems. Setting òe small value allows the discrete fiscrete-time models to have good local approximation. To obtain a trajectory of the system using such models, the dynamics function needs be evaluated fixed intervals of & Because is small, the number of decisions required to solve even simple task can be quite large. From the perspective of planning or MBRL, this translates to longer planning horizons (or rollouts) which can limit their effectiveness.
Shooting based planners like Cross Entropy Method (CEM) and Model Predictive Path Integral (MPPI) [Kobilarov, 2012, Botev al., 2013, Williams et al., 2017, Chua al., 2018] rely on sampling to estimate the highly rewarding regions of the state space. They are known to perform poorly as the planning horizon increases, especially in environments with noisy dynamics or sparse rewards. This is because in such environments the variance of their estimates increases and they need a large number of samples in order to act reliably [Chatterjee et al., 2023]. In addition, when planning with learned models as in MBRL, longer rollouts can lead to compounding errors [Janner al., 2019].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Simulators like MuJoCo [Todorov et al., 2012] or Arcade Learning Enviromment [Bellemare et al., 2013] make use of a fixed frame-skip in addition to timescale. A frame-skip or action repeat of means that the same action is repeated for n times before the agent is allowed to act again. So the decision timescale becomes n 6t. Frame-skip values are normally set heuristically, but as shown by Braylan et al. [2015], the ideal values of frame-skip vary depending on the environment and can heavily influence the performance of the agent. This has led to many efforts in trying to control the decision timescale by learning policy to control either frame-skip [Durugkar et al., 2016, Lakshminarayanan et al., 2017, Sharma et al., 2017], or timescale [Ni and Jang, 2022]. To the best of our knowledge, none of the previous approaches modeling the decision timescale consider the planning problem or the use of planning in MBRL.
In this paper, we propose to control the decision timescale directly by treating the action duration (ót) as an additional optimization variable. At each decision step, the planner optimizes for the standard primitive action as well as its duration resulting in a remporally-extender action. In contrast to using a fixed frame-skip, the duration for each temporally-extendee planner to choose actions of varying durations, can be seen in Figure 4. This provides the planner with an added degree of flexibility and, at the same time, it reduces the search space by constraining the structure of the trajectories that the planner searches over. Further, as small increase in the temporally- cerporally-extendeed extended planning horizon can translate to a large increase in the primitive planning horizon, this allows the agent search deeper, enabling it to solve environments which are otherwise too difficult.
action can be different allowing the
Finally, unlike previous work, we do not learn a policy. Our work is in the context of MBRL. We learn model that predicts the next state and reward due to a temporally-extended action and use the learned model to plan, leading to superior performance as well as faster training. Selecting suitable range for št is an important step as it can directly impact the performance of the planner. We show that posing this as a non-stationary multi-armed bandit (MAB) problem [Besbes et al., 2014, Lattimore and Szepesvári, 2020] allows us to select the range automatically.
To summarize, our main contributions are as follows:
1. We propose to control the decision timescale directly by letting the planner optimize for primitive action variables as well as the duration of the action, and evaluate this idea both in planning and in MBRL.
2. We show that our approach decreases the search space and the planning horizon for the planner, helping it solve previously unsolvable problems. It also improves the stability of the search by allowing the planner to search deeper and solve problems with sparse rewards.
3. We show that learning temporally-cxtended dynamics models and selecting a suitable range for öt can be integrated into a single algorithm within the MBRL framework, and that planning using these learned models leads to faster training and superior performance over the standard algorithm.
2 Related Work
Macro-actions [Finkelstein and Markovitch, 1998, Hauskrecht et al., 2013] and options [Sutton et al., 1999] are two of the most common methods introduce temporal abstraction in planning and RL. A macro-action is a usually defined to be sequence of primitive actions that the agent will take. On the other hand, an option consists of a policy, an initiation set and terminating condition. The initiation set determines when the option can be taken, while its duration depends on when its terminating condition is satisfied. Both macro-actions and options can either be predefined or can be learned from data [Durugkar et al., 2016, Machado et al., 2017a.b, Ramesh et al., 2019].
Frame-skipp skips or action-repeats are simpler forms of macro-actions where each macro-action is essentially a single primitive action repeated multiple times. While frame-skipping has been used as a heuristic in many deep RL solutions [Bellemare et al., 2012, Mnih et al., 2015], Braylan et al. [2015] showed that using static value of frame-skips across environments can lead to sub-optimal performance. Following this, there have been multiple efforts using the model-free RL framework to make the frame-skip dynamic. Lakshminarayanan et al. [2017] propose to learn a joint policy on an inflated action space of size n ||| where each primitive action is tied to n corresponding action-repeats. The value of is usually small to limit the size of the inflated action space. A more

---

practical approach is to learn a separate network alongside the standard policy to predict the number of action repeats [Sharma al., 2017, Biedenkapp al., 2021].
Frame-skips or action-repeats are discrete values and, while they introduce a temporal abstraction for discrete time systems, a continuous representation is both more realistic and more flexible. Some prior work in model-free RL has explored this problem. Ni and Jang [2022] use modified Soft Actor Critic [Haarnoja al., 2018] to learn a policy and control the timescale rather than the action duration, with an explicit constraint on the average timescale of the policy. Their formulation of the optimization objective is similar to ours except that they use a linear approximation of the reward function to compute the reward due to macro-action (details in Section B). Wang and Beltrame [2024] introduce Soft-Elastic Actor Critic output the duration of the action along with the action itself. However, they modify the reward structure by introducing penalties for the energy and the time taken by each action. Our work is in the planning and MBRL. setting. Rather than learn a policy, we use either exact or learned transition and reward functions to plan. Our objective arises naturally from the formulation without the need to add additional constraints or engineer rewards. Further, previous work in model-free RL treat the maximum value of action-repeats Instead, we use MAB framework, similar to Li et al. [2018] and Lu et al. [2022]. and automatically select the maximum action duration removing the need for tuning an additional hyperparameter.
timescale as hyperparameter.
Finally, in our work, action durations are chosen by the planner, which is different than planning with durative actions [Mausam and Weld, 2008] where the durations are given by the environment.
3 Modeling cemporaaly-Exxtenee Actions
The standard sntcrete-time Markov Decision Process (MDP) is specified by AS R,7} where S and A are the state and action spaces respectively and E (0, 1) is the discount factor. & E S and a A represents the state and primitive action at timestep l. The one-step transition distribution is given by (( a+) and the one-step reward distribution is given by (R at). The expected
discounted return given by Jr = EPD 7 y'R(s145; 0c44)) where D is the planning horizon.
Discrete time simulation of continuous systems assumes that T and R capture the transitions and the corresponding reward due to exactly 6?Th duration, where 8TTH is the timescale of the MDP. In cases when T and R are unknown to the agent, their empirical estimates, T and R, can be learned using data collected by interacting with the MDP. For the following discussion, we overload the term dynamics to mean both the transition and reward function.
Although we eventually care about what primitive actions to take in the environment, a planner can work at an abstract level by using temporally- remporally-extended extended actions. We use the terms decision steps and execution steps distinguish between the number of times the agent outputs an action and the number of primitive actions that are actually executed in the environment.
Let us assume that the agent has access to the primitive dynamics function (f) which is accurate for all 0 S S detT In the standard setup, the agent uses f at each decision step and outputs an action whose duration is implicitly deth The return due to trajectory of length L(T) is given by
1= -/+) R 0e) Here, the number of execution steps is exactly equal to the number of decision steps.
J1
In our proposed framework, the planner explicitly outputs the duration of the action along with the action itself. At decision step k. let the planner output an action dg and its corresponding duration Stg E |ōtmin 8tmaax] Now the number of execution steps need not necessarily be equal to the number of decision steps. Let c = _ + +1 (ôt mod 8tTh)) be the number of execution steps associated with decision step k. Further, let eck ΣØ= ej be the total number of execution steps taken prior to decision step k. The return due to trajectory will be
L(T)
J/
"(s((ce (e)
(I)
1=1
where e<1 = 0. This formulation provides an approximation that avoids the need for a precise continuous time model of discounting. Note that when ôt, = 85Th for all k then J2 J1-
3

---

Algorithm One decision step Action selection using a lemporally-extended dynamics function (F) with a generic shooting- based planner
Require: lemporally-extended dynamics function (F), current state (s)
initial action distribution (Na, var.), number of rollouts (N), planning horizon (DrE)
1: for = 1 to optimization steps do
2: Sample N action sequences of length DTE using Mas var, do 4: //a; includes standard action variables and the action duration
3: for all N action sequences are =
5:
Simulate trajectory using F from s due to am and compute aggregate reward 6: end for
7: //update rule is specific to the planner being used.
8: Update Ma, var, using action sequences and aggregate rewards
9:
end for
10: return sample action using Na: var
Let R = Ei= 12-12-211 RR +t) (1)=++t) be the reward due to the temporally-extendee action at decision-step Then we have slightly different view of J2 where the returns due temporally- extended action is discounted based on the number of primitive actions taken prior to the current timestep. Formally,
L(T)
L(T)
J3
Rock
TE
Gakk 11
12 t-
R(((R(((e))))) ((ectt))
11
1=
This view allows us to have different discount factors, ? and 72. giving us more fine-grained control over the behavior of the agent.
Method
We want the planner to have access to temporally-extended dynamics function (F) that can work with temporally-extendee example, one can use shooting-based planner with J to select an action from any state as shown in Algorithm 1. However, F is usually not readily available. In the standard plamning setup, the agent has access to the primitive dynamics function (f) which samples from the step primitive transition and reward distribution. A simple solution is to wrap f in loop (as shown in Algorithm Al in the Appendix) to obtain Fip. This holds as assumed that ff is accurate for all 0 < t S 87Th We call this the iterative primitive dynamics function. Although Fip accurately captures F, it fails functioon to facilitate the speed of execution. The time required by Fip simulate the outcome due temporally-extenn exxended action is dependent on the duration of the action itself. However, if we had access F. this would have been constant time evaluation.
actions. If F is available, then using for planning is straightforward. For
Approximating F : We address this by using neural networks to approximate using ÂTE. This allows to predict the next state and reward due to temporally-extent state in constant time. For learning this model, we use the framework of MBRL. We collect data by interacting with the environment and use the data to train FIE to predict distribution over the next states and a point estimate for the reward. The learned model can then be used for planning. For example, Algorithm I can be used with F'r instead of F to select actions, as we do in this work.
action from given
Selecting óin and òtmax : Unlike other action variables, the range of does not come from the environment and has to be set by the planner. If the range is too large, the search for the optimal öt becomes harder while a smaller range can limit the search. However, as the range simply influences the spread of the distribution used to sample òt, its value need not be set precisely.
We set 6t = 8:Th to be the timescale of the environment. For 8t we consider m exponentially- spaced candidates, where m loga (T') and T is the maximum number of primitive steps defined by the environment. We model each 86max candidate as an arm and pose selection as a MAB problem. Pulling an arm is equivalent to selecting one of the 8lmax candidates and using the planner with this value of 5tmax to collect data for one episode. The aggregate reward observed at the end of episode indicates the quality of the arm. However, note that the rewards obtained are directly

---

related to the quality of the learned dynamics model the agent has. As the agent collects more data, the model quality and hence the reward distribution of the arms change, making the problem of arm selection non-stationary. To address this we propose to use an exponential moving average that focuses the reward estimate on recent episodes and hence ameliorates the effect of non-stationarity.
Let Rr be the average reward per timestep obtained in episode T and let R.,T RAT-- a(Rr
RAT-1) be the exponentially moving average (EMA) of rewards per timestep obtained till episode T due candidate i. Let (i, T') be the number of times candidate i was chosen before iteration T. At the start of iteration T + 1, mtx is selected using the UCB heuristic [Auer et al., 2002] in Equation (3) where we set Tu to the mean reward obtained across 5 episodes using the randomly initialized dynamics models, prior any training.
2 log RRST + + N(i.))
arg max
Another important question whether learn single dynamics model be shared by all the arms or learn separate dynamics model for each arm, Our preliminary experiments, which we describe in Section 1.1, suggested that learning separate dynamics model for each 6tmax candidate is better.
To summarize, we treat each stmax candidate as a bandit arm. Before each episode, we use a UCB heuristic with EMA of rewards to select arm and the selected value of ôl and the corresponding dynamics model to plan for the duration of the episode.
Discussion
Consider two agents ASTD that uses primitive actions and ATE that uses temporally- extended actions and let DSTD and DTE be their corresponding planning horizons. Further, for ATE. let max m x 8hhh for some positive integer
The planning horizon for the two agents are at different scales and not directly comparable. We introduce the term maximal primitive horizon (H) which is the maximal number of primitive actions taken by the agents while planning. For ASTD. H DSTD. while for ATE, H m DIE- A fixed value of H ensures that, while planning, ATE. does not consider more primitive actions than ASTD. at any instant. For practical purposes, ATE. will take less primitive actions than ASTD- We argue that even in this unfair setting, using temporally-extenne
actions can be beneficial.
Using temporally-exter extended actions helps to reduce the space of possible trajectories that ATE searches over. For simplicity, let us consider an environment with binary actions. The search space for ASTD is 2H, while for ATE, the search space reduces to H/m However, this reduction comes a cost of the flexibility of trajectories as when is larger the generated trajectories have more rigid structure.
ArE needs to optimize less variables than ASTD- In general, if the action space has dimensions [A], then at each decision step. ASTD has to optimize for H ||| action variables, while ATE will have to optimize for (H/m) + 1) action variables.
3. FIe evaluates the outcome of cmmporally-extende that taken by f to evaluate the outcome of a primitive action. As DIE < DSTD for all practical purposes, using I' with ATE helps it make decision faster than ASTD-
actions in constant time, which is similar to
Manipulating the value of m allows us to scale up the maximal planning horizon of ATE without adding any extra variables. This can be useful in environments where rewards are uninformative and a deeper search required.
Experiments
We evaluate the use of temporally-extente actions in planning and in MBRL. We begin by elaborating on our method for both these settings, before discussing the experiments and results.
Experimental setup We compare the planning performance of ASID and An using CEM, which shooting- based planner. CEM maintains sequence of sampling distributions from which generates multiple action sequences. For each action sequence, it instantiates multiple particles and computes the trajectory and reward due to each particle. Then, it computes the mean reward per action sequence and uses the top l action sequences to bias its sampling distributions. For evaluation,
5

---

Algorithm 2 Using ArE with a fixed 5lmax as planner for MBRL
Require: dynamics model ensemble(Fri) (F:), batch size (b), number iterations (n), 8taxx I: Initialize empty dataset D
2: for = 1 to n. do
3: //data collection
4: for / = 1 to end of episode do
5: //a; includes standard action variables and the action duration coti 0lmax)
6:
0r = plappaen. choose action
7:
//env.step() implicitly uses temporally- extendee dynamics St+1, = env env. (a;
8:
9:
D D. apppnd ((5;; ae 8t41,Te))
10:
end for
//training
12:
num of obacchsseseeeeaaeaeeeeeeeeeeeee_ = len (D) / b for j 1 to num_of_batcher do for each dynamies model in FTE do
15:
Sample mini-batch of size b from D
Compute the loss and gradients using the mini-batch Update the model using the gradients end for
17:
18:
19:
end for
20:
end for
we wrap the primitive dynamics function of the simulation environment in an iterative loop using Fip as discussed in the previous section.
For experiments with RL, following Chua et al. [2018], we learn an ensemble of neural networks to approximate the dynamics function. Specifically, ASTD learns approximate while ATE learns to approximate F. Both the agents use the same network architecture and configuration of parameters. The only difference that the model learned by ATE has an extra input variable corresponding to 8t, in addition to the state and action variables. The data collection and the training procedure for ATE with fixed mtx in described in Algorithm We use the TSoo variant from Chua al. [2018] for planning with the learned ensemble of dynamics models (see Section E for details about planning with ensembles). Note that ASTD equivalent to TS from PETS [Chua et al., 2018).
We consider two variants of our proposed algorithm- Arg which uses fixed and manually chosen value of ōtmax and (D) which uses the proposed MAB framework to dynamically select the value of max at the beginning of each episode. Both variants use 8t.in 85 A key difference between the
that (D) maintains a separate dataset and learns a separate temporally-exter -xxtendd dynamics model for each 8t,xx candidate. Specifically, ATE(F) uses Algorithm 2 exactly, while ATE (D) has an additional step where uses the MAB framework to select 8tmax value before the start of each episode and then uses the corresponding dataset and model for the duration of the episode. For our discussion, we use ATE to refer both the variants in general while specifying the variant wherever required.
Experimental Evaluation
First, we experiment in planning regime where the agent has access to the exact dynamics function. In this setting, ATE uses iterative primitive dynamics function (Algorithm Al) for planning. For experiments, use the Mountain Car environment from Gymnasium [K IKwiatkowski et al., 2024], multi-hill Mountain Car environment from the Probabilistic and Reinforcement Learning Track the International Planning Competition (IPC) 2023 [Taitler et al., 2024], and the Dubins car environment from Chatterjee et al. [2023]. Then, we experiment the MBRL setting where T and R are not known. In this case, we learn ceapporall--ettnnde model discussed above, by interacting with the environment. We use Cartpole from Gymnasium and Ant, Half Cheetah, Hopper, Reacher, Pusher and Walker from MuJoCo [Todooov al., 2012]. We each experiment across 5 different seeds and aggregate the results. Full details of the experimental setup are given in Section E.
The experiments organized to answer set of questions outlined below.

---

Instance
Rewards
Decision sleps +
Time for an episode
Success Probubility 1
Standard
Ours
Standard
Ours
Standard
Ours
Standard
Ours
90.98 0.26
1.55
108.8 +:
3.0
4.86
0.19
5.7
100
2
0.11 0.01
88.67 + 0.65
300.0 1 0.0
2.8 1: 1.1
7.98 1: 0.28
5.59 0.91
0.00
1.00
0.01
86.15
+ 1.11
3.2 2.8
7.94 0.15
5.44 0.76
0.00
1.00
4
-0.1 0.01
66.50
+ 43.54
300.0 + 0.0
7.73 1133=00 0.28
6.01 1.05
0.00
0.80
5
-0.1 1: 0.01
83.41
+ 0.54
300.0 + 0.0
3.0
8.0 =: 0.28
5.82 0.2
0.00
1.00
Table 1: Results on Multi-hill Mountain Car from IPC 23 across 5 seeds where ATE (Drr = 12)
solve all instances while ASTD (DSTD 175) can only solve 1/5.
(a) ASTD. |A|=2
(b) ATE, |A|=2
(c) ATE. |A|=
Figure 2: When the number of actions is small (as in and b), both the agents are able to solve the problem. But when the number of actions increases (c), ATE is still able to solve the problem while ASTD fails due to large memory requirements. The shape of the curves is an artifact of the action space in this environment. Note that constant action in acceleration space yields curved paths. The path in (c) is composed of actions different durations, as marked by the colors.
Does planning with temporally-extente actions help? Mountain Car has sparse reward, requiring a large planning horizon to succeed. compare the performance ASTD and ATE by varying the planning horizon. As shown in Figure 1, ATE Dre 2 4) solves the environment with much smaller planning horizon than ASTD DSTD 2 60).
Next, we experiment with the multi-hill version of Mountain Car from IPC 2023 (illustrated in Figure Alb in the Appendix). Each instance of the environment increases the difficulty by either adding more hills or altering the surface of the hills. shown in Table I, while ASTD is able to solve only the first instance, ATE solves all the instances. Note that, even though ATE takes small number of decision steps, the computation time required to finish an episode comparable to ASTD. This is because ATE uses iterative version of the dynamics (Fir) for simulation. Another observation is that because the episode terminates when the agent reaches the goal, the number of decision steps is smaller than DTE.
Planning Horizon (Ours)
150
100
Rewards
0
-50
40
60
80
100
Planning Horizon (Standard)
Figure 1: ASTD requires a large planning horizon (DSTD 60) to succeed in Mountain Car, but ATE using 8t = 100 can work with small planning horizon (DrE 2 4).
Can temporally-extene form infeasible problems to feasible ones? We use the Dubins Car environment (d = 0.2) and experiment with u-shaped map shown in Figure 2. This is a challenging configuration, where shooting-based planners often fail, because the car is initially facing the obstacles and a naive forward search hits the obstacles and does not yield useful information. In addition, as the reward is sparse, this map requires the planning horizon to be large. To solve the environment, AsTD requires 10,000 samples with planning horizon of 1000, while ATE requires a planning horizon of 75 and ötmax 20 (see detailed discussion in Section G.2).
actions help trans-
To increase difficulty, we augment the action space with 100 dummy action variables. Although these variables do not contribute to the dynamies or reward, the agent is unaware this and has account for all the action variables. We show that in this difficult setting well, ArE succeeds while ASTD
7

---

fails due to large memory requirements. A simple computation shows that ASTD needs around 4GB of memory to track the sampled actions, whereas ATE: requires just 103MB. In other configurations that reduce the memory requirements (detailed in Section G.2), the search of ASTD fails to find the goal. This shows that using tenporally-extended actions can often turn infeasible search problems into feasible ones.
How does varying the two discount fac- tors impact the agent's behaviour? The experiments so far use single value o y
7
72
Decision Steps
Primitive Steps
while the proposed formulation in Equa- tion (2) has two discount factors. As we
1.0
20.2 +: 0.84
121.8 + 2.95
0.99
20.2 1 0.45
122.0 1: 2.00
Case I: Fixed
0.99
0.9
21.6 + 1.34
121.8 = 0.84
discuss next, while some expectations on
1
122.6 1.52
the effect of ?/1 and "/2 are intuitive, a com- plete characterization is not obvious.
0,7
23.2 + 0.45
122.8 = 1.64
1,0
19.0 1 0.71
128.6 1: 1.52
0.99
20.2 + 0.84
121.8 + 2.95
Case 2: Fixed 7y2
First, we note that even though DTE fixed, the primitive planning horizon is dependent on the duration of the actions chosen by the planner. This can result in different levels of discounting in different trajectories. In contrast, in the standard framework, the dis- counting due to planning horizon DSTD is fixed. This makes predicting the behavior of ATE, upon varying "/1 and 72, difficult. Second, for all decision steps >> 1, 7 will always be the dominating component in the objective. This is because the exponent term for " is the number of primitive steps before the decision step k. while the exponent term for 7/2 is proportional to the duration of the temporally-extender action.
+:
117.6
0.9
16.4 1: 0.55
115.4 :: 0.55
Table 2: When 7 is fixed, decreasing 7/2 leads to an increase in number of decision steps. When 7/2 is fixed, decreasing 71 leads to decrease in the number of prim- itive steps. Results averaged across 5 random seeds.
To explore the impact of 7/1 and 7/2 we fix one and vary the other. We use the cave-mini map (Figure Ala) in the Dubins Car environment, where the agent has to navigate through multiple obstacles. We have two hypotheses. First, for a fixed 71, smaller values of 72 will prefer shorter action durations and hence larger number of decision steps. Second, for a fixed ?2, decreasing ?1 should prefer longer action durations and a smaller total number of primitive actions. The intuition is that if we reduce ?1 and the planner does not reduce the number of primitive actions, then the impact of discounting on the overall objective will be higher. Table 2 confirms both these trends.
Do the benefits transfer if we learn the dynamics? Having access to the transition and reward function is not realistic. We would like be able to learn these as we interact with the environment. For this experiment, we use Ant, Half Cheetah, Hopper, Reacher, Pusher and Walker from MuJoCo along with Cartpole. By default, the MuJoCo simulators use a preset value of frame-skip which vary depending on the environment. This results in the effective timescale being greater than the original timescale. For our experiments, we modify the environments to use a frame-skip of 1.
ATE uses a learned temporally-extended model for planning, while for evaluation we wrap the primitive transition and reward functions in an iterative loop similar to Fi.. ATE(F) uses fixed 5tmax throughout the entire learning process while ATE(D) uses the proposed MAB framework to dynamically select the value of ml at the start of every episode.
Each experiment run for specific number of iterations, where in each iteration, we train the model using multiple mini-batches drawn from the dataset of past transitions, and then use the trained model to act in the environment for one episode. The rewards collected during the episode are used for evaluating the agent's 's performance. We plot the mean and standard deviation, across 5 different seeds, of the running average of scores obtained as well as the number of decisions made by the agents across training iterations. The main results from our experiments are shown in Figure 3.
First, we review the results across environments before discussing environment specific observations. Overall, ATE performs better than ASTD while being faster. ATE(F), with a suitable selection of 8tmax. results in the best performance. (D) requires more training iterations than ATE(F) as has to spend time exploring all the candidate arms but it is able to catch up to the performance of ATe(F) which uses a carefully selected value of 8m In addition, Ayg is significantly faster than AsTD in terms of computation time (see Table A3 in the Appendix). The speed-up for ATE comes from three sources (i) using temporally actions leads to fewer decision points, (ii) the
8

---

Caripole
Cartpole (Dense)
Half Cheetah
Ant
200
1000
500-
Score
1.0
200
750
0
1800
1.0
Traning Morcatona
Truining Merattnn
Training Incratioss
Training Rerations
Cartpole
Cartpale (Dense)
Half Cheetah
Ant
200
200-
1000
1000
Docision Paints
Dociatn Points
Decitin Points
1.00
200
500
20
150
150
Berratsone
Training tterittonss
Tratning Bemiitons
Training Merratooee
Hopper
Pusher
Reacher
500
-50
Scare
Soore
500
-100
0
100 200 300 460 500 600
100
200
300
400
100
200
300
400
100
200
300
400
500
Training Recations
Troining mecutions
Traininn Morations
Training Ieraatonn
Hopper
Wi
Decsunn Phents
Docision Points
Derttton Potns
1:00
200
500
50
100 200
300
400
300
600
100
200
300
400
100
200
500
400
300
2000
340
4.00
500
Traiting IerRItooS
Traninn Beratons
Training Recotions
Training Beraabons
OursiD)
Ours(F)
Standard
Figure Mean and standard deviation of the running averages (window size=10) of scores and number of decisions taken by the agents in classical control and MuJoCo domains across 5 different seeds. Ours(D) selects 8dmax automatically using the proposed multi-armed bandit framework while Ours(F) uses fixed 8l for every episode. Using temporally-extente actions results in better performance in many environments while requiring fewer decision points.
reduction in search space, due to temporally-extende actions further reduces planning time and, (iii) decrease in decision points results in a smaller sized dataset of past transitions that the model needs to learn from, thus reducing the model training time.
In Cartpole, an episode lasts for 200 primitive actions, unless the pole falls which terminates the episode. By default, the agent gets a reward of each timestep. We find that this reward formulation leads to highly unstable learning using the standard framework. The spike in number of decision points for ASTD (DSID 30) is because the agent learns to keep the pole from falling for the entire duration of the episode, but it soons crashes and the agent is unable to recover. Following Wang et al. [2019], we also experiment with more informative reward and find that stabilizes the learning of the standard framework. Both variants of ATE (TT 5) work well with both the reward functions.
An episode in Ant, Half Cheetah, Hopper and Walker lasts for 1000 primitive actions. By default, the reward per timestep in these environments is not bounded. As our bandit framework uses UCB heuristic for selecting ōt we modify the reward per timestep be between and Details of the reward functions are in Section D. In Half Cheetah, ATE DIE 15) performs similarly to ASTD DSID = 90) but is 8x faster. In Ant, Hopper and Walker, AyE significantly outperforms ASID- ATE (F) results in the best performance among agents showing the importance of properly selecting 5mmax- (Dr explores all the possible candidate arms before fixating on one, as can be seen from the plots of number of decision points. The value of 5lmax eventually chosen by ATE(D) is close to ATE (F). Note that (D) does not have the flexibility choose a value of 8lmax arbitrarily but has to choose from fixed list of exponentially spaced öt candidates.

---

In Pusher and Reacher, the performance of ATE is again similar to ASTD. But in this case, the number of decision points for ATE is also similar to that of ASTD, indicating that ATE identifies that a small decision timescale works better here. This illustrates that in cases when long duration is not suitable, ATE'S 's performance is still competitive with the standard solution.
How do temporally-extended actions compare to action repeats? Action repeats are special case of cemporally-extendee actions where all actions have the same fixed duration. While using temporally-extended actions provides flexibility to an agent in terms of choice of duration for each cemporally-etteme action, it does add slight overhead when compared to action repeats. order to understand whether the agent makes use of the provided flexibility or simply falls back to action repeats, we examine the distribution of action duration chosen by Aræ(F) in Ant, Half Cheetah, Hopper, and Walker during the course of one episode. Figure 4 shows the histogram of discretized action durations as action repeats chosen by the agent using the model at the end of training.
While in Ant and Half Cheetah, ATE (F) mostly chooses actions with similar öt, in Hopper and Walker, the values of öt vary significantly during the episode. This indicates that in environments like Ant and Half Cheetah, the agent can obtain a similar performance by using a fixed ót throughout the episode. However, in environments like Hopper and Walker, the agent benefits from flexibly adapting action duration in each step of execution.
Hopper (13, 28.83, 58)
Ant (4, 6.72, 7)
Holf Chestoh (5, 6.94 7)
Wallee (22, 28.66,
100
120
75-
Eroqueocy
20
Frequeocy
Prequerayy
20
60 Action repeats
Action rvpeata
Action sspoats
Action repoats
Figure 4: Histogram of action duration in terms of primitive action repeats from the one episode at the end of training for Ant, Half Cheetah, Hopper and Walker. In contrast using a fixed frame-skip. using temporally-eee exxeended actions allows the agent to take actions of varying durations. Numbers in the title indicate the minimum, average and maximum of action repeats taken.
Conclusion and Future Work
Using the standard framework of planning with primitive actions provides more granular control but can be computationally expensive. When TThh is small, agent requires large planning horizon which increases the search space as well as the number of variables it needs to optimize. We propose to temporally-extender actions where the planner treats the action duration as an additional optimization variable. This decreases the complexity of the search by restricting the search space and reducing the planning horizon, which in turn results in smaller number of variables for the planner to optimize. Further, we show that, with a suitable of learning temporally- extended dynamics models using MBRL be faster while outperforming the standard framework in many environments. Finally, rather than setting 5lmax as a hyperparameter, MAB framework can be used dynamically select it. This is slower converge but provides the same performance benefits as carefully chosen value of ótmax. Our proposed approach does have several potential limitations. First, using
actions does not guarantee improvement in every situation since the generated trajectories are not as flexible as those due to primitive actions. Second, the proposed MAB formulation requires fixed DIE for every bandit arm. Dynamic selection of both 8t.max and DrE together is left as future work.
Acknowledgements
We would like to thank the reviewers for their time and valueable feedback. This work was partly supported by NSF under grant 2246261. Some of the experiments in this paper were run on the Big Red computing system Indiana University, supported in part by Lilly Endowment, Inc., through its support for the Indiana University Pervasive Technology Institute.
10

---

References
Peter Auer, Nicolo problem. Machine learning, 47(2):235-255 256, 2002.
and Paul Fischer. Finite-time analysis of the multiarmed bandit
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Mare Bellemare, Joel Veness, and Michael Bowling. Investigating contingency awareness using atari 2600 games. Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 864 804-871,22 2012.
Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi -al---ammedadddtd problem with non- stationary rewards. Advances in neural information processing systems, 27, 2014.
André Bicdenkapp. Raghu Rajan, Frank Hutter, and Marius Lindauer. Temporl: Learning when to act. In International Conference on Machine Learning, pages 914 924.. PMLR, 2021.
Zdravko I Botev, Dirk P Kroese, Reuven Y Rubinstein, and Pierre L'Ecuyer. The cross-entropy method for optimization. In Handbook of statistics, volume 31, pages 35-59, Elsevier, 2013.
Alex Braylan, Mark Hollenbeck, Elliot Meyerson, and Risto Mikkkulainen. Frame skip is powerful parameter for learning to play atari. In Workshops the twenty-ninth AAAI conference on artificial intelligence, 2015.
Palash Chatterjee, Ashutosh Chapagain, Weizhe Chen, and Roni Khardon. Disprod: differentiable symbolic propagation of distributions for planning. In Proceedings the Thirtt-Second Interna- tional Joint Conference on Artificial Intelligence, pages 5324-5332, 2023.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
Ishan P Durugkar, Clemens Rosenbaum, Stefan Dernbach, and Sridhar Mahadevan. Deep reinforce- ment learning with macro-actions. arXiv preprint arXiv:1606. 04615, 2016.
Lev Finkelstein and Shaul Markovitch. A selective macro-learning algorithm and its application to the пxn sliding-tile puzzle. Journal of Artificial Intelligence Research, 8:223-263, 1998.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870 PMLR, 2018.
Milos Hauskrecht, Nicolas Meuleau, Leslie Pack Kaelbling, Thomas L Dean, and Craig Boutilier. Hierarchical solution of markov decision processes using macro-actions. arXiv preprint arXiv: 2013.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.
Marin Kobilarov. Cross-entir Pross-entropy motion planning. The International Journal of Robotics Research, 31(7):855-88) 871, 2012.
Ariel Kwiatkowski, Mark Towers, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: A standard interface for reinforcement learning environments. 2024. URL httt:::/arxxi..orr//a//////. 17032.
Aravind Lakshminarayanan, Sahil Sharma, and Balaraman Ravindran. Dynamic action repetition for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.

---

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel sundit-based approach to hyperparamettr optimization. Journal of Machine Learning Research, 18(185):1-522 2018
Shiyin Lu, Yu-Hang Zhou, Jing-Cheng Shi, Wenya Zhu, Qingtao Yu, Qing-Guo Chen, Qing Da, and Lijun Zhang. Non-stationary continuum armeed bandits for online hyperparameter optimization. In Proceedings of the fifteenth ACM international conference on web search and data mining, pages 618-627, 2022.
Marlos C Machado, Mare Bellemare, and Michael Bowling, laplacian framework for option discovery in reinforcement learning, In International Conference on Machine Learning, pages 2295-2304. PMLR, 2017a.
Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv:1710.11089, 2017b.
Mausam and Daniel S. Weld. Planning with durative actions in stochastic domains. Journal of Artificial Intelligence Research, 31:33-82, 82, 2008.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-353 533, 2015.
Tianwei Ni and Eric Jang. Continuous control on time. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World, 2022.
Rahul Ramesh, Manan Tomar, and Balaraman Ravindran. Successor options: An option discovery framework for reinforcement learning. arXiv preprint arXiv:1905.0573: 2019.
Sahil Sharma, Aravind Srinivas, and Balaraman Ravindran. Learning repeat: Fine grained action repetition for deep reinforcement learning. arXiv preprint arXiv: 2017.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1 112(1-2):811-111 1999.
Ayal Taitler, Ron Alford, Joan Espasa, Gregor Behnke, Daniel Fišer, Michael Gimelfarb, Florian Pommerening, Scott Sanner, Enrico Scala, Dominik Schreiber, et al. The 2023 international planning competition, 2024.
Cem Tekin and Mingyan Liu. Online learning of rested and restless bandits. IEEE Transactions Information Theory, 5558-5555-, 2012.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEFF/RSJ international conference on intelligent robots and systems, pages 5026-5033 IEEE, 2012.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Sigi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi: http:://dii.rr//// 10.
2116:.simpa.2220.1000. 100022.
Dong Wang and Giovanni Beltrame. Deployable reinforcement learning with variable control rate. arXiv preprini arXiv:2401.090 009286, 2024.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model- based reinforcement learning, arXiv preprint arXiv:1907.02057. 2019.
Peter Whittle. Restless bandits: Activity allocation in changing world. Journal of applied probability, 25(A):287-298, 1988.
Grady Williams, Andrew Aldrich, and Evangelos A Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2): 344-357, 2017.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, claim that using lemporally-extendee actions adds addi- tional structure to the search and can thus improve planning and learning using MBRL. In the paper, showcase this empirically by performing extensive experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the conclusion, we acknowledge that our method has several potential limitations. The proposed framework does not guarantee improvement in every domain as also noted in MBRL experiments with Reacher and Pusher, Secondly, the meta algorithm for selecting max assumes that planning horizon is fixed. more robust solution will select more max and planning horizon simultaneously.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattio, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
13

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There theoretical results in the paper.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The environment specifications along with the necessary hyperparameerrrs are described in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
14

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code can be found at https: // github. com/ pecey/
MBRL- wi th- TEA/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
wwwpammmissoooooicyyy.oooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //iiss... wwwpcccpppc/..ee..c.ccccce.cetle muublicgguides..../m htteg/gguudeesoeeuuubbiiiiiiiiii SCodeSubmissiond sei.bbiiissiooicocccco.oo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We provide the relevant experimental configuration in Section E
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
We report the mean and the standard deviation across 5 seeds for each experiment.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details regarding the computational resources in Section F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: tttttps//neurrri /nwuriisu...p// cc cccppblic///ecccccccccccccccclct cettiiisseeeeeeeeeneeenene lins?
Answer: [Yes]
Justification:
Our research conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper presents foundational research that is not tied to particular applica- tion. Broader impact concerns are hence generic.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We are not releasing any data or models part of this research.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the code required for this research was written by using standard open source frameworks.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We are not releasing any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper does not use LLMs any form.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp///errpp/////ee cc/Confferecces/2255 /LLM) for what should or should be described.
19

---

Appendix
Table of Contents
List of Symbols
B Connection to Ni and Jang [2022]
21
Iterative Primitive Dynamics Function
22
Environment Overview
22
MuloCo environments
22
E Experimental Details
23
Computational Resources
24
Additional Planning Experiments
25
G.I Experiments with IPC Mountain Car
25
G.2 Experiments with U-shaped Maps in Dubins Car
25
H Additional MBRL results
27
How do the action repeats due to ATE vary across training iterations?
27
I Ablations
27
L.1 Single model Separate models
27
I.2 Which is better approximation for F Fix Fip?
28
20

---

A List of Symbols
8;
Base timescale of the system.
Decision timescale the environment Frameskip x òz. Action duration chosen by the planner.
8t 8tminn max
Range for the action duration.
f
One-step dynamics function. S A S
FiP
Iterative primitive transition function.
F
Temporally extended dynamics function. S (A +
Fip
Indirectly approximates F. Learns a one-step model approximate and then iterates over it.
PTR.
Directly approximates F. Learns model to predict the next state due to remporally-extendeer action.
ASTD
TSoo variant of the CEM agent from PETS [Chua et al., 2018). Proposed agent that used fixed and manually chosen value of 81 Proposed agent that used MAB framework to dynamically select dtxax before every episode.
(F)(F)
ATe(D)
Dsm
Planning horizon used by the standard agent. Planning horizon used by the proposed agent.
DTE
B Connection to Ni and Jang [2022]
Ni and Jang [2022] propose learn a policy that outputs the action variables as well as the time scale. They have similar looking objective. In this section, compare objective to theirs. First, we repeat Equation (1) which computes the returns of trajectory t using the proposed framework:
D(r)
6k
J(T)
Recett))
A similar objective is used by Ni and Jang [2022]. However, it is important to note that they control the timescale while keep the timescale fixed and control the action duration. Let dş E [ómins 8 be the timescale associated with the action at decision step k. For simplicity, they assume that the timescale is in integers (in physical unit of seconds), which means that 6ę is also the number of execution steps associated with the action. Using this, their objective reduces to
L(r) J(T)
Σ
1 eck RecRR(seat11 01)0
k=1
where R(secs+1. 0g )o, is a linear approximation of the reward function. So, the objective used by Ni and Jang [2022] can be viewed approximation of Equation (1) where the reward function due to a temporally-extended action has been replaced by a linear approximation.
Algorithm A1 FiP Iterative primitive dynamics function
Require: primitive dynamies function (f), current state (sk),
temporally-extencll action (ax), duration of action (8t,)
1: repeats /blz/ag") 1
2: for = 1 to repeats do 3:
5k: = f(kk, ak deml
Control
4: end for
5: Sk41 f(kk, ag, öt mod 8)))
Control 8
6: refurn Sk+1
21

---

C Iterative Primitive Dynamics Function
In most cases we have access to the primitive dynamics function (f). A trivial way of making f work with temporally-extended actions is to wrap it in loop. For the computation be exact, should be explicitly dependent on time. If it is implicit, performing the computation at line of Algorithm A1 will not be possible and the algorithm computes the number of action-repeats instead.
The reward is simply aggregated in the loop and is not shown explicitly here. Note that for given environment, f is fixed, while 8k, ag and 61g are dependent the timestep.
D Environment Overview
In this section, we provide overview of the environments used in our experiments. Table A1 lists the dimensionality of the observation and the action space of the environments, the reward functions along with the episode length (which is equivalent to the maximum number of primitive steps allowed).
Domain
n$
IA
Episode steps
Reward Function
Dubins Car
300
100
Limuse from D...c...
L =
Catipole
1
200
Caripole (Dense)
200
cos(O,) - 0.001.²
Mountain Car
B
0.1a
IPC Mountain Car
2
500
ayy 100 0.1
Reacher
17
7
150
- disttfingger obj) of
Pusher
23
7
150
-0.5 x dist disttfinger,ol obj) dist(object,, goal) 0.1a,
sigmoid( ) di. e.e
Hall Cheetah
18
1000
sigmoid(1f, (1f,) ) if X 10 else
Hopper
(sigmoid) ai, if
1) healthy
Walker
17
1000
1/2((sigmoi((;;) if 2, s clse 1) + healthy reward)
Table A1: State and action space, maximum episode lengths and reward functions for different environments.
Details for Dubins Car environment: : We use the Dubins Car environment of Chatterjee et al. [2023]. The state space comprises of the and y co-ordinates, and the orientation (0) of the car, along with the current linear (v) and angular velocity (w) of the car and the action space is the change in linear velocity (Ar) and angular velocity (Aw). An episode terminates if the agent reaches the goal or if it takes 300 primitive steps.
D.1 MuJoCo environments
Controlling oسn Each environment defined in MuJoCo has its own value of ô; which is defined in the corresponding XML file along with the integrator to be used. Further, each environment has a predefined frame-skip skip which is defined in the constructor of the corresponding class. The combination of these two control 6h in MuJoCo. In order to operate at the base o,, we simply set the frame-skip value to 1.
The default reward functions in MuJoCo are not bounded. For the MAB frame- work, we require bounded reward function. The problem is amplifed in Ant, Half Cheetah, Hopper and Walker, where the rewards over an episode can be very large. Therefore, for these environments, adapt the bounded reward functions from DeepMind Control Suite [Tunyasuvunakool al., 2020], so that the agents always get reward between 0 and 1, with the specifics depending on the environment.
Reward functions
Maximum episode lengths: MuJoCo environments have a predefined value of frame-skip. The default episode length is with respect to this frame-skip. For example, Half Cheetah has a frame-skip of 5 and a episode length of 1000. An equivalent episode Half Cheetah with frame-skip of should be of length 5000. However, we limit the episode to 1000 otherwise ASTD would have taken too long train. Even with episode lengths of 1000, ASTD requires more than 40 hours to train (Table A3).
22

---

(a)
(b)
Figure A1: (a) An example of AnE solving the cave-mini map (?1 = 0.99, ?2 = 1.0, 8t max 20).
(b) An instance of the IPC Multi-hill Mountain Car.
E Experimental Details
Among the set of hyperparameters in our framework, the two crucial ones are the range of action duration, which is specified by 8tmii and ötmax. and the planning horizon. Rather than controlling both these values, 8tmin is always set to 8t and modifying òf controls the range.
For the environments Cartpole and Mountain Car, we use values of planning horizon from prior work for the standard framework, and adjust the depth for Are by exploring related values. We could not do this for MuJoCo-based environments since we set the frameskip parameter to 1. Rather we searched over some potential values for the planning horizon for the standard framework, and 8tax and planning horizon for our framework and chose the configuration with the best performance. The other hyperparameters related to online training have been borrowed from Chua et al. [2018].
Details for Cartpole, Mountain Car and MuJoCo:
Domain
Range of action duration
Planning horizon
Learning Rate
Standard
Ours
Standard
Ours
Duhins Car (hashaped map]
0.2
[0.2-20]
1000
75
Dubins Car [cave-mini map]
0,2
[0.2-2]
120
50
Caripole
[1-10]
le.3
Mountain Car
[1-100]
100
10
IPC Moontain Car
[1-125]
175
Reacher
0.01
10.01, 0.21
25
le-3
Pusher
0.01
[0.01. 0.2]
25
le:3
Half Cheetah
0.01
[0.01] 0.07]
lc-3
Ant
0.01
[0.01, 0.07]
70
15
3e-4
Hopper
[0.002, 125]
Walker
0,002
[0.002, 0.0625]
70
15
3e-4
Table A2: Hyper- parram-prraeters for different environments. Range of action duration is for the fixed variant of the algorithm.
Details for Dubins Car environment: The default timescale for the environment is 0.2 while òtmax for our formulation is 20. Setting 5t.aa to such large value allows us to have the same value across maps and just allow more time for optimization. The planning horizon needs be tuned for different maps. Table A2 contains the planning horizon for the maps used in the experiments. Note that for experiment with the cave-mini map, reduce 8tmax to prevent the agent from completing the map in a few decision steps.
Model learning: For MBRL, we use the same model architecture as Chua et al. [2018]. We learn an ensemble of models, where each model is a fully connected neural network. For Ant, Half Cheetah and Hopper, the model has 4 hidden layers while for all other environments, it has 3 hidden layers. Each hidden layer has 200 neurons. The learning rates for each environment are given in Table A2.
Planning with an ensemble of models: We use the PETS framework from Chua et al. [2018]. They introduce Trajectory Sampling (TS) which uses CEM the planner. They maintain sequence of sampling distributions from which N action sequences are generated, and for every action sequence, they create P particles. Each particle is propagated using a particular member of the ensemble. For propagating particle p. TSI uniformly resamples ensemble member at each time step, while TSoo samples an ensemble member before an episode and uses that to propagate particle p for the entire duration of the episode. The mean reward is computed for each action sequence. Then, the mean and
23

---

Algorithm A2 Planning using TS with an ensemble of dynamics model.
Require: current state (s), dynamics model ensemble(s = (fi...., fx]) initial action distribution (Ma, var.)
planning horizon (D),
1: for == 1 to number of updates do
2: Sample N action sequences {ar" ag ........,....................... a using May On-
3: for = to N do Initialize P particles {ai ( P.P 5: for = to P do
4:
22
"D P.
6:
//propagate each particle using one model from ensemble j p mod K
7:
8:
Propagate particle using n.P = fs( R.P di " and compute aggregate reward.
9: end for
10: Compute mean reward for action sequence using E. Ep r(o," a/
11: end for
Choose the top & action sequences and recompute Pax var,
13:
end for
Execute first action from the optimal action sequence.
the variance of the top l action sequences gives the updated sampling distribution. Formally,
Σ
No k
(5)
aEK
1 h aEK
Σ (a - (4a))
var,
(6)
where K. is the set of top action sequences. This process is repeated for fixed number of steps. For our experiments, we learn an ensemble of feed-forward networks and use the TSoo variant with P = 20 for planning. In Algorithm A2, present the pscudocode for using TSoo with an ensemble of dynamics models.
F Computational Resources
Each MBRL experiment was performed on single node with a single GPU (using mix of V100 and A100), single CPU (AMD EPYC 7742) and 64GB of RAM. The time required for training for different algorithms are in Table A3. We note that although the experiments were done on two different types of GPUs, the running times for both of them are comparable.
Ens
Standard
Ours (F)
Ours (D)
Cartpole
0.65
0.4
0.5
Half Chectah
5.5
Ant
7
Hopper
6.5
Reacher
1.6
1.5
Pusher
2.6
2
1.5
Walker
5.8
Table A3: Approximate time hours) required to finish training for each environment.
In Reacher and Pusher, the rollout length is mostly same for the standard algorithm and our proposed algorithm, but the planning horizon for the standard algorithm is 25 while for our proposed algorithm 15, resulting shorter training times.
In Ant, Half Cheetah, Hopper and Walker ATE is significantly faster than ASTD The dynamic variant of our proposed algorithm takes slightly longer than the fixed variant as the rollout length of the dynamic variant is longer average than the rollout length of the fixed variant.

---

G Additional Planning Experiments
G.1
Experiments with IPC Mountain Car
How does the performance change if the increase the planning horizon? We vary the planning horizon of the agents and compare the performances across instances in Table A4. For the easier problems, a small planning horizon is sufficient, but for the difficult instances, deeper search is required. The performance of the planner remains relatively stable as the planning horizon increases.
D
Instance
Instance 2
Instance 3
Instance 4
Instance 5
60
37.49 + 51.36
.0.0
-0.0 0:0:::0
-0.0 1.00::0
-0.0 0.0
80
74.54 + 41.7
-0.0 0.0
-0.0 +:0=:00 0.0
-0.0 0.0
-0.0 0.0
-00
125
90.92 1: 0.22
0.0 0.0
.0.0 1:00:.00
0000 1.00
-0.0 0.0
91.07 +:
1:0:000
0.0
4.0
175
91.12 + 0.3
-0,0
-0,0
+0,0
-0.0
200
91.11 1: 0.27
0.0
0.00
0.0 0.0
-00 0.0
225
91.2 0.32
-0.0
+0.0
+.00
-00
250
91.01
+ 0.12
-0.0
-0.0 0.0
-0.0 +.0
-00 -00 0.0
300
90.91 1: 0.1
0.0 0.0
0.0 0.0
0.0 0.0
0.0 1 0.0
.0.01
0.01
.0.0
.0.0
...0
+00
5
73.52 + 43.45
-0.0 0.0
-0.0 1:0+00 0.0
-0.0 + 0.0
-00
10
71.43 1: 46.05
89.79 1 1.28
87.27 1: 1.86
68.11 1: 42.49
85.02 0.26
92.41 +:
90.11 +
+:
15
92.33
+ 2.0
90.52 - 0.6
87.91 + 0.8
87.51 + 0.98
85.19 = 1.15
20
92.52
+: 1.46
90.12 1 1.36
87.74 1: 1.99
87.08 + 2.57
86.43 0.31
92.39
1.85
90.54
88.39 + 1.36
87.71 + 1.61
87.06 0.46
30
72.07
+ 45.41
91.08 + 0.37
88.94 + 0.49
68.0 = 45.24
86.08 1.81
57.79
40
71.21
1: 48.64
90.12 1: 1.42
89.75 1 2.19
87.38 1: 2.38
67.86 45.2
Table A4: Performance of ASTD and ATE in various instances of IPC Multi-hill Mountain Car the planning horizon is increased.
G.2 Experiments with U-shaped Maps in Dubins Car
We experiment with varying depths using u-shaped maps in the Dubins Car environment. The problem is not trivial as the car faces the obstacle and it needs to first turn around and then find path the goal, which gives the agent a reward 100. Every collision with an obstacle gets a penalty of 10. An episode terminates when the agent reaches the goal or when the agent takes 300 primitive steps.
To be successful in this setting, observe that ASID requires planning horizon of 1000 and ArE requires a planning horizon of 75. Both the agents use 10,000 samples and 50 optimization steps for each decision. ATE succeeds but not in all runs. We look at the failure cases for ATE and have an interesting observation. For planning horizon of and 100, the failure is due to the fact that the agent runs out of primitive actions, while for planning horizon 50, the failing scenario corresponds to the agent not being able to find a path out of the obstacles region. This means that if we allow the episodes results are in Table A5 and the failure cases for ATE are shown in Figure A2.
for longer, the former failure be mitigated but the latter cannot. The detailed
D
Rewards
Decision Steps
Decision time J
Time for episode
P(snocess) 1
250
99.55 1: 0.0
300.0 0.0
0.13
413.46 1: 4.16
500
-9.95 + 0.0
300.0 0.0
0.29 0.0
806.97 + 8.82
0
Asıo
0.42
1000
100.0 + 0.0
90.8 + 20.17
0.56 0.02
480.38 + 102.06
.97.53 & 137.16
4.0 1: 0.0
2.31 0.09
14.1 1. 0.43
0
16.32 + 187.12
42 - 0.45
3.66 0.1
20.03 = 1.59
0.8
ATE
0.24
100
80.0 4: 44.72
4.4 4: 1.14
6.57 0.21
34.56 7.79
0.8
Table A5: Performance of ASTD and ATE on the u-shaped map in Dubins Car as the planning horizon varied. Both agents use 10,000 samples and 50 optimization steps.
25

---

#samples
Rewards 1
Decision Steps
Decision time 4.
Time for un episode 4
P(success) 1
250
9.95 +: 0.0
300.0
0.66
580.85 1: 2.91 970.11 = 378.22
0
500
12.04 = 49.17
253.2 + 100.18
1.38 0.01
0.2
AsID
10000
0
1000
0
100
919.43 238.36
300.0
0.46 0.03
1567.29 25.31
0
AsTD
1000
1000
-15.92 + 8.9
300.0 + 0.0
0.73 0.0
1663.75 + 8.56
0
10000
0
-179.62
145.46
U.D
2.31 0.09
14,05 +:
0
18.31
182.67
3.6 1.
1: 0.12
18.72 1: 2.41
0.8
AT
10000
100.0
4.6
5.36 0.16
30.01 +: 4.56
1.0
44.72
4.89
6.87 0.12
35.74 =: 6.44
0.8
Table A6: Performance of ASTD and ATE on the u-shaped map in Dubins Car when the action space augmented with 100 dummy actions. Both agents use 50 optimization steps. Missing values indicate that the particular configuration was not feasible due to large memory requirements.
(a) DTE 50, |A|
(b) DTE 75, ||| 2
(c) DTE 100, |A|
(d) DIE = 50, |A| 102
(e) DTE 100, |.|| 102
Figure A2: Failure cases for ATE in the u-shaped map. When |A| 2, for DTE 75 and DTF. 100, the agent identifies a path around the obstacles but runs out of the maximum number of primitive steps. On increasing the action space, similar fallure occurs for DTE 100. The failures due to DTE 50 happen because the agent does not find a path around the obstacles.
Next, we make the problem difficult by augmenting the action space with 100 dummy action variables. As the agents unaware that these variables don't contribute to the reward the dynamics, they still need to search over these variables. We set up the experiment similar to the previous one by keeping the number of samples 10000 and varying the planning horizon. For ASTD. almost all the runs fail either because it does not find path around the obstacle, or crashes due to huge memory requirements. In order reduce the memory requirements, we perform second experiment, fixing the planning horizon to 1000 and varying the number of samples. Reducing the number of samples fixes the issue of memory requirements, but does not help the agent identify a path. In contrast, the performance of ArE is mostly unaffected by this addition of dummy action variables. The detailed results are in Table A6. ATE uses the same configuration as the earlier experiment and achieves similar performance. As observed earlier, the failure case of ATE for planning horizon of 50 is due to the fact that cannot find a path around the obstacles, while the failure case for planning horizon of 100 is because it runs out of primitive steps.
26

---

Additional MBRL results
H.1
How do the action repeats due to ATE vary across training iterations?
Cartpole
Cartpole (Dense)
(orban sepuet
100
150
200
50
100
150
2000
100
200
500
400
500
100
200
300
400
Truining Hersitioss
TTwining meredons
Triining iterations
Triining Ieeratooe
Hopper
Pusher
Reacher
Walker
Truining thrraaat
TTrining Blcrstions
theratione
OurstD)
Ours(!))
Standard
Figure A3: Mean and standard deviation of the running averages (window size=10) of log of action repeats. Values due ATH(D (D) converge in the neighborhood of values due ATE
While the histogram in Figure gives us good sense of the distribution of discretized action duration (8t/s;") chosen by ATE in one episode after the training has been completed, it does not capture the trend across time. As the average action repeat for (DE can have a large range, in Figure A3, we works with primitive
plot the log of average action repeat instead. ASTD has an action repeat of actions. For ATE (F), the average action repeat is mostly stable. In Cartpole, Reacher and Pusher, the agent has larger action repeats (or takes large values of ōt) in the beginning before realizing that smaller action repeats work better. The more interesting plots are those of AnE (D) as it has to identify the correct range of öt. From the plots, we see that the average action repeats decreases gradually before stabilizing around the average action repeat used by (F) Note that larger values of average action repeats does not necessarily mean smaller number of decision points. (D)) has to choose mux candidates from a list of exponentially spaced candidates and large ót, selection can easily skew the average action repeat.
Ablations
L.1 Single model vs. Separate models
In non-stationary bandits, the reward distribution of the arms changes with time. If the reward distribution of an arm changes only when is pulled, is called rested bandit, while if the reward distribution of an arm changes irrespective of whether that particular arm was pulled or not, it is called restless bandit [Whittle, 1988, Tekin and Liu, 2012].
In the MBRL setup, the performance of the agent depends on the quality of the learned dynamics model. If we learn single dynamies model be shared across all arms, then the underlying reward distribution for an arm can keep on changing even if the arm is never pulled. This is the setting of restless bandit. On the other hand, learning a separate dynamics model for each arm results rested bandit setting as it ensures that the underlying reward distribution for an arm only changes when that arm pulled and the corresponding dynamics gets updated.
Hence, choosing whether to use a single dynamics model which is shared between all the bandit arms using separate dynamics model for each bandit is important consideration. Both the modeling choices have their advantages and disadvantages.
While using single dynamics makes the most use of the available data, there is a data imbalance issue. Larger values of St. will result in fewer experiences being captured. Even if we uniformly sample the arms, we will have fewer examples due candidates with large values of 8t than
27

---

will have for candidates with small values of max As all of the data is used to train a single model, the model is likely overfit on examples from candidates with small values of mt On the other hand, using separate dynamics model for each bandit makes the model more focused, but we do not make full use of the available data.
As shown in Figure A4, we find that using the UCB heuristic, the agent performs better if it learns a separate dynamics model for each bandit arm.
I.2 Which is a better approximation for F FIR or FIP?
While we proposed to use FIE to approximate the temporally-exterr dynamics function F, one could also consider using Fip which approximates the one-step dynamics function f and then iterates over it. While we know that FTE leads to better time complexity than fip. it is interesting to explore how it compares in terms of prediction quality.
To answer this question, we perform two experiments with random action sequences and compare the prediction quality of FIE and Fip. In both the experiments, we sample random action sequences of length 500 and compute the -ean--quared error (MSE) between the actual states observed upon taking the action sequences in the environment and the states as predicted by the learned model. The difference between the two experiments lies in the type of action sequences being sampled. For the first experiment, we randomly sample one-step action sequences which implicitly correspond to randomly sample cemporally-extendee action
duration 8lmi while for the second experiment, sequences with action duration sampled from (6lmin max)) Each experiment was repeated 10 times in order get a better estimate and the mean results are shown in Table A7.
Random one step sequences
Random temporally extended sequences
Env
68min 6mmax
Pr::
Fir
P'
Fir
Pir without outliers
Catpplle
0.0107
2.517 10-7
0.0184
0.065
0.065
Ant
0.01
0.07
0.126
0,0149
0.298
4.525
0.60 10
Half Cheetah
0.01
0.07
0.30
0.04
1.752
2.526
2.526
Table A7: Comparison between the prediction error (in terms of MSE) of FIE: and Fip on random action sequences.
In Ant, when predicting using temporally-extended action sequences, in out of 10 experiments, the MSE due to Fir was extremely high. We hypothesize that the states observed during these experiments were probably outside the range of examples seen during the training of the one-step model that is used by Fi.. Fir without outliers reports the MSE after removing these 4 experiments.
As can be observed, for one-step sequences Pip has lower MSE but for temporally-extender sequences, PTE is clearly the better choice. This is not surprising since using temporally-extende action sequences with Fip requires iterating over the one-step model multiple times which can lead to compounding errors. Thus, FInE is better choice both in terms of time complexity and prediction quality when using temporally-extencee action sequences.
Ant
Half Cheetah
500
800
Score
Score
600
250
400
100
200
300
400
500
100
200
350
400
500
Training iterations
Training sterations
Single
Separate
Figure A4: While learning a single dynamics model to be shared across all 8t candidates is sufficient for Half Cheetah, we need to learn separate dynamics models for each otaax candidate in Ant.
28


Reviews:
[{"review_id": "W2J8CjrasS", "reviewer": "Reviewer_chCv", "summary": "The paper introduces a novel approach for planning (and model-based reinforcement learning) that treats action duration as a decision variable. Instead of only issuing a primitive action at every simulator time-step, the agent samples both an action and how long to repeat it. To support this, the authors learn a temporally-extended dynamics model that predicts state-reward transitions over variable time gaps. They then frame the choice of the maximum admissible duration as a multi-armed bandit, where each arm corresponds to an upper-bound $\\delta t_{max}$. The resulting “action-plus-duration” space is then embedded inside a cross-entropy method (CEM) planner from prior works. Experiments on several continuous-control benchmarks (classic control, MuJoCo, Dubins car, Mountain Car) compare this temporally-extended agent ($A_{TE}$) against a baseline that plans with single-step actions ($A_{STD}$). $A_{TE}$ often reaches similar or higher episodic returns while requiring far fewer decision points and achieving notable wall-clock speed-ups. Impressively, $A_{TE}$ is also able to solve tasks which are unfeasible for $A_{STD}$ given compute and memory constraints.", "questions": "Please refer to my weaknesses above. Mainly:\n\n- Define all functions and spaces. Provide explicit domain/range (e.g. in terms of the state and action spaces) for f, F, $F_{IP}$, $\\hat F_{TE}$. \n- Please comment on the point regarding the search space claims. If my interpretation is correct, please revise the claims in the paper accordingly.\n- Ablations for fairness: \n  - Match the total number of ensemble models across ASTD and ATE, or show that model count does not drive performance.\n  - Can the authors run ablations of these hyperparameters for $A_{TE}$: number of sequence samples, planning iterations, and model training batch-size (using smaller values than the ones used currently for the experiments)? Or provide a discussion that addresses my concern here.", "limitations": "An analysis or discussion of the failure cases of the proposed approach is not done. That is, settings where temporally extended actions are not desirable in the large majority of the state space. For example, the experiments are only in deterministic environments, even though the approach is presented to be more general. How well does the approach handle stochastic dynamics (e.g. the U-map navigation with increasing noise on the dynamics)? Also safety-critical environments, for example the cave-mini map modified such that hitting walls leads to failure.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n### Quality\n- The motivation is sound with a clear goal. The authors provide simple neat method to address the practical burden of long horizons by letting the planner adjust its temporal resolution online.\n- The paper delivers an end-to-end pipeline (bandit + ensemble world-model + CEM planner) and evaluates it in both pure-planning and learned-model settings.\n- Across many domains $A_{TE}$ achieves equal-or-better return with fewer decisions and significantly faster training. The authors also provide an impressive number of hypermeter ablations. The ones on the effect of the different scale discount factors was particularly interesting.\n\n### Clarity\n- The high-level intuition (trading planning depth for breadth via action durations) is easy to follow, and the algorithmic components are standard.\n- Visualisations of decision frequency and horizon length make the speed-up tangible.\n\n### Significance\n- A generic, task-agnostic way to obtain temporal abstractions could benefit robotics and MBRL practitioners.\n- The work reconnects classical durative-action planning ideas with modern learned models.\n\n### Originality\n- Optimising action duration inside a sampling-based planner and adapting the duration range via a bandit scheme appears novel.\n- Sparks theoretical questions about search-space complexity under temporal abstraction.\n\n# Weaknesses\n\n- The paper repeatedly frames itself as “MBRL” even though no policy is learned. Policy learning is a key key component of RL (see MBRL works like Dreamer for example). This work is closer to model-based planning with a learned simulator. This distinction needs to be explicit from the outset.\n\n- There's a general lack of preciseness. For example, heavy reuse of “=” for definitions and equalities (esp. Sec. 3) hurts readability; “:=” or “≜” is recommended. It is also unclear how the update in step 9 of Algorithm 1 is done (Algorithm A1 references also hyperlinks to Algorithm 1). Section 4 is imprecise in general. \n  - Primitive dynamics function (f) and temporally-extended dynamics function (F) are not precisely defined. What are their domains and ranges?  \n  - The iterative primitive dynamics function ($F_{IP}$) in line 151 is not defined nor explained in the main text despite being referred to throughout. \n  - It is unclear what the statement $\\hat F_{TE} \\approx F$ means. Does that sentence (line 159-160) mean the authors only use near-perfect models in the experiments? \n  - It is still unclear how the neural network is trained precisely. What is the input space, and what is the output space? What is the set of arms (is it $\\\\{\\delta t_{min}, 2\\delta t_{min}, ..., m\\delta t_{min}\\\\}$)? What is \"candidate i\" in line 180 (is it a bandit arm i)? What's the informal justification for equation 3?\n  - It seems like the approach also requires a suitable choice for m (for the number of arms and the upper-bound action duration arm), but this is not discussed.\n\n- The claim \"Using temporally-extended actions helps to reduce the space of possible trajectories\" is false (line 192). It strictly increases the search space, since the search space now also includes the selection of action durations (and this is not even including the selection of $\\delta t_{max}$). For $A_{TE}$, the search space is at least $\\bigcup\\limits_{k=1}^{m} (k|\\mathcal{A}|)^{H/k}$. Similarly, the claim \"$A_{TE}$ needs to optimize less variables than $A_{STD}$\" is false (line 197).\n\n- Hence one would expect $A_{TE}$ to take longer to plan due to the increase in the search space, but lead to shorter plans than $A_{STD}$ due to the temporally extended actions. However, this is not what is observed in the experiments. In fact, even when the authors demonstrate results in environments where temporally extended actions are not appropriate (Pusher and Reacher in Fig 3), $A_{TE}$ still performs similarly to $A_{STD}$ despite the much larger search space (and the fact that the dynamics model that is being learned is also more complex for $A_{TE}$). This suggests a potentially serious hyperparameter bias. That's $A_{TE}$ planning may be getting enough sequence samples, planning iterations, and model training batch-size to account for the increase in the search space size. Can the authors run ablations of these hyperparameters for $A_{TE}$?\n\n- The approach uses several models in it's ensemble (for example a separate model for each bandit arm). It is unclear if the authors used the same total number of models for the baselines. Additionally, the use of only 5 seeds hurts the significance of the demonstrated results.", "was_revised": true, "final_justification": "The authors really took some time and effort to properly address my concerns and provide a very detailed response. I am happy with their clarification and happy to increase my score to an accept accordingly.", "rebuttal": "Thank you for taking the time to review our work, for catching the hyperlink issue and all your thoughtful comments. Please find our responses to the questions below:\n\n- We are indeed performing model-based planning as pointed out. However, we think MBRL is an umbrella term that includes model-based planning and this terminology has been used in the literature in the past. For example, [1] and [2]. [3] benchmarks MBRL algorithms and includes solutions which do not need to learn a policy.\n\n- In Algorithm 1, we do not specify the details of the update as the algorithm is written for any shooting based planner. For our experiments, we use CEM. In order to update its action distribution, it looks at the top-k rewarding action sequences (after the for loop in lines 3-8), and then the mean and standard deviation of the action distribution is set to be the mean and standard deviation of the top-k action sequences.\n    \n- $f: S \\times \\mathcal{A} \\rightarrow \\Delta S$, $F = S \\times (\\mathcal{A} \\cup \\delta t) \\rightarrow \\Delta S$. The ranges and domains of $F_{IP}$ and $\\hat{F}_{TE}$ are the same as $F$.\n\n- We thank you for bringing the hyperlink issue to our notice. We had to move the explicit definition of $F_{IP}$ to the Appendix (Section A) due to lack of space, but it is defined concisely in the main text in lines 149-151.\n\n    We wanted the arms to be exponentially spaced rather than linear, and hence we set $m = \\log_2(T)$ where $T = \\text{max primitive steps}$. Here max primitive steps is set by the environment. For simplicity, assuming the environment timescale is 1, the set of arms are $\\{ T/2, T/4, T/8, \\dots, T/2^m \\}$.\n\n    Equation 3 is a UCB-like metric where we use exponential-moving average of the reward rather than simply the average reward. This is done as the underlying dynamics model is changing and the older rewards might not reflect the current model.\n\n- We do not fully understand the source of the equation being proposed, but we think the claim that we make is true. Firstly, to clarify, the claim made is for $A_{TE}$ with a fixed $\\delta t_{\\max}$ and does not include the MAB framework. (In the paper, we overload the term $A_{TE}$ to include the bandit framework, but the bandit framework is like a hyperparameter search, and each bandit technically corresponds to a single $A_{TE}$ agent.)\n    \n    Secondly, we add just a single action duration variable to the action space regardless of the number of action variables. So, the size of action space for $A_{STD}$ = $|\\mathcal{A}|$ while the size of action space for $A_{TE} = |\\mathcal{A}| + 1$. \n\n    For simplicity, let the environment timescale be 1 and let us consider $A_{STD}$ first. It has a fixed planning horizon $H$. So the computation graph that is created will have $H|\\mathcal{A}|$ variables.\n\n    Now let us consider $A_{TE}$ with a planning horizon of $H_1$ with a fixed $\\delta t_{\\max} = m$ (This is same as line 185 with $\\delta_t^{\\text{env}} = 1$). So the computation graph in this case will have $H_1(|\\mathcal{A}|+1)$ variables. $A_{TE}$ will select actions with duration in the range $(1, m)$. For most practical purposes, the action duration will be greater than 1. This means we can set $H_1$ to be less than $H$ in order to have a similar effect.  \n\n    Comparing $H_1$ and $H$ exactly is not possible without the actual action duration values. But let us assume that the $A_{TE}$ always chooses a fixed action duration $m_1 \\in (1, m)$. Then $A_{TE}$ with $H_1 = H/m_1$ should have a similar reach as $A_{STD}$ that has a planning horizon of $H$. \n    \n    We hope this clears out any confusion that might have been present regarding our claim.\n\n- In all the experiments that we perform, the same configuration (number of neural networks in the ensemble, number of samples, number of planning iterations, batch-size, learning rate etc) is used for $A_{STD}$ and $A_{TE}$. In the paper, we mention that both the agents use the same architecture (line 217), but we should have added that they use the same configuration as well. \n    \n    For $A_{STD}$ and $A_{TE}$, we learn an ensemble of 5 neural networks. For the bandit framework with $A_{TE}$, we learn an ensemble (of 5 neural networks) for each bandit arm. Before beginning an iteration (which comprises of one episode of data collection followed by learning), we select one bandit arm, and hence only one ensemble of neural networks is active.\n\nThank you again for all the valuable feedback and we hope that we were able to clarify all the questions. \n\nReferences:\n\n\n[1] Deep reinforcement learning in a handful of trials using probabilistic dynamics models (Chua et al.)\n\n\n[2] Integrating Sample-Based Planning and Model-Based Reinforcement Learning (Walsh et al.)\n\n\n[3] Benchmarking model-based reinforcement learning (Wang et al.)"}, {"review_id": "jJ4yzWgxqC", "reviewer": "Reviewer_XEPz", "summary": "This paper proposes a novel planning framework that dynamically adjusts the length of action repeats. The method automatically selects the optimal maximum repeat length with Multi-Armed Bandit (MAB) problem. Experimental results demonstrate that integrating dynamic action repetition into planning-based methods significantly reduces the number of decision points, thereby accelerating planning, while also achieving better performance compared to no action repetition baselines. MAB-based selection of maximal action repeat range also performs reasonably well in practice.", "questions": "**Q1. Training Iteration vs. Number of Steps**\n* What is the update-to-data (UTD) ratio used during training?\n\n**Q2. Training of $F_{\\text{TE}}$**\n* How is the transition estimator ($F_{\\text{TE}}$) trained? \n* Training such a model requires data that includes action durations. How is this data collected, and what supervision signal is used for training $F_{\\text{TE}}$?\n\n**Q3. $F_{\\text{TE}}$ vs. $F_{\\text{IP}}$**\n* Can the authors provide a comparison of the prediction error between $F_{\\text{TE}}$ (transition estimator with duration) and $F_{\\text{IP}}$ (instantaneous predictor)? \n* This would help clarify the impact of modeling action durations for model prediction accuracies.\n\n**Q4. Comparison with Fixed $\\delta t$**\n* In many cases, a fixed action repetition value is used (as also mentioned in the introduction). \n* How does the proposed dynamic approach compare quantitatively against models using a well-tuned fixed $\\delta t$? Is dynamic repetition consistently better, or are there trade-offs?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n**S1. Simplicity**\n* The proposed method is conceptually straightforward and easy to implement.\n\n**S2. Reduced Hyperparameter Tuning**\n* An important hyperparameter, the maximum action duration ($\\delta t_{\\text{max}}$), is automatically tuned through the MAB-based approach, reducing the need for manual tuning.\n* Moreover, the performance is robust to other parameters, such as discount factors (they indeed have a sweet-spot, but they do not have significant degradation on the overall performance ).\n\n**S3. Thorough Analysis**\n* The paper includes an in-depth analysis of various aspects of the proposed method, such as the effect of planning horizon, discount factor, and the use of model ensembles.\n\n&nbsp;\n\n**Weaknesses**\n\n**W1. Scalability to Complex Tasks**\n* While the proposed method shows promise, its scalability to more complex tasks remains uncertain (experimental results are given for simple tasks). \n* Specifically, in planning, two main sources of error—model prediction error and optimization error—can accumulate over long horizons. \n* The paper does not fully address how these challenges are mitigated in high-dimensional or long-horizon tasks.", "was_revised": true, "final_justification": "The response have addressed the questions I had:\n* The pseudocode effectively addressed most of the confusion I had regarding the training details.\n* New experimental result clearly shows that $F_{TE}$ works better in long horizon rollouts than $F_{IP}$.\n* The Dubins car example and the $\\delta t$ distribution of hopper being spread out shows the importance of having variable $\\delta t$.\n\nOverall, I think this paper shows the potential and importance of using \"variable length\" of rollouts in planning.", "rebuttal": "Thank you for taking the time to review our work and all of your thoughtful comments. Please find our responses to the questions below:\n\n- For the questions regarding update-to-data ratio and the training of $F_{TE}$, we begin by providing the pseudocode for the entire data collection and training loop (for fixed $\\delta t_{\\max}$}) explicitly . In the algorithm,  $n$ refers to the number of training iterations, $D$ is the dataset and $b$ is the batch size.\n  \n  ``` \n  for i = 1 to n\n    // Data collection\n    for t = 1 to end of episode\n        a_t = planner.choose_action(s_t)\n        s_{t+1}, r_{t} = env.step(a_t)\n        D = D.append((s_t, a_t, r_t, s_{t+1}))\n    // Training\n    num_of_batches = len(D)/b\n    for j = 1 to num_of_batches\n        sample a mini-batch of size b from D\n        compute the loss and the gradients due to the mini-batch\n        update the dynamics model using the gradient\n  ```\n\n  Note here that to be in sync with the notation in the main paper, $a_t$ includes the duration as well. More explicitly, $a_t$ is simply a concatenation of $|\\mathcal{A}|$ primitive action variables and one variable for the action duration. \n\n  So, the number of updates depends on the size of the dataset. \n\n  Regarding the training of $F_{TE}$, as implicit in the pseudo-code, the $(s_t,a_t,r_t,s_{t+1})$ in the dataset are exactly the ones selected by the planner, i.e., the model is trained on the temporally-extended actions that were used in the episode. We do not add any implied transitions to the dataset. \n\n- To compare the prediction error between $F_{TE}$ and $F_{IP}$, we ran two experiments by randomly initializing action sequences (of length 500), and computing the MSE between the next state means predicted due to the two models and the states actually observed by taking the same actions in the environment. If an action resulted in termination, the environment was reset. For each environment, the models were taken from similar training iterations (indicated by $T_{ID}$ in the table) ensuring that the scores had stabilized\n    \n  The difference between the two experiments lies in the duration of the actions. Random one-step sequences (cols 4-5 in the table below) comprises of actions of duration $\\delta t_{\\min}$. For random temporally-extended sequences (cols 6-8 in the table below), action duration $\\sim U(\\delta t_{\\min}, \\delta t_{\\max})$. For each environment, the values of $\\delta t_{\\min}$ and $\\delta t_{\\max}$ are the same as in the paper. Each experiment is repeated 10 times in order to get a better estimate and we report the means below:\n\n  |                               |       |       | Random one step sequences |          | Random temporally extended sequences |          |                           |\n  |-------------------------------|-------|-------|---------------------------|----------|--------------------------------------|----------|---------------------------|\n  | Env                           | $\\delta t_{\\min}$ | $\\delta t_{max}$ | $F_{TE}$                  | $F_{IP}$ | $F_{TE}$                             | $F_{IP}$ | $F_{IP}$ without outliers |\n  | Cartpole $(T_{ID} = 200)$     |   1    |  10     |        0.0107                   |     $2.514 \\times 10^{-7}$     |    0.0184                                  |    0.065      |                 0.065          |\n  | Ant $(T_{ID} = 401)$          |     0.01  |  0.07     |            0.126               |  0.0149        |           0.298                          |   $4.525 \\times 10^{10}$        |    0.60                      |\n  | Half Cheetah $(T_{ID} = 301)$ |     0.01  |  0.07     |                0.30           |     0.04     |             1.752                         |    2.526      |    2.526                       |\n\n  When using random temporally-extended action sequences, in 4 out of 10 experiments the MSE due to $F_{IP}$ explodes. We hypothesize that the states observed during these experiments probably were outside the range of examples seen during the training of the single-step model used by $F_{IP}$. In $F_{IP}$ without outliers, we report the MSE after removing such experiments.\n\n  As can be observed, for one-step sequences, $F_{IP}$ has a lower MSE, but when we evaluate on temporally-extended action sequences, the models incur a higher MSE. This is expected as we make a lot more forward passes through the model when using $F_{IP}$ and hence the error starts to accumulate. \n\n- The comparison between a fixed $\\delta t$ and our proposed approach might depend on the domain being considered. For example, consider the Dubins car environment. For a simple map that requires the car to go in a straight line, a large fixed $\\delta t$ would work extremely well, but if we introduce obstacles, one would require a small value of $\\delta t$. A map that has both of these elements (portions where the car has to travel straight for a long while, and portions where it has to navigate obstacles) would require one to have a conservative value of $\\delta t$. In such situations, having a dynamic approach can be very helpful as the agent is free to adjust the action duration depending on its current position.\n\n  Having said that, having a fixed $\\delta t$ does reduce the computation load on the system. There are environments like Half Cheetah, where for the majority of the episode, the agent takes steps of approximately the same duration even while using the proposed dynamic approach. In such situations, it might be better to just use a fixed value of $\\delta t$. \n\n  Figure 4 in the paper does touch upon this a bit. For Ant and Half Cheetah, we see that the histogram shows that most of the times, the agent takes an action with similar $\\delta t$, however in Hopper the distribution is quite spread out. \n\nThank you again for all your valuable feedback and we hope that we were able to clarify all the questions."}, {"review_id": "RreLY0bMst", "reviewer": "Reviewer_vtVV", "summary": "The paper studies the challenge of modeling continuous time systems using discrete time dynamics, which often requires small simulation steps and large planning horizons. This can lead to computationally demanding planning problems and reduced performance. The authors propose an approach that involves controlling the continuous decision timescale directly by using temporally-extended actions. This allows the planner to treat the duration of the action as an additional optimization variable, alongside standard action variables.\n\nThe authors applied the approach to both planning and model-based RL. in the MBRL setting, the authors learn a dynamic model that is conditioned on the action and its duration. \nthe author also propose to select the range of action duration dynamically over training using a multi-arm bandit method.", "questions": "see questions in weakness", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n- the paper is well motivated and the approach is well presented. overall, it is easy to read the paper.\n- the authors provide various experimental results and that illustrate the effectiveness of their approach in various environments, including both planning and MBRL.\n\nWeakness: \n- MAB solution doesn't seem to improve over the simple fixed range. it converges much slower.\n- planning over temporally extended action can indeed reduce the number of search space but we can lose control granularity. the author focus primarily on successful experiments, it would be helpful to discuss some failure cases.\n- the experimental section focuses mainly on short-horizon tasks (locomotion continuous control tasks). do the authors expect that their method would work in really long-horizon task: for example navigation in big mazes.\n- The authors compare only to planning with primitive actions. could the authors includes results with model-free RL? especially I am interested in there is any gain of the proposed method in term of sample complexity.\n- Moreover, the sample complexity dimension is not discussed. I am not sure in in figure 2 the training iteration means environment interaction or gradient updates ?", "was_revised": true, "final_justification": "The authors addressed most of my concerns.\nBut I still think that the paper don't provide convincing evidence of scalability of their approach to longer-horizon tasks with Learned Model.\n\nTherefore, I maintain my score \"borderline accept\"", "rebuttal": "Thank you for taking the time to review our work and all your thoughtful comments. Please find our responses to the questions below:\n\n-  When comparing the MAB to a fixed $\\delta t_{\\max}$ we are comparing to a good (manually tuned) value of $\\delta t_{\\max}$. Hence it is natural for the search for a value of $\\delta t_{\\max}$ to converge slower. What the experiments show is that the algorithm is able to self-tune $\\delta t_{\\max}$ through the MAB framework and achieve the same quality results. \n\n- We do lose granularity when using $A_{TE}$ and this can be detrimental in problems where longer actions cannot be used. In such cases $A_{TE}$ with a longer search horizon will have similar complexity for $A_{STD}$ (except for increasing the actions space by one dimension) and should perform similarly. In some cases, however, very deep search horizon is too expensive computationally.\n\n- While our experiments with MuJoCo do have shorter horizons, our experiment with u-map in Dubins Car has a planning horizon of 75 (line 256). We did experiment with longer planning horizons for Dubins Car, but since the agent was able to reach with a shorter horizon, we chose the smaller value. We do acknowledge that in this experiment, we use an exact transition function rather than a learned model. \n    \n  However, we think that in any case (learned or exact model), for many practical purposes, using primitive actions for planning result in a larger planning horizon than our proposed framework, and from past work [1] we know that longer planning horizons do lead to compounding errors. So we expect our proposed framework to perform better as compared to planning with primitive actions.\n\n- The comparison to MFRL is interesting. We do not have a direct comparison, but thanks to [2], we know that PETS has much better sample complexity than some of the MFRL algorithms, and the baseline algorithm we use, $A_{STD}$, is essentially the PETS agent. So, we think we can safely assume that the sample complexity of our method would be much better than the MFRL algorithms presented in [2]. \n\n- We think the question was meant for Figure 3 since Figure 2 is for experiments with u-map. \n\n  First, we provide the pseudocode for the data collection and training loop for a fixed $\\delta t_{\\max}$ as it might help clarify things better. In the algorithm,  $n$ refers to the number of training iterations, $D$ is the dataset and $b$ is the batch size.\n\n  ``` \n  for i = 1 to n\n    // Data collection\n    for t = 1 to end of episode\n        a_t = planner.choose_action(s_t)\n        s_{t+1}, r_{t} = env.step(a_t)\n        D = D.append((s_t, a_t, r_t, s_{t+1}))\n    // Training\n    num_of_batches = len(D)/b\n    for j = 1 to num_of_batches\n        sample a mini-batch of size b from D\n        compute the loss and the gradients due to the mini-batch\n        update the dynamics model using the gradient\n  ```\nNote here that to be in sync with the notation in the main paper, $a_t$ includes the duration as well. \n\n  In Figure 3, we plot the agent's performance against the training iterations. As described in the pseudocode above, one training iteration corresponds to one episode worth of data collection. For MuJoCo, the duration of one episode (by which we mean the number of primitive steps taken during one episode) is fixed, while for Cartpole, the max duration per episode is fixed but the actual duration depends on if and when the pole falls.\n\nThank you again for all the valuable feedback and we hope that we were able to clarify all the questions."}, {"review_id": "73hR5peDWJ", "reviewer": "Reviewer_q2pm", "summary": "The paper addresses the challenge of learning temporally extended actions in an MDP setting, for both planning and reinforcement learning. The proposed approach involves learning a transition model and reward estimates corresponding to repeating the same action over multiple time steps. The duration of each decision is also learned from experience using a non-stationary multi-armed bandit model. This method reduces search time in decision-making tasks by decreasing the number of decisions needed to reach a goal. The approach is evaluated across several benchmark domains.", "questions": "1. Does the model support concurrency, as in other frameworks (e.g., Mausam and Weld, 2008)? If not, could it be extended to handle concurrent actions?\n2. The temporal setting is not entirely clear to me. If I understand correctly, the model does not operate in fully continuous time but instead allows durations that are natural multiples of a standard time step. This seems more like an aggregated discrete-time setting than a true temporal one—is that correct?\n3. Assuming the previous point is accurate, does this design eliminate the need to integrate over stochastic effects? In other words, are the resulting stochastic outcomes simply the sum of effects over the individual execution steps?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The model-based representation of temporally extended actions is intuitive and well-motivated.\n2. The use of a non-stationary multi-armed bandit to model time step selection is innovative.\n3. The method is supported by a comprehensive empirical analysis.\n\nWeaknesses:\n1. The distinctions and similarities between the proposed model and other temporal frameworks (e.g., semi-MDPs, co-MDPs, PDDL2.1) should be articulated more clearly.\n2. The approach does not support fully continuous-time modeling.", "was_revised": true, "final_justification": "This is a useful paper, addressing an important setting both for planning and learning. I retain my score after rebuttal.", "rebuttal": "Thank you for taking the time to review our work, for catching the typos and all your thoughtful comments. Please find our responses to the questions below:\n\n- Our model has only a single action duration for all the action variables. So, at present, it does support action concurrency (when different action variables correspond to different parallel actions) but with the constraint that all action variables need to have the same duration. The extension to include multiple action durations would be an interesting (but challenging) extension.\n\n-  Our model is not an exact continuous time model but does not work in discrete unit length steps. Referring to lines 134-136 and Algorithm A1 (line 479 in the appendix): We assume access to a model that can simulate correctly all steps with duration up to $\\delta_t^{\\text{env}}$ and allow the planner to choose duration of arbitrary real values where $F_{IP}$ will perform a \"fractional action\" in its last iteration. As we say in line 136 this is not an exact integration of the reward. \n\n- Similar to the previous question, we do not have access to a continuous time model but instead assume a simulator with a correct simulation for up to $\\delta_t^{env}$. The effects evolve sequentially in that model, i.e., not in a sum but with a repeated sequential function application. \n\nThank you again for all the valuable feedback and we hope that we were able to clarify all the questions."}]

Author rebuttal:

Meta-review: This paper proposes a model-based planning and RL framework that directly optimizes action durations, enabling temporally extended actions that improve planning efficiency and reduce compounding model errors. The method is simple, novel, and broadly applicable. Reviewers pointed out the limited evaluation on simple environments, leaving questions about scalability to complex, long-horizon problems, as well as the unclear explanation of search space reduction. In rebuttal, the authors clarified their formulation but did not add new experiments. 

Overall, I recommend acceptance: the conceptual contribution is strong, the results are promising, and the approach has high potential impact despite the limited empirical validation.
Decision: Accept (poster)