Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control
Authors: Devdhar Patel
Hava T Siegelmann
Venue: ICLR 2025 Poster (2025)
Keywords: Decision Frequency
Action Sequence Generation
Model-Based Training
Model-Free Control
Efficient Learning
Reinforcement Learning
Primary area: applications to robotics, autonomy, planning
TL;DR: We introduce an algorithm that achieves competitive continuous control at extremely slow control frequencies using action sequences

Abstract: Reinforcement learning (RL) is rapidly reaching and surpassing human-level control capabilities. However, state-of-the-art RL algorithms often require timesteps and reaction times significantly faster than human capabilities, which is impractical in real-world settings and typically necessitates specialized hardware. We introduce Sequence Reinforcement Learning (SRL), an RL algorithm designed to produce a sequence of actions for a given input state, enabling effective control at lower decision frequencies. SRL addresses the challenges of learning action sequences by employing both a model and an actor-critic architecture operating at different temporal scales. We propose a "temporal recall" mechanism, where the critic uses the model to estimate intermediate states between primitive actions, providing a learning signal for each individual action within the sequence. Once training is complete, the actor can generate action sequences independently of the model, achieving model-free control at a slower frequency. We evaluate SRL on a suite of continuous control tasks, demonstrating that it achieves performance comparable to state-of-the-art algorithms while significantly reducing actor sample complexity. To better assess performance across varying decision frequencies, we introduce the Frequency-Averaged Score (FAS) metric. Our results show that SRL significantly outperforms traditional RL algorithms in terms of FAS, making it particularly suitable for applications requiring variable decision frequencies. Furthermore, we compare SRL with model-based online planning, showing that SRL achieves comparable FAS while leveraging the same model during training that online planners use for planning.

Published as a conference paper at ICLR 2025
OVERCOMING SLOW DECISION FREQUENCIES IN CONTINUOUS CONTROL: MODEL-BASED SEQUENCE REINFORCEMENT LEARNING FOR MODEL-FREE CONTROL
Devdhar Patel & Hava T. Siegelmann
Manning College of Information and Computer Sciences University of Massachusetts Amherst, MA 01002 USA (devdha (devdharpatel, rpatel, hava) umass .ddu
ABSTRACT
Reinforcement Jearonnn (RL) is rapidly reaching and surpassing human-level con- trol capabilities. However, state-of-the-art RL algorithms often require timesteps and reaction times significantly faster than human capabilities, which is imprac- tical in real-world settings and typically necessitates specialized hardware. We introduce Sequence Reinforcement Learning (SRL., an RI. algorithm designed produce a sequence of actions for given input state, enabling effective control at lower decision frequencies. SRL addresses the challenges of learning action sequences by employing both model and actor-critic architecture operating at different temporal scales. We propose a "temporal recall" mechanism, where the critic uses the model to estimate intermediate states between primitive actions, providing a learning signal for each individual action within the sequence. Once training is complete, the actor can generate action sequences independently of the model, achieving model-free control at slower frequency. We evaluate SRL on suite of continuous control tasks, demonstrating that achieves performance comparable to statc-of-the-at algorithms while significantly reducing actor sam- ple complexity. To better assess performance across varying decision frequencies, we introduce the Srsquency-Averagee Score (FAS) metric. Our results show that SRL significantly outperforms traditional RL algorithms terms of FAS, mak- ing it particularly suitable for applications requiring variable decision frequencies. Furthermore, we compare SRL with model-based online planning, showing that SRL achieves comparable FAS while leveraging the same model during training that online planners use for planning.
INTRODUCTION
Biological and artificial agents must learn behaviors that maximize rewards to thrive in complex environments. Reinforcement learning (RL), a class of algorithms inspired by animal behavior, facilitates this learning process (Sutton & Barto 2018). The connection between neuroscience and RL is profound. The Temporal Difference (TD) eтror, key concept in RL, effectively models the firing patterns of dopamine neurons the midbrain (Schultz et al., 1997] Schultz 2015 Cohen et al. 2012). Additionally, longstanding goal of RL algorithms is to match and surpass human performance control tasks (OpenAl at. 2019 2019900000rrre [Schrittiiserr et al. 2020 Kaufmann et al. 2023b
Wurman al. 2022a Vinyals al. 2019 Mnih et al. 2015).
However, most of these successes are achieved by leveraging large amounts data simulated environments and operating at speeds orders of magnitude faster than biological neurons. For exam- ple, the default timestep for the Humamoid task in the MuJoCo environment (Todorvv et al. 2012) in OpenAl Gym (Towers et al. 2023) is 15 milliseconds. In contrast, human reaction times range from 150 milliseconds (Jain et al. 2015) to several seconds for complex tasks (Limpert) 201T). Table shows the significant gap between Al and humans in terms of timestep and reaction times. When RT.

---

Published as a conference paper at ICLR 2025
agents constrained to human-like like decision frequencies, state-of- oote--of-heearrr algorithms struggle
to perform in simple environments Armld et al. (2021). Figure in Appendix).
Environment / Task
Timestep Reaction Time
Inverted Pendulum
40ms
Walker 2d
8ms
Hopper
8ms
Ant
50ms
Half Cheetah
50ms
Dota 2 Iv1 (OpenAl et al.
2019)
67ms
Dota 2 5v5 (OpenAl et al.
2019)
80ms
GT Sophy (Wurman et al. 20226)
23-30ms
Drone Racing (Kaufmann et al.
2023a)
10ms
Humans
2 150ms
Table I: Timestep/ reaction times for various benchmark environments and popular works that pit humans vs. AI.
The primary reason for this difficulty is the implicit assumption in RL that the environment and the agent operate at constant timestep. Consequently, in embodied agents that implement RL algorithms, all components: sensors, compute units, and actuators are synchronized to the same frequency the algorithmic level. Typically, this frequency is limited by the speed of computation in artificial agents (Katz et al.] 2019). As result, robots often require fast onboard computing hardware (CPU GPU) to achieve higher control frequencies (Margolis et al. 2024 Li al. 2022 Haarnoja et al. 2024).
To allow the RL agent to observe and react changes in the environment quickly, RL algorithms are forced to set high frequency. Even in completely predictable environments, when the agent learns walk or move, small timestep required to account for the actuation frequency required for the task, but it is not necessary to observe the environment as often or compute new actions as frequently. RL algorithms suffer from catastrophic failure due missing inputs (also referred as observational dropout). This behavior level gap between RL and humans can be bridged by bridging the gap in the underlying process.
Towards that end, we propose Sequence Reinforcement Learning (SRL), model for action se- quence learning based on the role of the basal ganglia (BG) and the prefrontal cortex (PFC). Our model learns open-loop control utilizing low decision frequency. Additionally, the algorithm uti- lizes a simultaneously learned model of the environment during its training but can act without for fast and cheap inference. We demonstrate the algorithm achieves competitive performance difficult continuous control tasks while utilizing a fraction of observations and calls to the policy. To the best of our knowledge, SRL is the first achieve this feat. To further quantify this result and set benchmark for control at slow frequencies, we introduce the Frequency Averaged Score (FAS) and demonstrate that SRL achieves significantly higher FAS than Soft-Actor-Crins (SAC) SeeeraaiveePlaannngg--
(GPM) (Zhang et al.) 2022). Additionally, we demonstrate that on complex environments (with high state and action dimensions), SRL also beats model-based online planning on FAS. Finally, in the appendix, we discuss the available evi- dence in neuroscience that has inspired our algorithm and also present promising initial results in the proposed future work of generative replay in latent space.
Haarnoja et al. (219)) and
2 NECESSITY OF SEQUENCE LEARNING: FREQUENCY, DELAY AND RESPONSE TIME
To perform any control task, the agent requires the following three components: Sensor, Proces- sor/Computer, Actuator. In the traditional RL framework, all three components act at the same frequency due to the common timestep. However, this is not the case in biological agents that have different sensors of varying frequencies that are often faster than the compute frequency or the speed at which the brain can process the information (Borghuis et al. 2019). Additionally, in order to af- ford fast and precise control, the actuator frequency is also much faster than the compute frequency (see Figure 9 in Appendix).
2

---

Published as a conference paper at ICLR 2025
Low-compute hardware faces two primary challenges for real-time control: delay and throughput. The high inference times associated with low-compute devices result a delay between receiving observations and performing corresponding actions in the environment. Additionally, they lead to low decision frequencies in sequential decision-making tasks.
While many prior works have focused addressing delay by designing delay-aware algorithms (Chen et al. 2020 2021 Derman et al.] 2021), mitigating delay alone does not resolve the perfor- mance issues caused by low decision frequency. Adapting RL algorithms operate effectively in low-frequency compute settings remains an open challenge (Dulac-Aroold et al. 2021.
The Sequence Reinforcement Learning (SRL) algorithm offers promising solution these low- decision frequency scenarios. To address the complete set of challenges posed by low-compute environments, SRL can be integrated with delay-aware algorithms to simultancously manage delays while achieving higher action frequencies. Moreover, SRL inherently addresses delays by producing sequences of actions that can bridge the gap caused by processing latency. For example, if output arrives with a delay of n timesteps, the first actions of the new sequence can be ignored, as they were already executed as part the previous sequence. This mechanism ensures smooth and continuous action execution despite processing delays.
Why low-frequency compute?
Recent advancements in reinforcement learning (RL) algorithms, combined with high-speed com- puting, have led to two common approaches for addressing the speed-accuracy trade-off:
1. Faster hardware: The use of GPUs has become standard for enabling rapid inference in autonomous agents (Long al. 2024| Csomay-Shankinn et al., 2024] Lazeano 2024). However, GPUs are often impractical in many real-world applications due to their high cost, energy demands, and large physical size. As a result, recent research has also focused on developing specialized embedded deep learning accelerators to address these challenges (Akkad al. 2023).
2. Software optimization: Techniques such as quantization (Jafarpourmarzounii et al. 2024). multi-exit networks (Rahmath et al. 2022). and model compression (Neill) 2020) are commonly employed to reduce inference times without requiring additional hardware.
In essence, these approaches focus on either accelerating hardware or optimizing software. In this work, we propose an alternative paradigm: enhancing accuracy at low operating frequencies in- stead of striving for high frequencies. By advancing research in this direction, we aim relax the dependency high-peus perefrrerfomecc hardware, enabling RL algorithms to operate effectively on low-compute devices while also making ultra ultra- ultra-high-frees high-freyeency control feasible on current hardware platforms.
RELATED WORK
3.1 MODEL-BASHE REINFORCEMENT LEARNING
Model-Based Reinforcement Learning (MBRL) algorithms leverage model of the environment, which can be either learned or known, to enhance RL performance (Moerland al. 2023). Broadly, MBRL algorithms have been utilized to:
1. Improve Data Efficiency: By augmenting real-world data with model-generated data, MBRL can significantly enhance data efficiency (Yarats al. 2021 Janner et 2019 Wang et al. 2021).
2. Enhance Exploration: MBRL aids in exploration by using models identify potential or unexplored states (Pathak et al., 2017 Stadie et al. 2015 Savinov et al. 2019).
3. Boost Performance: Better learned representations from MBRL can lead to improved asymptotic performance (Silver et al. 2017 Levine Koltun 2013).
4. Transfer Learning: MBRL supports transfer learning, enabling knowledge transfer across
different tasks environments (Zhang et al. 2018 Sasso et al. 2023)

---

Published as a conference paper at ICLR 2025
5. Online Planning: Models can be used for online planning with single-step policy (Fickinger et al. 2021). However, this approach increases model complexity, as each on- line planning step necessitates additional call to the model. This makes it unsuitable for applications with limited computational budgets and strict requirements for fast inference.
Compared to online planning, our algorithm maintains a model complexity of zero after training. eliminating the need for any model calls post-training for generating a sequence of actions. This significantly reduces the computational and energy requirements, making it more suitable for prac- tical applications in constrained environments. Additionally, model-based online planning is less biologically plausible than SRL. Wiestler Diedrichsen (013) demonstrated that the activations in the motor cortex reduce after skill learning, suggesting that the brain gets more efficient at perform- ing the task after learning. In contrast, model-based online planning does not reduce the compute and model complexity, but rather might increase in complexity as we perform longer sequences. SRL, on the other hand, has a model complexity of zero after training and thus is biologically plau- sible based on this observed phenomenon.
3.2 MODEL PREDICTIVE CONTROL
Similar to model based reinforcement learning, Model Predictive Control (MPC) utilizes a model of the system to predict and optimize future behavior. In the context of modern robotics, MPC has been effectively applied to trajectory planning and real-time control for both ground and aerial vehicles. MPC has been applied to problems like autonomous driving (Gray al. 2013) and bipedal control Galliker et al.] (002)). Similar to online planning, MPC often requires access to a model of the system after training.
Additionally, similar to current RL, MPC requires very fast operational timesteps for practical appli- cations. For example, Galliker Gaaple.Galllkkreee al. (2022) implemented a walker 10 ms. Farshidian et al. (2017) implemented a four-legged robot at 4 ms, and Di Carlo et al. (2018) implemented the MIT Cheetah at 33.33 ms.
3.3 MACRO-ACTIONS ACTION REPETITION, AND FRAME-SKIPPING
Reinforcement Learning (RL) algorithms that utilize aacconactions demonstrate many benefits, in- cluding improved exploration and faster learning (McGovern et al., 1997). However, identifying effective macro-actions is challenging problem due to the curse of dimensionality, which arises from large action spaces. To address this issue, some approaches have employed genetic algorithms Chang et al., (202)) relied expert demonstrations to extract macro-actions (Kim et al. 2020). However, these methods are not scalable and lack biological plausibility. In contrast, our approach leams macro-actions using the principles of RL, thus requiring little overhead while combining the flexibility of primitive actions with the efficiency of macro-actions.
To overcome the curse of dimensionality while gaining the benefits of macro-actions, many ap- proaches utilize frame-skipping and action repetition, where macro-actions are restricted to a single primitive action that is repeated. Frame-skipping and action repetition serve as form of partial open-loop control, where the agent selects a sequence of actions to be executed without considering the intermediate states. Consequently, the number of actions is linear the number of time steps
(Kalyanakrishman et al. 2021 Srinivas et al. 2017 Biedenkapp et al. 2021 Sharma et al.) 2017
Yu et al.) 2021).
For instance, FiGaR (Sharma et al. 2017) shifts the problem of macro-action learning to predicting the number steps that the outputted action can be repeated. TempoRL (Biedenkapp et al. 2021 improved upon FiGaR by conditioning the number of repetitions on the selected actions. However, none of these algorithms can scale to continuous control tasks with multiple action dimensions, as action repetition forces all actuators and joints to be synchronized in their repetitions, Icading to poor performance for longer action sequences.
TLA Patel et al. 2024) has recently shown enhancement of TempoRL through the implemen- tation of two hierarchical policies functioning at varying timesteps, coordinated by a third policy. Although TLA exhibits commendable results in environments characterized by single action di- mension, its advantages are constrained in muulti-dimensonall environments. This limitation arises as extended timesteps necessitate synchronization across all degrees of freedom, thereby diminishing
4

---

Published as a conference paper at ICLR 2025
performance during prolonged timesteps. In contrast, SRL is capable of executing distinct actions at each timestep without an increase in decision count.
3.4 TEMPORALLY CORRELATED EXPLORATION
Recent advancements in reinforcement learning have extended the concepts of macro-actions and action-repetition to improve exploration by incorporating temporally correlated exploration, where successive actions during exploration exhibit temporal dependencies. For instance, Dabney et al. (202T) proposed temporally extended e-greedy exploration, which involves repeating actions for random durations during exploration. Building on this foundation, subsequent works have investi- gated approaches such as state-dependent exploration |affnn et al. (2022), episodic reinforcement learning Li al. (2024), and temporally correlated latent noise Chiappa al. (2024) to enhance ex- ploration efficiency and improve the smoothness of resulting policies. However, these methods are limited in their adaptability to challenges such as observational dropout, low decision or observa- tional frequency, as the trained policy requires state input at each timestep. To address long-horizon temporally correlated exploration, Zhang et al. (2022) introduced the Generative Planning Method (GPM), which employs recurrent actor network similar to the architecture used in SRL generate sequences of actions from single state. We provide an empirical comparison to GPM Section
SEQUENCE REINFORCEMENT LEARNING
dg
des
dg
dest
Actor (GRU)
BG
PFC
Critic
Model
state 5
reward
Environment
The Sequence Reinforcement Learning (SRL) model. The SRL takes inspiration from the function of the basal ganglia (BG) (Top/Blue) and the prefrontal cortex (PFC) SBottom/Yellowy We train an actor with gated recurrent unit that can produce sequences of arbitrary lengths given single state. This is achieved by utilizing critic and model that acts finer temporal resolution during training/replay to provide an error signal to each primitive action of the action sequence.
Figure 1:
introduce a novel reinforcement learning model capable of learning sequences actions (macro- actions) by replaying memories at finer temporal resolution than the action generation, utilizing a model of the environment during training. We provide the neural basis for our algorithm in the Appendix A.9
COMPONENTS
The Sequence Reinforcement Learning (SRL) algorithm learns to plan "in-the-mind" using a model during training, allowing the learned sequences be executed without the need for model- based online planning. This is achieved using an actor-critic setting where the actor and critic op- crate at different frequencies, representing the observation/computatioo and actuation frequencies, respectively. Essentially, the critic is only used during training/replay and can operate at any tempo- ral resolution, while the actor is constrained to the temporal resolution of the slowest component in the cansing-compute actuaaton loop. Denoting the actor's timestep as t' and the critic's 's timestep as i, our algorithm includes three components:
Model St+1 (,;
Critic q: (s,,
Actor my:++J-1 = ar, te 0r (Str
We denote individual actions in the action sequence generated by the actor using the notation
Ta (st)s

---

Published as a conference paper at ICLR 2025
We denote individual actions in the action sequence mga'+J-1 des, ap'-t; 42' generated by the actor using the notation (w( ) to represent the action ap+4t-
1. Model: Learns the dynamics of the environment, predicting the next state 8++1 given the current state 8, and primitive action
2. Critic:
Takes the same imput as the model but predicts the Q-value of the state-action pair.
3. Actor: Produces sequence of actions given an observation time t'. Observations from the environment can occur at any timestep t or t', where we assume t' > t. Specifically, in our algorithm, I' Jl where 1; J E Z.
Each component of algorithm is trained in parallel, demonstrating competitive learning speeds.
We follow the Soft-Actor-Critic (SAC) algorithm (Haarnoj et al. 2019) for learning the actor-critic. Exploration and uncertainty are critical factors heavily influenced by timestep size and planning horizon. Many model-free algorithms like DDPG (Lillicrap et al. 2019) and TD3 (Fujimoto et al. 2018 explore by adding random noise each action during training. However, planning sequence of actions over longer timestep can result in additive noise, leading to poor performance during training and exploration the noise parameter not tuned properly. The SAC algorithm addresses this by automatically maximizing the entropy while also maximizing the expected return, allowing our algorithm to automatically tune its exploration based on the selected sequence length parameter (J).
LEARNING THE MODEL
The model is trained to minimize the Mean Squared Error of the predicted states, For a tra- jectory (st, ag S(+1) drawn from the replay buffer D, the predicted state is taken from 8141 ms az). The loss function is:
C. = En - s1+1)²
(2)
For this work, the model is feed-forward neural network with two hidden layers. In addition to the current model m also maintain a target model m that the exponential moving average of the current model.
LEARNING THE CRITIC
The critic is trained to predict the Q-value of a given state-action pair q: = (s), az) using the target value from the modified Bellman equation:
r /B 1) 1g ($1+1, a++1) a log nw (ac-
(3)
Here, 9 the target critic, which is the exponential moving average of the critic and a is the temperature parameter that controls the relative importance of the entropy term. Following the SAC algorithm, we train two critics and use the minimum of the two 9 values to train the current critics. The loss function is:
C. Er-Dl(âa á 2 E 1,2
Both critics are feed-forward neural networks with two hidden layers. should be noted that while the actor utilizes the model during training, the critic does train on any data generated by the model, thus the critic training is model-free and grounded in the real environment states.
LEARNING THE POLICY
The SRL policy utilizes two hidden layers followed by Seted-Reeurrente Unit (GRU) (Cho et al., 2014) that takes input the previous action in the action sequence, followed by two linear layers that output the mean and standard deviation the Gaussian distribution of the action. This design allows the policy to produce action sequences of arbitrary length given single state and the last action.
A naive approach to training sequence of actions would be to augment the action space to include all possible actions of the sequence length. However, this quickly leads to the curse dimensional- ity, each sequence is considered unique action, dramatically increasing the policy's 's complexity.

---

Published as a conference paper at ICLR 2025
Additionally, such approach ignores the temporal information of the action sequence and faces the difficult problem of credit assignment, with only a single scalar reward for the entire action sequence.
To address these problems, we use different temporal scales for the actor and critic. The critic assigns value to each primitive action of the action sequence, bypassing the credit assignment problem caused by the single scalar reward. However, using collected state-action transitions to train the action sequence is impractical, changing the first action in the sequence would render all future states inaccurate. Thus, the model populates intermediate states, which the critic then uses to assign value each primitive action the sequence.
Therefore, given trajectory = (at-1. St, at s(+1), we first produce the J-step action sequence using the policy: M (st). We then iteratively apply the target model to get the interme- diate states $141#+J-1- Finally, we use the critic to calculate the loss for the actor as follows:
Lw = LiND alog (s āt)
a log Tw ((r+5]55tt) q (5o4j acts)
(5)
EXPERIMENTS
OVERVIEW
We evaluate our SRL approach on continuous control tasks, comparing it against SAC (Haarnoja al. 2019) and GPM (Zhang al.) 2022). We utilize the OpenAl Gym (Brockman al. 2016) implementation of the MuloCo environments (Todorov et al. 2012).
EXPERIMENTAL SETUP
We train SRL with four different action sequence lengths (ASL), J 2, 2,4,8, 16, referred to as SRL-J. During training, SRL is evaluated based on its J value, processing states only after every actions. All hyperparameters are identical between SRL and SAC, except for the actor update frequency: SRL updates the actor every 4 steps, while SAC updates every step. Thus, SAC has four more actor update steps compared to SRL. Additionally, SRL learns a model in parallel with the actor and critic. Additionally, we also train SAC at different step sizes that correspond to SRL, forming SAC-J where 1, 2, 4, 8, 16. Note that we do not provide SRL-I since for sequences of length SRL is the same algorithm as SAC.
We present the learning curves of SRL and SAC across continuous control tasks in the appendix. We find that on all environments except Swimmer, SAC-1 demonstrates optimal performance and often significantly outperforms the longer timesteps. Thus, the default environments are picked to maximize performance under the standard RL setting where the observation, decision, and the action frequency are the same. It should be noted that the learning curves presented for SRL-J and SAC-J take in states every J steps.
FREQUENCY-AVEREE
SCORE
Transitioning from simulation to real-world implementation (Sim2Real) in control systems is chal- lenging because deployment introduces computational stochasticity, leading to variable sensor sam- pling rates (throughput) and inconsistent end-to-end delays from sensing to actuation (Sandha et al., 2021). This gap not captured by the mean reward or return that the norm current RL literature. To address this, we introduce curve (AUC) of the performance vs. decision frequency plot. We provide plots for all environments in the Appendix. We note that this experimental setup is similar to the challenge 7 introduced in by Dulac-Arnold et al. (202l) and SRL addresses the challenge of low throughput that introduced in that work. The FAS captures the overall performance of the policy different decision frequencies, timesteps or macro-action lengths. A High FAS indicates that the policy performance generalizes across decision frequencies, observation frequencies and timestep sizes.
Seaquency-Averagee Score (FAS) that is the normalized area under the
Tables 2 and 3 present the Frequency Averaged Score (FAS) for SAC and SRL across vary- ing action sequence lengths. Overall, SRI-16 demonstrates strong and consistent performance
7

---

Published as a conference paper at ICLR 2025
Environment
SAC-1
SAC-2
SAC-4
SAC-8
SAC-16
Pendulum
0.44 + 0.03
0.42 + 0.03
0.50 0.03
0.49 + 0.04
0.33 0.05
Lunar Lander
0.20 + 0.02
0.23 1 0.02
0.33::0:00
0,45 0.03
0.56 + 0.09
Hopper
0.07 + 0.01
0.09
0.01
0.14 0.03
0.14 + 0.04
0.26 0.08
Walker2d
0.07 + 0.01
0.08 + 0.03
0.14 + 0.04
0.23 + 0.07
0.15 0.04
Ant
-0,05 + 0.04
0.11 111::0000 0.01
0.16 + 0.02
0.16 + 0.01
0.13 + 0.01
HalfCheetah
0.01 + 0.01
0.04 0.01
0.03 0.00
0.02 = 0.01
0.01 0.01
Humanoid
0.06 = 0.01
0.06 1 0.01
0.08 = 0.03
0.17 = 0.02
0.18 - 0.04
InvertedPendulumm
0.05 = 0.02
0.07 #.00
0.14 0.00
0.31 = 0.02
0.34 + 0.20
DnvertedDPendullle
0.02 + 0.00
0.07 0.00
0.09 + 0.01
0.01 + 0.00
0.01 0.00
Reacher
0.65 + 0.07
0.78 + 0.01
0.84 + 0.03
0.86 + 0.02
0.87 + 0.02
Swimmer
0.08 + 0.02
0.28 + 0.04
0.46 0.03
0.53 + 0.03
0.54 + 0.06
Table 2: Mean Frequency-Averageed Score (FAS) and standard deviation for different environments for SAC-J configurations (J = 1,, 2,2,4,8 4,8, 8, 16. J is the action sequence length during training). Each value is averaged over trials (rounded to two decimals, highest value highlighted).
Environment
SRL-2
SRL-4
SRL-8
SRL-16
Pendulum
0.49 0.04
0.68
0.02
0.78 + 0.04
0.88 = 0.02
Lunar Lander
0.14 0.06
0.52 0.03
0.73 + 0.04
0.84 0.03
Hopper
0.10 0.02
0.23
0.42 0.04
0.57 0.02
Walker2d
0.25 - 0.06
0.28 + 0.06
0.24 0.11
Ant
0.04 =
0.29 0.09
0.45 + 0.14
0.54 0.13
HalfCheetah
1.06:0.0
0.13 + 0.02
0.22 0:01
0.28 0.01
Humanoid
0.07 0.00
0.18 + 0.02
0.37 0.04
0.46 0.04
InvertedPendulum
0.09 + 0.03
0.16 0.03
0.27 0.02
0.44 0.04
InenruudDPendulumm
0.07 0.00
0.13 + 0.02
0.03 0.02
0.02 0.00
Reacher
0.90 0.01
0.93 - 0.00
0.95 + 0.00
0.96 - 0.00
Swimmer
0.32 0.05
0.38 0.17
0.31 + 0.02
0.42 = 0.15
Table 3: Mean Frequency-Averaged Score (FAS) and standard deviation for different environments for SRL-J configurations (J 2, 2,4, 4,4,8 8, 16. the action sequence length during training). Each value is averaged over trials (rounded to two decimals, highest value highlighted).
across most environments and a wide range of frequencies. However, in the Walker2d-v2 and Invered/Daableeendduum222 environments, SRL faces challenges when learning longer action quences. We hypothesize that these difficulties stem from higher modeling errors in these envi- ronments. Future work aimed at improving environmental models could potentially address these issues.
SAC, in contrast, performed poorly across all environments, highlighting the limitations of tradi- tional RL methods in adapting to changes in frequency. Although training SAC with larger timesteps (J) improves FAS, this approach compromises performance shorter timesteps. ultimately reduc- ing the overall score (see Appendix Fig. 5)
exception to this trend is the Swimmer environment, where SAC benefits from improved explo- ration due to extended actions. SRL, which does not use action repetition, does not perform as well this specific case. However, this limitation could be addressed by incorporating action repetition or action correlation during exploration- enhancement that lies beyond the scope of the current work.
In order further validate the utility of FAS, test all the policies (SAC and SRL-J) in stochastic timestep environment. The timestep (time until next input) is randomly chosen from a uniform distribution of integers in [1,16] after each decision. This is more realistic setting it tests the performance of the policy when the frequency is not constant. Each policy is evaluated over 10 episodes with stochastic timesteps.
In all tested environments, except for the Inverted Double Pendulum, there is a strong Pearson correlation coefficient (greater than or equal 0.82) between FAS and performance in stochastic
8

---

Published as a conference paper at ICLR 2025
Peoditmm-
Laneel mnket Unnimeeee-22
20g Kerant
Avg. Koranti
Avg Ronanti
FR
1003
GPM--1
GPM-
SRL
110)
0.M
Trining Steps
Trxning Steps
Trising Stegs
Trainig Steps
Pendetum--
Lameel suket (mnimaaceee
3000
1000
Ang Kownn
Ang. Kowand
Ang. Rowani
Ang. Rewsed
12
GPM-
583 16
S8L
200
58L 15
3EL
SAC-4
-300
SAC-16
SAC-16
SAC-4
1000 1590
5K0
1960
500
1993
1000
1500
A100000 Seggenoe Length
Seqpence
Segpecee Lengts (mS)
Segreace Length
Figure 2: Comparison of SAC and SRL to GPM. Top: Learning curves. Bottom: Performance of the trained policies at different action sequence lengths. The action sequences for SRL and GPM are generated using the recurrent actor while SAC utilizes action repetition. GPM achieves FAS of 0.41, 0.04, 0.04, 0.04 on the environments from left to right respectively.
conditions. This high correlation confirms the effectiveness of FAS as metric for measuring policy's generalized performance across various timesteps and frequencies. The Inverted Double Pendulum, however, presents unique challenge due to its requirement for high precision at low decision frequencies, leading to significantly lower FAS scores for all algorithms and thus it is an outlier. Comprehensive plots for all nine environments are included in the appendix (Fig. 7
COMPARISON TO GENERATIVE PLANNING METHOD
The Generative Planning Method (GPM) (Zhang al.) 2022) uses a recurrent actor, like SRL, to generate actions for improved exploration. Originally designed for different context and evaluated the standard RL setting, GPM optimizes plan actions maximize Q-value, potentially exceeding SAC in FAS score. We compare SRL and GPM in four environments test this.
In the original work, GPM was trained with a plan length 3, similar to the parameter in our study. Though shorter plans may restrict generalization to longer sequences, GPM is robust to plan length variations. For fair comparison, we use the best-performing J values for SRL in each environment.
Figure 2 shows the learning curves and FAS evaluation plots for GPM compared to SAC and SRL. While GPM generates a plan by optimizing a sequence of actions, it achieves optimal performance only at sequence lengths of one. As result, its FAS score is even lower than that of SAC-J.
Notably, on the InvertedDoubleeenduuum mance at action sequence lengths (ASL) of 4, which aligns with their training J = 4. However, their performance decreases at shorter ASLs. In contrast, GPM shows a similar FAS profile to SAC-1, indicating that its performance does not generalize well longer action sequences.
environment, both SAC and SRL exhibit high perfor-
COMPARISON TO MODEL-BASED ONLINE PLANNING
Model-based online planning is another approach that allows the RL agent to reduce its observa- tional frequency. However, it often requires highly accurate model of the environment and incurs increased model complexity due to the use the model during control.
Since SRL incorporates model of the environment that is learned in parallel, we compare the performance of the SRL actor utilizing the actor cetor-geneeated action sequences against model-based online planning, where the actor produces only single action between each simulated state.
Table compares the FAS score SRL to online planning using the same model in online plan- ning versus the action sequences generated by the SRL policy. We that SRL can learn action

---

Published as a conference paper at ICLR 2025
Environment
SRL
Online Planning
State Space
Action Space
Lunar Lander
0.84 + 0.03
0.79 + 0.08
2
Hopper
0.57 1 0.02
0.59 + 0.19
11
3
Walker2d
0.28 0.06
0.20 0.05
6
Ant
0.54 + 0.13
0.34 + 0.08
27
8
HalfCheetah
0.28 + 0.01
0.19 + 0.02
17
6
Humanoid
0.46 + 0.04
0.18 0.03
376
17
InvPendulum
0.44 + 0.04
0.63 - 0.10
4
1
InyDPendulum
0.13 + 0.02
0.10 - 0.07
11
Reacher
0.96 + 0.00
0.95 0.00
2
Swimmer
0.42 + 0.15
0.43 + 0.14
2
Table 4: Comparison of the FAS of SRL and corresponding model-based online planning policies across different environments.
sequences and is competitive to model-based online planning. Notably, SRL performs better in en- vironments with larger action and state space dimensions. Such environments are harder to model. Thus, SRL leverage inaccurate models to learn accurate action sequences, further reducing the required computational complexity during training. We hypothesize that this superior performance due to the fact that the actor learns J-step action sequence concurrently, while online planning only produces one action at a time. Consequently, SRL. is able to learn and produce long, coher- ent action sequences, whereas single-step predictions tend to drift, similar to the "hallucination" phenomenon observed in transformer-based language models.
DISCUSSION AND FUTURE WORK
SRL bridges the gap between RL and real-world applications by enabling robust control at low de- cision frequencies. Its ability to learn long action sequences expands the potential for deploying RL in noource-constraine
environments, such robotics and autonomous systems. Additionally, it shows promise for applications where obtaining observations is costly, such as in medical diagnos- tics and treatment planning. Future work will explore hierarchical policies and biologically inspired attention mechanisms.
The current RI. framework encourages synchrony between the environment and the components of the agent. However, the brain utilizes components that act at different frequencies and yet is capable of robust and accurate control. SRL provides an approach to reconcile this difference between neu- roscience and RL, while remaining competitive on current RL benchmarks. SRL offers substantial benefits over traditional RL algorithms, particularly in the context of autonomous agents in con- strained settings. By enabling operation at slower observational frequencies and providing gradual decay in performance with reduced input frequency, SRL addresses critical issues related to sen- sor failure and occlusion, and energy consumption. Additionally, SRL generates long sequences of actions from a single state, which can enhance the explainability the policy and provide opportu- nities to override the policy early in case of safety concerns. SRL also learns latent representation of the action sequence, which could be used in the future to interface with large language models for multimodal explainability and even hierarchical reinforcement learning and transfer learning.
7 CONCLUSION
In this paper, we introduced Sequence Reinforcement Learning (SRL): a model-based action se- quence learning algorithm for model-free control. We demonstrated the improvement of SRL over the existing framework by testing it over various control frequencies. Furthermore, we introduce the Frequency-Averagee-SSeoo
(FAS) metric measure the robustness of a policy across different frequencies. Our work is the first to achieve competitive results on continuous control environments at low control frequencies and serves benchmark for future work in this direction. Finally, demonstrated directions for future work, including comparison to model-basee planning, generative replay, and connections to neuroscience.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like thank Dr. Terrence Sejnowski for his valuable discussions, insightful feedback, and guidance throughout this work. His expertise and support have been instrumental in refining the ideas presented in this paper.
REFERENCES
Ghattas Akkad, Ali Mansour, and Elie Inaty. Embedded deep learning accelerators: survey on recent advances. IEEE Transactions Artificial Intelligence, 2023.
Gregory Berns and Terrence Sejnowski. How the basal ganglia make decisions. In Neurobiology of decision-muking pp. 101-113. Springer, 1996.
Gregory Berns and Terrence J Sejnowski. A computational model of how the basal ganglia produce sequences. Journal of cognitive neuroscience, 10(1):108-13) 1998.
André Biedenkapp. Raghu Rajan, Frank Hutter, and Marius Lindauer. Temport: Learning when to act. In International Conference on Machine Learning, pp. 914 PMLR, 2021.
Bart G Borghuis, Duje Tadin, Martin JM Lankheet, Joseph S Lappin, and Wim A van de Grind. Temporal limits visual motion processing: psychophysics and neurophysiology. Vision, 3(1); 5, 2019.
LA Boyd, JD Edwards, CS Siengsukon, ED Vidoni, BD Wessel, and MA Linsdell. Motor sequence chunking is impaired by basal ganglia stroke. Neurobiology of learning and memory, 92(1): 35-44 2009.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
01500
Yi-Hsiang Chang, Kuan-Yo Yu Chang, Henry Kuo, and Chun-Yi Lee. Reusability and transferabilityy of macro actions for reinforcement learning. ACM Transactions Evolutionary Learning and Optimization, 2(1):1-16, 2022.
Baiming Chen, Mengdi Xu, Zuxin Liu, Liang Li, and Ding Zhao. Delay-aware multi-agent reinforcement learning for cooperative and competitive environments, 2020. URL https
//arxiv. wtwpiv.oorr//aaa abs attaaaaa......a.aa 05441
Baiming Chen, Mengdi Xu, Liang Li, and Ding Zhao. Delay-aware model-based reinforcement learning for continuous control. Neurocomputing, 450:119-122 128, 2021.
Alberto Silvio Chiappa, Alessandro Marin Vargas, Ann Huang, and Alexander Mathis. Latent ploration for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1724 1734. ACL, 2014. doi: 10.3115/V1/D14-1173 URL 3115///.....111111111 1144331111/////111733
Jeremiah Y. Cohen, Sebastian Haesler, Linh Vong, Bradford B. Lowell, and Naoshige Uchida. Spurseitype-speccfic signals for reward and punishment in the ventral tegmental area. Na-
fure 2012 482:7383, 482:85-88, 2012. ISSN 1476-4687. doi: 10.1038/nature10755. URL
https: www. ure com wwrmm/atiiclen http:///rcceeee/ateee1.......
Noel Csomay-Shanklin, William D. Compton, and Aaron D. Ames. Dynamically feasible path plan-
ning in cluttered environments via reachable bezier polytopes, 2024. URL https: : potttps//aaxii. //arxiv.
org/abs/220 /411 13507
11

---

Published as a conference paper at ICLR 2025
Will Dabney, Georg Ostrovski, and André Barreto. Temporally-extended e-greedy exploration. In 9th International Conference on Learning Representations, ICLR 2021, Viruall Event. Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://ppenzeevew. het/forummiiic
ONEPHE7 zG4
Esther Derman, Gal Dalal, and Shie Mannor. Acting in delayed environments with non-stationary markov policies. In 9th International Conference on Learning Reprexentations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. httt:::t//ppeeeeeeeeet
net/forum? (immmidd=))
RMMKe 1 P
Jared Di Carlo, Patrick M Wensing, Benjamin Katz, Gerardo Bledt, and Sangbae Kim. Dynamic locomotion in the mit cheetah through convex preec-predictive control. In 2018 IEEF/RSI international conference on intelligent robots and systems (IROS), pp. 1-9. IEEE, 2018.
Allison J Doupe, David Perkel, Anton Reiner, and Edward A Stern. Birdbrains could teach basal ganglia research a new song. Trends neurosciences, 28(7):353-365 2005.
Gabriel Dulac-Amold, Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learn- ing, 2021. URL httph: http://aaxvvvv.. org/ abs 2003 11881
Farbod Farshidian, Michael Neunert, Alexander W Winkler, Gonzalo Rey, and Jonas Buchli. An efficient optimal planning and control framework for quadrupedal locomotion. In 2017 IEEE International Conference Robotics and Automation (ICRA), pp. 93-100. IEEE, 2017.
Natalia Favila, Kevin Gurney, and Paul G Overton. Role of the basal ganglia in innate and learned behavioural sequences. Reviews in the Neurosciences, 35(1):35-55, 2024.
Arnaud Fickinger, Hengyuan Hu, Brandon Amos, Stuart Russell, and Noam Brown. Scalable online planning via reinforcement learning fine-tuning. Advances in Neural Information Processing Systems, 34:16951-16966 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Manuel Y Galliker, Noel Khomay-Shanklin, Ruben Grandia, Andrew J Taylor, Farbod Farshidian, Marco Hutter, and Aaron D Ames. Planar bipedal locomotion with nonlinear model predictive control: Online gait generation using whole-body dynamics. In 2022 IEEE-RAS 21st Interna- tional Conference on Humanoid Robots (Humanoids), pp. 622-629. IEEE, 2022.
Eric Garr. Contributions of the basal ganglia to action sequence learning and performance. Neuro- science & Biobehavioral Reviews, 107:279-295. 2019.
Christoph F Geissler, Christian Frings, and Birte Moeller. Illuminating the prefrontal neural corre- lates of action sequence disassembling in response- response binding. Scientific Reports, 11(1): 22856, 2021.
Andrew Gray, Yiqi Gao, J Karl Hedrick, and Francesco Borrelli. Robust predictive control for semi-autonomous vehicles with an uncertain driver model. In 2013 IEEE intelligent vehicles symposium (IV). pp. 208-213, IEEE, 2013.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications, 2019. URL https: http///axiv.vv. org/
05905
Tuomas Haarnoja, Ben Moran, Guy Lever, Sandy H Huang, Dhruva Tirumala, Jan Humplik, Markus Wulfmeier, Saran Yunyasuvunakool, Noah Y Siegel. Roland Hafner, et al. Learning agile soccer skills for a bipedal robot with deep reinforeement learning. Science Robotics, :8a)x8228 2024.
Maarten A Immink, Monique Pointon, David L Wright, and Frank E Marino. Prefrontal cortex activation during motor sequence learning under interleaved and repetitive practice: a two-channel near-infrared spectroscopy study. Frontiers in Human Neuroscience, 15:644968, 2021.
12

---

Published as a conference paper at ICLR 2025
Reza Jafarpourmarzouni, Yichen Luo, Sidi Lu, Zheng Dong, al. Towards real-time and efficient perception workflows software-defined vehicles. IEEE Internet of Things Journal, 2024.
Aditya Jain, Ramta Bansal, Avnish Kumar, and KD Singh. A comparative study of visual and auditory reaction times on the basis of gender and physical activity levels of medical first year students. International journal of applied and basic medical research, 5(2):124-12)) 2015.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in neural information processing systems, 32, 2019.
Xin Jin and Rui M Costa. Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature, 466(7305) 2010.
Xin Jin and Rui M Costa. Shaping action sequences in basal ganglia circuits. Current opinion in neurobiology, 33: 18-,96, 2015.
Xin Jin, Fatuel Tecuapetla, and Rui M Costa. Basal ganglia subcircuits distinctively encode the parsing and concatenation of action sequences. Nature neuroscience, 17(3):423 430-430, 2014.
Shivaram Kalyanakrishman, Siddharth Aravindan, Vishwajeet Bagdawat, Varun Bhatt, Harshith Goka, Archit Gupta, Kalpesh Krishna, and Vihari Piratla. An analysis of frame-skipping in rein- forcement learning, 2021. URL httLhhtts://et attp//axxvvaa http:/awwwwv...oaaaa//o 0 03718
Benjamin Katz, Jared DI Carlo, and Sangbae Kim. Mini cheetah: A platform for pushing the limits of dynamic quadruped control. Proceedings IEEE International Conference on Robotics and Automation, 2019 M019--Myyy9956663333
2019. ISSN 10504729. doi: (0.1109/CRA.2209..177 8793889.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):98-998 2023a.
Elia Kaufmann. Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-leved drone racing using deep reinforcement learning. Nature 2023 620:7976, 620:982 900,982-877, 8 2023b. ISSN 1476-4687. doi: 10.
URL https: www nature wtw.:occciieeeeeeeeee wrt.aattiicccc. s41 58 0---588-----20 0 6419-4]
Heecheol Kim, Masanori Yamada, Kosuke Miyoshi, Tomoharu Iwata, and Hiroshi Yamakawa. Re- inforcement learning in latent action sequence space. In 2020 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 5497-5503. IEEE, 2020.
Vanel Lazcano. Depth map completion using specific graph metric and balanced infinity lapla- cian for autonomous vehicles. In Iberoamerican Congress on Pattern Recognition, pp. 187-197. Springer, 2024.
Sergey Levine and Vladlen Koltun. Guided policy search. International conference on machine learning, pp. 1-9. PMLR, 2013.
Ge Li, Hongyi Zhou, Dominik Roth, Serge Thilges, Fabian Otto, Rudolf Lioutikov, and Ger- hard Neumann. Open the black box: Step-based policy updates for temporally-correlated episodic reinforcement learning. resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://cpenreviie net/ forum? forum?id=nniie a.d-mniipvv17..
The Twelfth International Conference on Learming Rep-
Qikai Li. Guiyu Dong, Ripeng Qin, Jiawei Chen, Kun Xu, and Xilun Ding. Quadruped rein- forcement learning without explicit state estimation. In 2022 IEEE International Conference Robotics and Biomimetics (ROBIO), pp- 1989-1994, IREE, 2022.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel. Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2019.
URL httthtttts: http://///aat..orr///a/////0.000
Rudolf Limpert. Brake design and safety. SAR international, 2011.
13

---

Published as a conference paper at ICLR 2025
Junfeng Long. Junli Ren, Moji Shi, Zirui Wang, Tao Huang, Ping Luo, and Jiangmiao Pang. Learn ing humamoid locomotion with perceptive internal model, 2024. URL abs abs/2411.. 14386
Paola Malerba, Katya Tsimring. and Maxim Bazhenov. Learning-inn induucdd sequence reactivation during sharp-wave ripples: a computational study. In Advances in the Mathematical Sciences: AWM Research Symposium, Los Angeles, CA, April 2017, pp. 173-204. Springer. 2018.
Gabriel B. Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid lo- comotion via reinforcement learning. International Journal of Robotics Research, 43:572- 587, 4 2024. ISSN 17413176. doi: 10. 7107///77344/211224/////EE/// 10.1 001177007784422122445---110....
URL httpp:://our.aaas..... sagepub. com/
doi fat./Fa11100.1177777766622233333
Miriam Matamales, Zala Skrbis, Matthew R Bailey, Peter D Balsam, Bernard W Balleine, Jürgen Götz, and Jesus Bertran-Gonzalee A corticostriatal deficit promotes temporal distortion of auto- matic action in ageing. ELife, 6:c29908, 2017.
Amy McGovern, Richard S. Sutton, and Andrew H. Fagg. Roles of macro-actions in accelerating reinforcement learning. 1997.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane I Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature 2015 51:: 518:529 533, 2015. ISSN 1476-4687. doi: 10. 1038/nature URL,
https: www. nww.ratuuee.... com//ooooomoooommmmooooe/o/cctce art www.aaatiilleee.mm https//nattreeeeee
Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model-based rein- forcement learning: survey. Foundations and Trends in Machine Learning, 16(1):1-118, 2023.
James O' Neill. An overview of neural network compression, 2020. URL https://arxi::
org/ abaaabss20000 2006 03669
OpenAl, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov. Henrique
O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning, 2019. URL hhttps://azxiv..rttt abs / 06680
Devdhar Patel, Terrence Sejnowski, and Hava Siegelmann. Optimizing attention and cognitive con- trol costs using temporally layered architectures. Neural Computation, 36(12):2734 2763, 11 2024. ISSN 0899-76677. doi: 10. 1661neoo a a.01718. URL httthttt////ciii... http://cc..org////////
neco_a a 017118
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
James G Phillips, Ed Chiu, John Bradshaw, and Robert lansek. Impaired movement sequencing in patients with huntington's disease: kinematic analysis. Neuropsychologia, 33(3):365-36,, 1995.
Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Conference on robot learning, pp. 1634-1644, PMLR, 2022.
Haseena Rahmath P, Vishal Srivastava, Kuldeep Chaurasia, Roberto Pacheco, and Rodrigo Couto. Early-exit deep neural network-a comprehensive survey. ACM Computing Surveys, 2022.
14

---

Published as a conference paper at ICLR 2025
Daniel B Rubin, Tommy Hosman, Jessica N Kelemen, Anastasia Kapitonava, Francis R Willett, Brian Coughlin, Eric Halgren, Eyal Y Kimchi, Ziv M Williams, John D Simeral, et al. Learned motor patterns are replayed in human motor cortex during sleep. Journal of Neuroscience, 42 (25):5007-5020, 2022.
Sandcep Singh Sandha, Luis Garcia, Bharathan Balaji, Fatima Anwar, and Mani Srivastava. Sim2real transfer for deep reinforcement learning with stochastic state transition delays. In Con- ference on Robot Learning, pp. 1066-1083. PMLR, 2021.
Remo Sasso, Matthia Sabatelli, and Marco A. Wiering. Multi-source transfer learning for deep model-based reinforcement learning. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL http:///eppeeeeeeeeeeeaeeooooono
Nikolay Savinov, Anton Raichuk, Damien Vincent, Raphaël Marinier, Mare Pollefeys, Timothy P. Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. In 7th International Con- ference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, Open- Review.net, 2019.
htt.:/ett//ooteeee id SkeK3s0qKQ
Julian Schritwiieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature 2020 588: 588:604 609, 12 2020. ISSN 4686- doi: 10. 10.100//411886000 18844110-0000005-55-0 020-03051-4. URL
https: www naturatuuee.... com/ htt//aaticcllse sel.e/////411555 03051--43
W. Schultz, P. Dayan, and P. R. Montague. A neural substrate of prediction and reward. Science,
275:1593-1599, 1997. ISSN 00368075. doi: 10. (A1112//SCEENC.275.5555555 193//SSET/
URL httphhtps//wwwtt www.:wwwcince... org http://wwwcceeeeeeeecc.oo..oooe/
275. 530....0...5533
Wolfram Schultz. Neuronal reward and decision signals: From theories to data. Physiological doi: 10. V0.115277YYSRRV00022200444AS
Reviews, 95:853-951, 7 2015. ISSN 15221210.
MAAGISLLARGE/220001122200099.9
URL fttpp::/jjorraass.pyys...
erg/doi/10. 11t.:/...1112p... 000.33...e................0...
Danesh Shahnazian, Mehdi Senoussi, Ruth M Krebs, Tom Verguts, and Clay B Holroyd. Neural representations of task context and temporal order during action sequence execution. Topics in Cognitive Science, 14(2):223-244 2022.
Sahil Sharma, Aravind S. Lakshminarayanan, and Balaraman Ravindran. Learning to repeat: Fine grained action repetition for deep reinforcement learning. In 5th International Confer- ence Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. (.1.7.RRLhEEE
net forum?iddd
B1 B1GOWV5eg
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-355 2017.
A. Srinivas, Sahil Sharma, and Balaraman Ravindran. Dynamic action repetition for deep reinforce- ment learning. In AAAI, 2017.
Bradly C. Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforce- ment learning with deep predictive models, 2015. 00814
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi:
15

---

Published as a conference paper at ICLR 2025
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, An- drea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymna- sium, March 2023. URL /ttp:ttts////eedo.oorro
zenodo. www.://ooooooooooooooooooooooooo
Gido M Van de Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature communications, 1 11(1):4069, 2020.
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnekki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Aga- piou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, To- bias Pfaff, Yuhuai Wu, Roman Ring. Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature 2019 575:7782, 575:350-333 354, 2019. ISSN 1476-4687, doi: 10010008/41188-019-111111 019- 1724
URL https: www nature com wat.ture.coo/atttceee///////////
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 34:29420- 29432, 2021.
Tobias Wiestler and Jörn Diedrichsen. Skill learning strengthens cortical representations of motor sequences. Elife, 2,c00801, 2013.
Peter R. Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J. Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, Leilani Gilpin, Piyush Khandelwal, Varun Kompella, Hao Chih Lin, Patrick MacAlpine, Declan Oller, Takuma Seno, Craig Sherstan, Michael D. Thomure, Houmehr Aghabozorgi, Leon Barrett, Rory Douglas, Dion Whitehead, Peter Dürr, Peter Stone, Michael Spranger, and Hiroaki Ki- tano. Outracing champion gran turismo drivers with deep reinforcement learning. Nature 2022 602:7896, 602:223-228, 2 2022a. ISSN 1476-4687. doi: 021 04357-7
URL
https:/ www. nature. com http///ticesssss11111..
Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Out- racing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896):223- 228, 2022b.
Nicholas F Wymbs and Scott T Grafton. The human motor system supports sequence-specific representations over multiple deeeing- timescales. Cerebral cortex, 25(11):4213-4225, 2015.
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https:// 2020.
github. http://cc/deeiiyiyeee htt.h//..ooneeenlllaaaaaattrrooe
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improv- ing sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10674-10681, 2021.
Haonan Yu, Wei Xu, and Haichao Zhang. Taac: Temporally abstract actor-critic for continuous control. Advances in Neural Information Processing Systems, 34:29021-29033, 2021.
Amy Zhang, Harsh Satja,, and Joelle Pineau. Decoupling dynamics and reward for transfer learn- ing. 6th International Conference on Learning Representations, ICLR 2018, Vancouver BC, Canada, April 30- May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018. URL https: pettps://openreveee hwtp//eeeeeeiieeeep/o/pe net/ aeauumiddHHaoodyye
Haichao Zhang, Wei Xu, and Haonan Yu. Generative planning for temporally coordinated ex- ploration in reinforcement learning. In The Tenth International Conference Learning Rep- resentations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022. URL
https://openrevier net:/oorumid=
16

---

Published as a conference paper at ICLR 2025
Yi Zhao, Wenshuai Zhao, Rinu Boney, Juho Kannala, and Joni Pajarinen. Simplified temporal con- sistency reinforcement learning. In Interattional Conference on Machine Learning, pp. 42227- 42246. PMLR, 2023.
Mark C Zielinski, Wenbo Tang, and Shantanu P Jadhav. The role of replay and theta sequences in mediating hippocampal-prefronne interactions for memory and cognition. Hippocampus, 30(1): 60-72, 2020.
17

---

Published as a conference paper at ICLR 2025
A APPENDIX
TABLE OF CONTENTS
A. 1 SRL Algorithm
A. 2 Hyperparameters
3. Implementation Details
19
A. 4 Practical Considerations on Low- Compute Hardware
20
A. 5 Learning Curves
A. 6 Plots for Frequency Averaged Scores
23
A. 7 Plots for FAS vs. Performance for Stochastic Timestep
26
8. Generative Replay Latent Space
26
A. 9 Neural Basis for Sequence Learning
A. 10 Clarification Figure
A. 11 Learning Curves by
30
A. 12 Randomized frame-skipping
A. 13 Results for TempoRL Algorithm
A.1 SRL ALGORITIIM
Algorithm 1: Sequence Reinforcement Learning
Input: 4,, vi, 1/2, w. Initial parameters
+ o, 4/1 t V1, 4/2 :2:
1/ Initialize target network weights // Initialize an empty replay pool
D + 0
for each iteration do
{at, a...ttl1.....tt ( ~ Ta (a(
// Sample action
sequence from the policy for each action de in the sequence do
8141 ~ p(8841115 a)): // Sample transition from the environment D+DUP ((st, ae, r(s,, at), s411);: // Store transition in the replay pool
end
for each gradient step do 5 0- -m CoCs
10
// Update the model parameters
for i (1, do
12
1 < 4/: \Q V.,
// Update the O-function parameters
end
14
// Sample action if iteration mod actor-nupaat frequency == then for j (I) do
{ar, Ge+l, a++J-1) Tu((ar (441,....44411)))
sequence from the policy
15
8j+1 ~ m=(sstils),aj))
// Sample transition from the
target model end
- Aa VuLw;
// Update policy weights
end
21
+ aVaL(a))
// Adjust temperature
22
for E [1,2] do
1/ <- - + (1 T)
// Update target network weights
24
end
25
- TΦ + (1 ()
// Update target model weights
26
end
27
end
Output: Φ 1/1, 1/2, w;
// Optimized parameters
18

---

Published as a conference paper at ICLR 2025
A.2 HYPERPARAMETERS
The table below lists the hyperparameters that are common between every environment used for all our experiments for the SAC and SRL algorithms:
Hyperparameter Hidden Layer Size
Value 256
description
Size of the hidden layers in the feed forward networks of Actor, Critic, Model and Encoder networks
Updates per step
Number of learning updates per one step in the environment
Target Update Interval
Inverval between each target update
0.99
Discount Factor
T
0.005
Update rate for the target networks (Critic and Model)
Learning Rate
0.0003
Learning rate for all neural networks
Replay Buffer Size
10"
Size of the replay buffer
Batch Size
256
Batch size for learning
Start Ttmee-seps
10000
Initial number of steps where random policy is followed
Table 5: List of Common hyperparameters
Environment
Timestep
Eval frequency
LunarLanderCConiinuuss
500000
2500
Hopper-v2
1000000
5000
Walker2d-v2
1000000
5000
Ant-v2
5000000
5000
HaliCheetah-v2
5000000
5000
Humanoid-v2
10000000
5000
Table 6: List of environment-speciffe hyperparameters
A.3 IMPLEMENTATION DETAILS
Due to its added complexity during training, SRL requires longer wall clock time for training when compared to SAC. We performed minimal hyperparameter search over the actor update frequency parameter on the Hopper environment (tested values: 1, 2, 4, 8, 16). All the other hyperparamters were picked be equal the SAC implementation. We also did not perform hyerparameter search over the size of GRU for the actor. It was picked to have the same size as the hidden layers of the feed forward network of the actor SAC. The neural network for the model was also picked have the same architecture as the actor from SAC, thus it has two hidden layers with 256 neurons. Similarly encoder the latent SRL implementation was also picked to have the same architecture. For the latent SRL implementation we also add an additional replay buffer to store transitions of length 5, to implement the temporal consistency training for the model. This was done for simplicity of the implementation, and it can be removed since it redundant to save memory.
All experiments were performed on a GPU cluster the Nvidia 1080ti GPUs. Each run was performed using single GPU, utilizing 8 CPU cores of Intel(R) Xeon(R) Silver 4116 (24 core) and 16GB of memory.
We utilize the pytorch implementation of SAC https:///gtthuuu.cmm/eeeeiie The official github repository for SRL is: hetps:
pytorch_ sac) (Yarats & Kostrikov (020))
7/github. com wwt.//ee012//eeeeeeeeeeeee Re neeinfee Reinfoorement: Learningnnnnn
19

---

Published as a conference paper at ICLR 2025
A.4 PRACTICAL CONSIDERATIONS ON LOW-COMPUTE HARDWARE
In this work, we utilize a GRU for action generation. However, we did not test the performance of other recurrent architectures transformers. Depending on the hardware constraints and the application, more complicated or simple architecture could be utilized. Furthermore, we also leave the exploration of actor complexity generalization to larger action sequences future work.
Autonomous agents often have observation processing before it is fed into the RL algorithm. It should be noted that observation processing often forms a significant portion of the latency while the recurrent portion of the actor for SRL governs the actuation frequency. Furthermore, mentioned before, SRL can also inherently handle delays by acting in predictive manner where the sequence of actions performed in anticipation of the next state that is being processed. Furthermore, in such cases, where there an overlap between two consecutive action sequences, additional MSE loss can be utilize to align two action sequences. We also leave this exploration to future work.
20

---

Published as a conference paper at ICLR 2025
A.5 LEARNING CURVES
Pondulum-r1
Lunarlannderoomme
Hopper-v2
Rewrad
Reward
3000
SAC-1
SAC-2
2000
SAC-2
1000
SAC-2
200
SAC-4
Avg.
1000
SAC-4
SAC-8
SAC-8
1500
4000
SAC
SAC-16
100
3000
100
1990
Training Steps
Training Steps
Trinning Steps
Walker2d-v2
Ani-v2
HallCheelah-v2
204
5000
8000
Arg. Reward
4000
Reward
SAC-1
3000
SAC-2
4000
SAC-1
1004
SAC-1
2000
SAC-4
SAC-2
SAC-2
2000
1000
SAC-8
5k
SAC-16
SAC-8
PW
198 0
2000
SAC-16
SAC-16
3M
4M
Training Steps
Training Steps
Training Steps
Humanoid-v2
InveriedPendolum-22
InverredDDoubeeeedddlas..
1200
10k
1000
80
Reward
Ang. Reward
600
SAC-1
SAC-1
SAC-1
SAC-2
SAC-2
SAC-2
SAC-4
SAC-4
200
24
SAC-16
SAC-16
SAC-16
200
PCE
WF0
RCD
WFD
109 0
6M RM
Training Steps
Training Steps
Training Steps
Reacher-v2
Swimmer-y2
300
Arg. Reward
Reward
SAC-1
200
SAC-1
SAC-2
SAC-2
SAC 4
100
SAC-16
1002
MODS
M006
VV2'
Triining Steps
Training Steps
Figure 3: Learning curves for extended action Soft-Actor Critic (SAC-J) (Haarnoja al. 2019) over continuous control tasks. The default timestep J = 1 is the optimal for all environments except the swimmer and lunar lander. Larger timesteps support better exploration but also result worse performance. These results demonstrate that on all environments except swimmer and lunar-lander, the default timestep is picked to optimize for the sweet-spot between better exploration and better performance.
21

---

Published as a conference paper at ICLR 2025
Pendulam-vi
Lunarl SataalllaanCCoottmmaaaee
Hopper-42
300
4000
200
200
Arg Rewrrn
Rewand
100
Reward
-100
SRL-2
2000
SRI-2
SRT-2
SRT-4
-200
1000
SRL-4
SRL-8
300
1000
SRI.8 8
SKL-16
SRI-16
1200
SRI-16
-400
7008
INP
10k
2016
Training
Training Sleps
Training Sleps
Walker2d-v2
Ani-v2
HallCheetabb-v2
5000
8000
1.5K
6000
Avg. Reward
Avg. Reward
3000
4000
2000
SRL-4
SRL-8
2000
SKL-4
SKL-2
5k
SKL-4
SRL-16
SRL-8
SRL-8
SKL-16
SKL-16
20000
0
1M 2M 3M 4M
0.
1M
2M 3M 4M
Training Steps
Training Steps
Training Steps
Humanoid-v2
InversedPendslumn-22
InvensdDeubkeennhllaa-
106
1200
1000
Arg. Rewerd
Reward
800
8k
64
600
6k
41
SRI -2
SRT
Avg.
2k
SRT
SRI-4
SRL-S
SRL-8
24
SRI.
SRI-16
SR1-16
MP0 INC
X30
XP0
N90
189
0 2M 4M OM SM
Training Steps
Training Steps
Training Steps
Reacher-v2
Swimmer-r2
120
Reward
Reward
SR1-4
SRT
10
SR1-8
SRL-S
SRI-16
SKI-16
1000
XPO
X90
N800
Training Steps
Training Steps
Figure 4: Learning curves of SRL-J (Haarnjja et al. 2019) over continuous control tasks. During
evaluation, SRL receives input after J primitive actions. All curves are averaged over trials, with shaded regions representing standard deviation.
22

---

Published as a conference paper at ICLR 2025
A.6 PLOTS FOR FREQUENCY AVERAGED SCORES
Figure 6 shows the plots for FAS. The ASL of in the figure represents the performance of each policy the standard reinforcement learning setting. We can see that SRL is competitive with SAC on ASL of on all environments tested. Larger H results in better robustness at longer ASLs but often comes at the cost of lower performance shorter ASLs.
Additionally, as the FAS reflects, SRL is also significantly more robust across different frequencies than standard RL (SAC).
23

---

Published as a conference paper at ICLR 2025
Pendulam-vi
Hopper-v2
200
300
3500
-400
200
3000
Arg Rewred
Arg. Rewsrd
2500
-800
2000
SAC-1
SAC-2
SAC-2
Arg.
1500
SAC-2
1000
SAC-4
-100
SAC-4
1000
SAC-4
-1200
SAC 8
SAC. 8
500
SAC 8
SAC-16
SAC-16
SAC-16
200
1000
1500
10
20
50
100
200
Action Sequenee Longth (izs)
Action Sequence Leugth
Action Sequence Longth (in)
Walker2d-v2
Ani-v2
HaaliChettab-v2
5000
7000
4000
6000
Arg. Reward
Avg. Reward
5000
Avg. Reward
3000
4000
10k
SAC-1
SAC-1
SAC-1
2000
1000
SAC-4
SAC-2
2000
1000
SAC-4
SAC-2
5k
SAC-4
SAC-2
SAC-8
SAC-8
SAC-8
SAC-16
-1000
SAC-16
SAC-16
100
200
500
1000
1500
500
1000
1500
Action Sequence Length (ms)
Action Sequence Length (ms)
Action Sequence Length (ms)
Humanoid-v2
InversedPondslun--22
Inversd/Douhceenddulnnn-
10k
1000
8000
800
Arg. Rewrrd
Arg. Reward
Arg. Reward
6000
600
6k
SAC-1
SAC-1
SAC-1
SAC-2
SAC-2
4k
SAC-2
SAC-4
SAC-4
SAC-4
2000
SAC 8
200
SAC 8
2k
SAC 8
SAC-16
SAC-16
SAC-16
200
400
500
1000
500
1000
1500
Action Sequetoce Length (ms)
Action Sequence Length (ms)
Action Soquence Length (ms)
Reacher-v2
S Swimmer-v2
10
250
Arg. Reward
Reward
-15
200
SAC-1
SAC-1
1.5
-20
SAC-2
SAC-2
SAC-4
100
SAC-4
-25
SAC s
SAC
20
SAC-16
SAC-16
0
200
400
600
500
1000
Action Sequenee Length (ms)
Action Sequenee Length (ms)
Figure 5: Performance of SAC-J at different Action Sequence Lengths (ASL). SAC repeats the same action for the duration. All policies were tested on ASI. of 1, 2, 4, 8 ... 30. All markers are averaged over 5 trials, with the error bars representing standard error.
24

---

Published as a conference paper at ICLR 2025
Pendulun-vi
SummallaaderConttnnen
Hopper-42
-00
300
3500
-400
3000
Arg. Rewrad
Arg. Reward
2500
-600
1001
2000
800
SRT-2
SRT-2
Arg.
1500
SRL-4
SRL-4
1000
SRI-4
100
SRI-8 8
SRL
500
SRI.
-1200
SRI-16
-700
SRI-16
SRI-16
500
1000
1200
10
20
30
100
200
Action Sequence Length (ms)
Action Sequeniee Length
Action Sequenee Length (mu))
Walker2d-v2
Ani-v2
HaalChhettab-v2
5000
7000
6000
Are. Reward
Ang. Reward
5000
Avg. Reward
3000
4000
10k
3000
2000
SRL-2
SRL-2
SRL-4
2000
SKL-4
5k
SKL-4
1000
SRL-8
1000
SRL-S
SRL-8
SKL-16
SKL-16
SKL-16
100
200
500
1000
1500
500
1000
1500
Action Sequence Length (ms)
Action Sequence Length (ms)
Action Sequence Length (ms)
Humanoid-v2
InversedPendhlun-222
Inversd/Douhheennaltnnn-
10k
Ang. Rewerrd
Reward
Arg. Rewar
600
6k
43
SRT-2
SRT
4k
SRT-2
SRI-4
SRL-4
SRI-4
SRI. 8
200
SRI.
SRI. 8
0-
SRT-16
SRT-16
200
400
500
1000
500
1000
1300
Action Sequeace Length (1z8)
Action Sequenoe Length (1z))
Action Sequenee Length (un)
Reacher-v2
Swimmer-r2
Avg. Rewarrd
Arg. Reward
90
80
SRT->
SRT
SRI-4
SRI-4
-10
SRI. 8
SRI.
30
200
400
600
500
1000
Action Sequenee Length (ms)
Action Sequeniee Length (ms)
Figure 6: Performance of SRL-J at different Action Sequence Lengths (ASL). All policies were tested on ASL 1, 2, 4, 8 30. All markers are averaged over trials, with the error bars representing standard error.
25

---

Published as a conference paper at ICLR 2025
A.7 PLOTS FOR FAS Vs. PERFORMANCE FOR STOCHASTIC TIMESTEP
In Figure 7 we present the plots for FAS vs performance for all environments. For all environments except InvertedDoubeePendulun difficult problem at slow frequency and demonstrates poor performance of less than 200, thus it does not correlate FAS.
we see a high correlation. InverteiDoubeeeendttun- is
LunderL 2
Hopper-v2
Walksr2d-v2
300
20000
3000
250
Reward
2011
Reward
2500
Avg. Reward
1500
2000
1200
1000
1000
50
-50
0.5
0.2
0.4
0.1
0.2
0.3
FAS
FAS
FAS
Ant-v2
HoleCheetah--v2
Inverreedeendalum-
5000
100
5000
4000
Reward
3000
Ang. Reward
Reward
3000
2000
2000
40
1000
1000
20
-1000
0.2
0.4
0.6
0
0.2
0.1
0.2
0.3
FAS
FAS
FAS
Invers:DDebbePnnuulun2
Swimmer-r2
Reacher-v2
Pranse 0:21
140
Prinoc
Perron n
180
160
120
10
Arg. Rewrrd
140
Reward
100
Arg Reward
120
4-
-15
80
-20
40
TAS
Figure 7: Performance vs. FAS of different policies (SAC, SRL-2, SRL-4, SRL-8, SRL-16). For each algorithm, we test 5 policies over 10 episodes.
A.8 GENERATIVE REPLAY IN LATENT SPACE
Previous studies have shown that generative replay benefits greatly from latent representations Van de Ven et al., 2020). Recently, Simplified Temporal Consistency Reinforcement Learning (TCRL) (Zhao et al. 2023) demonstrated that learning latent state-space improves not only model-
based planning but also model-free RL algorithms. Building on this insight, we introduced an en- coder to encode the observations in our algorithm.
Following the TCRL implementation, we use two encoders: an online encoder eg and a target en- coder eg-, which the exponential moving average of the online encoder:
Encoder: e = Co (st)
(6)
26

---

Published as a conference paper at ICLR 2025
Walker2d-v2
Walker2d-v2
5000
5000
Avg. Retum
4000
Avg. Reward
4000
3000
SAC
3000
SRL 2
2000
2000
SRL 16
SRL 4
SAC
1000
SRL 8
1000
TempoRL
SRL 16
0.2
0.4
0.6
0.8
100
200
Million Steps
Action Sequence Length (ms)
Figure 8: Left: Learning curve of SRL with latent state-space on the Walker2d-v2 environment. Right: Performance of latent SRL-16 on different ASL, compared to SAC and TempoRL. Utilizing a latent representation for state space is especially beneficial for the Walker2d environment that outperforms SAC even when training upto sequence lengths = 16.
Thus, the model predicts the next state in the latent space. Additionally, we introduce multi-step model prediction for temporal consistency. Following the TCRL work, we cosine loss for model prediction. The model itself predicts only a single step forward, but we enforce temporal consistency by rolling out the model H-steps forward predict 1++12+1+
Specifically, for an H-step trajectory = (2t, at, 2+1))t++ drawn from the replay buffer D, we the online encoder get the first latent state (or). Then conditioning on the sequence of actions d the model is applied iteratively to predict the latent states ēt④ mo az). Finally, the target encoder calculate the target latent states 1:41:2444 Co- (0++12414 The Loss function is defined as:
H Σ h=
beth lec
ēith (1eettt)
Ca.o = EYND
-
We set 5 for our experiments. Both the encoder and the model are ford-forward neural net- works with two hidden layers.
We provide preliminary results for the Walker environment. Utilizing the latent space for generative
replay significantly improved performance, making competitive even at 16 steps (128ms) (Figure 8).
We also provide the TempoRL (Biedenkapp al. 2021 algorithm as a benchmark it is al- gorithm that successfully reduces the number of decisions per episodes. TempoRL is designed to dynamically pick the best frameskip (for performance), therefore we report the avg. action sequence length for TempoRL.
A.9 NEURAL BASIS FOR SEQUENCE LEARNING
Unlike artificial RI. agents, learning in the brain does not stop once an optimal solution has been found. During initial task learning, brain activity increases as expected, reflecting neural recruit- ment. However, after training and repetition, activity decreases cient representations of the action sequence, commonly referred to as muscle memory (Wiestler & Diedrichsen 2013). This phenomenon is further supported by findings that sequence-speche activ- ity in motor regions evolves based on the amount of training, demonstrating skill-specific efficiency and specialization over time (Wymbs & Grafton 2015).
the brain develops more effi-
The neural basis for action sequence learning involves a sophisticated interconnection of different brain regions, each making distinct contribution:
27

---

Published as a conference paper at ICLR 2025
1. Basal ganglia (BG): Action chunking is cognitive process by which individual actions grouped into larger, more manageable units or "chunks," facilitating more efficient storage, retrieval, and execution with reduced cognitive load (Favila et al. 2024). Importantly, this mechanism allows the brain to perform extremely fast and precise sequences of actions that would be impossible if produced individually. The BG plays crucial role in chunking. encoding entire behavioral action sequences as a single action (Jin et al. 20144, Favila et al., 2024 Jin & Costa 2015 Berns Sejnowski 1996 1998 Garr 2019). Dysfunction in the BG 18 associated with deficits in action sequences and chunking in both animals (Doupe) et al., 2005 Jin & Costa, 2010 Matamales al., 2017) and humans (Phillips et al. [995]
Boyd et al. 2009 Favila et al., 2024p. However, the neural basis for the compression of individual actions into sequences remains poorly understood.
2. Prefrontal cortex (PFC): The PFC is critical for the active unbinding and dismantling of action sequences to ensure behavioral flexibility and adaptability (Geissler et al. 2021). This suggests that action sequences are not merely learned through repetition; the PFC modifies these sequences based on context and task requirements. Recent research indicates that the PFC supports memory elaboration (Immink et al.) (20TT) and maintains temporal context information (Shahnazian et al. 2022 in action sequences. The prefrontal cortex receives inputs from the hippocampus.
3. Hippocampus (HC) replays neuronal activations of tasks during subsequent sleep at speeds six to seven times faster. This memory replay may explain the compression of slow actions into fast chunks. The replayed trajectories from the HC are consolidated into long- term cortical memories (Zielinski et al. 2020, Malerba et al. 2018). This phenomenon extends to the motor cortex, which replays motor patterns at accelerated speeds during sleep (Rubin et al. 2022).
28

---

Published as a conference paper at ICLR 2025
A.10 CLARIFICATION FIGURE
Computation Frequency
Observation Frequency
Actuation Frequency
Frequency: f2 fi
f1 Hz
fi Hz
f1 Hz
Observation
Computation π(s, a)
Actuation
Standard RL
f2 Hz
fi Hz
f2 Hz
Observation
Computation TTss, (1, a2 a3 ...)
Actuation a1 a2 az, ...
Sequence Learner
Figure 9: Illustration of the control process in RL agent, comprising three key components: observation, computation, and actuation. In a standard RL framework, these components typically operate at the same frequency, with each observation leading single action after computation pass. However, the sequence learner can achieve faster actuation by generating multiple primitive actions per observation. It's important note that during training, the observation frequency must be at least equal to the actuation frequency and, after training, must match the computation frequency.
29

---

Published as a conference paper at ICLR 2025
A.11
LEARNING CURVES BY J
Powr Amvv
Tras
Thildin
Figure 10: Learning curve of SRL-2 and SAC-2.
CANE
Figure 11: Learning curve of SRL-4 and SAC-4.
24
4.22 sob
GLM K-N CAM
234
C2M CAME GU02 COM
10c 30c X0E AODE
4.25 CAN RCMM
Figure 12: Learning curve of SRL-8 and SAC-8.
30

---

Published as a conference paper at ICLR 2025
RANN Thilking Darry
To Phays
Thaliin farm
Tha
The
250
Figure 13: Learning curve of SRL-16 and SAC-16.
12 RANDOMIZED
As shown, SAC trained on a constant timestep cannot adapt to different timesteps. For a fairer comparison, also present results randomized frame-skipping implemented on SAC during training.
Pendulum-vl
Hopper-v2
800
300
-900
Reward
Reward
-1100
1000
-1200
SAC-16
f
SAC-16
500
SAC-16
-1400
SAC-R-16
-100
SAC-R-16
SAC-R-16
500
1000
20
Action Sequence Length (ms)
Action Sequence Length
Action Sequence Length (ms)
Figure 14: Performance of SAC and randomized SAC (SAC-R).
Figure comopprss the performance of randomized SAC (SAC-R) to SAC J 16. Surprisingly. we find that randomized frame-skipping during training improves the performance at shorter action sequence lengths (ASL) for simple environments like pendulum and lunar lander. However, for Hopper, SAC-R performs worse than SAC. This is most probably due to the stochasticity introduced due to the randomized frame-skipping. Even with randomized frame- trame-skipping, SAC fails to achieve performance similar to SRL simple environments, thus further reinforcing the results presented in this paper.
A.13 RESULTS FOR TEMPORL ALGORITHM
To further provide provide context for the contribution of this work in comparison to previous work, we provide further comparison to TempoRL (Biedenkapp et al. 2021) and also discuss performance compared to recent work on observational dropout.
TempoRL cannot be adapted the FAS setting since after each action is picked, it further picks the duration for the amount of time the action will be performed. Yet, since it promotes action repetiton, it results in lower decision frequency and longer action sequence lengths than standard algorithms like TD3 and SAC.
Table 7 demonstrates the results of training TempoRL algorithm some of the benchmarks pre- sented in this paper. We did quick hyperparameter search over the max sequence length parameter and pick the highest number over 3 that did not result in significant drop in performance. We
31

---

Published as a conference paper at ICLR 2025
Environment
Avg. Reward
Avg. Sequence Length
Max sequence Length
Pendulum
-44.338 #31.26
71.74ms
6
Hopper
2607.86 =342.23
22. 4ms
9
Walker2d
4581.69 1561.95
25.54ms
Ant
3507.85 +559..5
62.66ms
HalfCheetah
6627.73 =2500.77
56.20ms
Inv Pendulum
984.21 =47.37
73.92ms
10
InvD Pendulum
9352.61 +22
58.76ms
Table 7: Results of running TempoRL on Mujoco Tasks, All results are averaged over 10 seeds.
find that while TempoRL achieve optimal performance on environments with single dimensions like pendulums, it demonstrates significant drop in performance on environments with multiple dimen- sions like Ant and HalfCheetah. Furthermore, on all environments, it maintains relatively short action sequence length and even though given the option of picking long action sequences, rarely does so. This result further demonstrates the contribution of SRL at maintaining performance at really long sequence lengths in environments with high action dimensions.
32


Reviews:
[{"review_id": "N8CDLWlShl", "reviewer": "Reviewer_YC71", "summary": "This paper proposes a new model-based RL method that enables a lower frequency for action decision-making, slower than the frequency of sensing and execution. To accomplish this, the proposed method, Sequence RL (SRL), predicts a sequence of actions at each decision step, allowing for a lower decision frequency while employing a high-frequency dynamic model learner and action-value estimator. This setup mimics biological behavior, where the brain’s computation frequency is lower than that of sensing and actuating processes. Additionally, the paper introduces a new metric, FAS, to assess an RL method’s robustness and adaptability across different frequency settings, demonstrating that SRL achieves a high FAS score.", "questions": "Please address the issues outlined in the weaknesses section. The effectiveness of the proposed method is significantly limited by the low quality of the experiments, particularly the empirical results. It is essential to include extensive comparisons with related works to more convincingly demonstrate the method’s effectiveness, beyond merely relying on the proposed frequency metric.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The research motivation of the paper, specifically the comparison of decision patterns and frequencies between RL and humans, is compelling.\n- The paper establishes a strong connection to biological fundamentals, providing relevant examples and insights throughout.\n- The main idea—designing an RL algorithm inspired by biological principles, where each component operates at different frequencies—is novel, and the storyline leading up to the experiment section is smooth and easy to follow.", "weaknesses": "## Lack of related works\n- This paper primarily focuses on its connection to biological insights and motivation but overlooks relevant efforts in the RL literature addressing frame-skipping, action repetition, long-horizon exploration, and action correlations. Several studies, such as [1,2,3,4], have explored similar topics from different perspectives. Although these works are not biologically motivated, their contributions are highly relevant to this paper and should not be ignored.\n\n- The way of using GRUs in sequence action generation is very similar to the work in [4], please consider adding discussion.\n\n## Technical Issues\n- From line 252 to line 259, it is unclear that why using max entropy RL techniques, such as SAC, can address the issue of additive noise and automatically lower entropy for deeper actions arther from the observation.  \n\n- Inproper declaration from line 292 to 296. There are several works that focus on credit assignment for action sequences, such as [2, 5].\n\n- Does the current work fully address the issues listet in line 93? Namely the sparse rewards, jerky control, high compute cost, and catastrophic failure due to missing inputs. If not, it is better to remove this misleading content. \n\n## Poor Experiment Quality (Critical Issue for this Paper)\n- The empirical results only include SAC as a baseline method, which was introduced in 2018. This is clearly insufficient. All the methods mentioned above have demonstrated advantages over SAC by employing different techniques for modeling action sequences or predicting temporally correlated actions. These approaches should theoretically achieve good FAS scores as well. Please consider including some of these methods to enrich the experiments and enhance the technical rigor of the paper.\n\n- Even with SAC as the only baseline, the proposed method, SRL, only outperforms SAC in 6 out of 10 tasks. Despite its advantage in the proposed FAS metric, SRL's performance over SAC is marginal and unconvincing.\n\n- In 5 out of 10 tasks (InvPendulum, Hopper, InvDPendulum, Reacher, and Swimmer), the Online Planning method outperforms SRL, contradicting the description in the caption of the \"second Table 3\" on page 9.\n\n- Many important results are placed in the appendix, while the main paper is disproportionately occupied by the introduction to biological concepts.\n\n## Excessive Discussion of Biological Concepts\n- Section 6 should be either removed entirely or moved to the appendix, as it detracts from the paper’s main focus on learning representation rather than biology.\n\n## Minor issues\n- It is better to use different math symbols to distinguish the macro action and per-step action, especially in equation 1. Both sides used $a_{t'}$\n- There are two \"Table 3\" captions in the paper: one on page 8 and another on page 9.\n- The layout of figures and text in the paper is of low quality and requires adjustments and proofreading. For example, the title of Section 6 is oddly placed.\n\n## References\n[1] Raffin, Antonin, Jens Kober, and Freek Stulp. \"Smooth exploration for robotic reinforcement learning.\" Conference on robot learning. PMLR, 2022.\n\n[2] Li, Ge, et al. \"Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning.\" ICLR 24.\n\n[3] Chiappa, Alberto Silvio, et al. \"Latent exploration for reinforcement learning.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[4] Zhang, Haichao, Wei Xu, and Haonan Yu. \"Generative planning for temporally coordinated exploration in reinforcement learning.\" ICLR 22.\n\n[5] Ni, Tianwei, et al. \"When do transformers shine in rl? decoupling memory from credit assignment.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K0Z29DdgdK", "reviewer": "Reviewer_thGh", "summary": "This model introduces Sequence Reinforcement Learning (SRL), a framework inspired by biological learning agents that allows prediction of action sequences of variable length. The method builds on the Soft-Actor-Critic algorithm, comprising a policy and a state-action-value function. Additionally, a model of the environment is trained and used for updating the policy. The policy includes a GRU RNN allowing the computation of arbitrary length action sequences starting from a single observation. While the model and the critic are trained on truly experienced trajectories only, training of the policy is done by predicting intermediate states using the model and the critic for assigning state-action-values. The method is evaluated on several continuous control tasks. The authors introduce a metric they call Frequence-Averaged Score (FAS) which is defined as the area under the curve of the reward vs decision frequency plot, and find that the SRL framework achieves significantly higher FAS-scores compared to the SAC baseline. They show that this metric is useful in predicting the average reward achieved on a randomized timestep version of the environment, arguing that a policy trained with SRL is better equipped for bridging the sim-to-real gap. The paper ends with a comprehensive comparison to structures found in the mammalian brain.", "questions": "1. Like SAC, SRL is using batched experience replay. Can you elaborate on how this interferes with biological plausibility of the method?\n2. What is the $\\alpha$ that shows up in eq. 3? What does the $\\alpha \\log \\pi$ term represent?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The introduced SRL method is explained well and is easy to understand while also showing significant improvement in the introduced FAS score.  Experimental evaluation is good and the usefulness of the FAS score is adequately demonstrated. The proposed SRL framework is well motivated using recent neuroscientific discoveries.", "weaknesses": "Demonstrating the improvement in sim-to-real transfer would add to the quality of the paper.\nLikewise, including methods that use action repetition and macro-actions could be an interesting addition.\n\nMinor mistakes:\n- Line 104: demonstrate**s**\n- Line 376: performance of the policy **in** when the frequency is not constant\n- Line 535: we introduce**s** the Frequency-Averaged-Score (FAS) metric\n\nFinally, I can't say that I fully agree with the statement \"simple tasks like walking can be performed without input states if learned properly\". Biological agents also use a range of inputs to maintain gait.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "79Y9eV9Kjj", "reviewer": "Reviewer_Y33V", "summary": "This paper proposes Sequence Reinforcement Learning (SRL) as a method to lower effective control frequency by predicting multi-step open-loop action sequences. The action sequences are optimized on model rollouts at the given low-level frequency, while the policy only needs to be queried at a lower rate during deployment. The policy can then be deployed over the full open-loop horizon, or re-queried at intermediate points, where total reward obtained across action sequence lengths favors longer SRL variations over SAC.", "questions": "- What variations of SRL are included in “all the policies” in line 374? If e.g., SRL-2 is included how would it provide open-loop actions for 16 steps if 16 is sampled from the uniform distribution in line 376?\n- The setting on Figure 2 when considering the left-most action sequence length appears similar to an MPC setting for the SRL-J agents, where performance over model-rollouts is learned but only the first action is executed before predicting a new sequence. Could you elaborate on the performance delta of SRL-8/16 to SAC? Is this due to poor model fit?\n- Could you elaborate on what you mean by the following sentence: “The SAC algorithm addresses this by maximizing the entropy of each action in addition to the expected return, allowing our algorithm to automatically lower entropy for deeper actions farther from the observation.” How does this lower the entropy of “deeper” actions?\n- Is there a good argument of why Gym-v2 versions of the tasks were chosen?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper investigates an interesting problem, considering learning of temporally consistent open-loop action sequences that require fewer queries to the policy during deployment. While the approach is reliant on the quality of the underlying learned model, this enables learned MPC-like control at one end (deterministic 1-step) while also accommodating stochastic sampling periods by using open-loop actions until the next query is possible. The authors evaluate the method across several environments, and compare across 5 seeds against the strong SAC baseline. The paper is generally well-written and clearly motivated.", "weaknesses": "- The selection of baselines and or environments should be expanded, as the paper makes a quantitative argument of performance. Other model-free agents (e.g. D4PG, etc.) as well as model-based agents would help to better put the results into perspective. It could furthermore be interesting to run these experiments on the DeepMind Control suite as well, as Gym and DMC tasks have interesting differences in their underlying dynamics and resulting agent performance.\n- It would be very informative to see how SAC performs when learning actions at the same rates as SRL-J, e.g. by applying action repeats of J steps. If “SAC-J” would significantly underperform SRL-J, it would provide a stronger argument for adding complexity via model learning.\n- Potential inspiration for additional real-world-inspired environment modifications can be found in [1]\n- I partially agree with the statement in line 334 that SRL-J is at a disadvantage. However, the agent still uses access to J=1 information via model and critic learning and so the learning curves do provide an accurate comparison across time steps.\n- Instead of a learned model, it would be interesting to see an ablation with the ground-truth dynamics to determine how well SRL-J works w/o this “handicap” (ideal model)\n- The sentence starting in line 160 highlights that SRL has a model complexity of zero after training. It should be made more clear that this actually applies to many MBRL algorithms (e.g. references in lines 138-143).\n- Line 204: consider providing a brief summary of the insights of Section 6 here, or moving Section 6 forward. Otherwise it’s difficult to follow insights that have not been presented, yet.\n- Sentence repetition in abstract lines 17-19\n- Minor: line 49 —> missing period; line 265 —> model not subscripted\n\n[1] G. Dulac-Arnold, N, Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. \"An empirical investigation of the challenges of real-world reinforcement learning.\" arXiv, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k1MK2mjpy9", "reviewer": "Reviewer_pfCy", "summary": "The paper argues that the application of current reinforcement learning (RL) methods to real-world problems is hampered by the assumption that the actor can be executed at a high frequency (matching that of the actuators). Depending on the architecture of the policy, this can lead to a high computational burden. The authors take inspiration from action chunking observed in the brain to propose an RL algorithm that directly predicts action sequences (or macro actions). After training, it can thus run at slower frequencies than the environment MDP.  Experiments on several simulated continuous control benchmarks demonstrate good performance when predicting macro actions at frequencies lower than that of the MDP, as well as for variable stochastic time steps. The paper furthermore discusses connections of macro actions to neuroscience.", "questions": "* In line 098 the claim “Our model learns open-loop control utilizing a slow hardware and low attention, and hence also low energy.” is made. Why should that be true?\n* In line 150 the claim that online planning is not feasible in robotics or the brain is made. In my experience this is not true. How did the authors come to this conclusion?\n* What is meant by “reducing actor sample complexity” in line 028 in the abstract? Sample complexity usually refers to the amount of transitions needed for training.\n* In the paragraph starting in line 050, the authors compare the performance of RL agents on MuJoCo tasks with the performance of humans on complex tasks and relate to reaction times. Why would these settings be comparable as the embodiment, action representation, and tasks are completely different?\n* In table one the < sign in the last row should be >.\n* What is meant by online planning in the paper? And how is it separate from MPC? It is not clear to me what online planning means.\n* In line 054 there seems to be a typo, a redundant “the shows”.\n* In line 169 the statement that MPC is limited to systems that have already been modeled accurately is made. However, algorithms like TD-MPC2 use MPC successfully with (partially) learned models.\n* The next paragraph states that MPC requires very short timesteps to work. However, that many robotics papers use high frequencies does not mean that it is strictly required. For example, [1] does MPC over skills which has a very low frequency.\n* The training curves in the appendix are plotted on top of each other such that it is almost impossible to make out some of the algorithms. Would it be possible to improve these plots?\n* Would it be possible to visualize the action sequences predicted by SRL compared to those of SAC?\n* In line 516 the claim “In deterministic environments, a capable agent should achieve infinite horizon control for tasks like walking and hopping from a single state.” is made. This is infeasible in environments with complex, possibly unstable dynamics since errors do not disappear but can instead compound and blow up. I would recommend revising this paragraph.\n\n[1] Shi, Lucy Xiaoyang, Joseph J. Lim, and Youngwoon Lee. \"Skill-based model-based reinforcement learning.\" arXiv preprint arXiv:2207.07560 (2022).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper relates problems that robotics researchers encounter in a creative way with insights from neuroscience. The issue of high computational demands of modern RL and imitation learning algorithms is quite relevant for the robotics community. The writing is furthermore clear and easy to understand. The proposed algorithm is simple, and easy to understand and implement.", "weaknesses": "The paper shifts its motivation from real-time control for robotics to biological plausibility over the course of the paper. The requirements of robotics and biological plausibility are not aligned well, however. Running policies at high frequencies is not necessarily an issue in robotics as long as the computational resources are sufficient. Furthermore, RL algorithms like Soft Actor Critic (SAC) usually do not aim at being biologically plausible, and the use of backpropagation through time in SRL could be viewed as not biologically plausible.\n\nIn particular, the small MLP policies trained with SAC should run at about 1 ms on modern hardware (or thereabouts). What is more, even when running them with an action repeat corresponding to a policy execution frequency of 50 Hz, the resulting policies usually perform fine [1]. Hence, the computational load of the experimental set up does not pose a challenge to modern hardware and is therefore not well aligned with the motivation. \n\nWhat is furthermore absent from the discussion of the results is actual inference times. According to the abstract, reducing the time it takes to produce an action is a main motivation for the paper. However, it is entirely unclear why computing the first action of the macro action should be faster with the proposed algorithm than with vanilla SAC. Hence, it is unclear which advantage the algorithm in its current form offers in a real-world scenario where primitive actions have to be supplied at a fixed frequency to the actuators. It might be possible to run at a higher frequency if a delay is introduced to account for the time needed to calculate the first action of a macro action. Discussing this trade off would be interesting.\n\nThe motivation of the paper moreover hinges on the claim that existing methods cannot deal well with low control frequencies in the presence of multiple action dimensions (line 197). In my opinion, this claim is unsubstantiated for two reasons: (i) There is a complete lack of baselines targeted at low control frequencies in the experiments. At the very least showing results for SAC with action repeat is an absolute necessity to put the results into perspective, in my opinion. (ii) The two prior works mentioned (FiGAR and TempoRL) do have experiments on continuous control environments with multiple action dimensions in which they do achieve competitive performance. They should also be included as baselines (TempoRL makes an appearance in the appendix and performs well but is not shown in the main text). The evaluation of SAC is furthermore entirely unfair as it was trained without any action repetition but is then evaluated at a substantially lower frequency. This cannot work and is not a relevant baseline (unlike SAC with action repeat, training with randomized time steps, FiGAR, and TempoRL). For these reasons, I think the experimental results lack relevant baselines and cannot be evaluated properly without them. I would suggest adding these baselines (in the main text).\n\nThe field of hierarchical RL is only mentioned in passing in the paper but offers many relevant approaches to easing the computational requirements for control. A higher level (or manager) modulates the lower level (or worker) which produces primitive actions. As the worker may have a simpler architecture, it can run at a higher frequency, while the manager runs at a lower frequency. The worker can be both open loop or closed loop. In practice, simple PD controllers are often used in conjunction with low-frequency policies in robotics and often provide good results. Discussing the hierarchical RL literature to some extent therefore seems necessary. \n\nIn line 055 the claim “When RL agents are constrained to human-like reaction times, even state-of-the-art algorithms struggle to perform in simple environments.” is made. This needs a citations or experiments to back this up.\n\n[1] Allshire, Arthur, et al. \"Transferring dexterous manipulation from gpu simulation to a remote real-world trifinger.\" 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)