Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control

## Abstract

Abstract Reinforcement learning (RL) is rapidly reaching and surpassing human-level control capabilities. However, state-of-the-art RL algorithms often require timesteps and reaction times significantly faster than human capabilities, which is impractical in real-world settings and typically necessitates specialized hardware. Such speeds are difficult to achieve in the real world and often requires specialized hardware. We introduce Sequence Reinforcement Learning (SRL), an RL algorithm designed to produce a sequence of actions for a given input state, enabling effective control at lower decision frequencies. SRL addresses the challenges of learning action sequences by employing both a model and an actor-critic architecture operating at different temporal scales. We propose a ”temporal recall” mechanism, where the critic uses the model to estimate intermediate states between primitive actions, providing a learning signal for each individual action within the sequence. Once training is complete, the actor can generate action sequences independently of the model, achieving model-free control at a slower frequency. We evaluate SRL on a suite of continuous control tasks, demonstrating that it achieves performance comparable to state-of-the-art algorithms while significantly reducing actor sample complexity. To better assess performance across varying decision frequencies, we introduce the Frequency-Averaged Score (FAS) metric. Our results show that SRL significantly outperforms traditional RL algorithms in terms of FAS, making it particularly suitable for applications requiring variable decision frequencies. Additionally, we compare SRL with model-based online planning, showing that SRL achieves superior FAS while leveraging the same model during training that online planners use for planning. Lastly, we highlight the biological relevance of SRL, showing that it replicates the ”action chunking” behavior observed in the basal ganglia, offering insights into brain-inspired control mechanisms.

## 1 Introduction

Biological and artificial agents must learn behaviors that maximize rewards to thrive in complex environments. Reinforcement learning (RL), a class of algorithms inspired by animal behavior, facilitates this learning process (Sutton and Barto, ). The connection between neuroscience and RL is profound. The Temporal Difference (TD) error, a key concept in RL, effectively models the firing patterns of dopamine neurons in the midbrain (Schultz et al., ; Schultz, ; Cohen et al., ). Additionally, a longstanding goal of RL algorithms is to match and surpass human performance in control tasks (OpenAI et al., ; Schrittwieser et al., ; Kaufmann et al., ; Wurman et al., ; Vinyals et al., ; Mnih et al., )

However, most of these successes are achieved by leveraging large amounts of data in simulated environments and operating at speeds orders of magnitude faster than biological neurons. For example, the default timestep for the Humanoid task in the MuJoCo environment (Todorov et al., ) in OpenAI Gym (Towers et al., ) is 15 milliseconds. In contrast, human reaction times range from 150 milliseconds (Jain et al., ) to several seconds for complex tasks (Limpert, ). Table 1 shows the shows the significant gap between AI and humans in terms of timestep and reaction times. When RL agents are constrained to human-like reaction times, even state-of-the-art algorithms struggle to perform in simple environments.

**Table 1: Timestep / reaction times for various benchmark environments and popular works that pit humans vs. AI.**
| Environment / Task | Timestep / Reaction Time |
| --- | --- |
| Inverted Pendulum | 40ms |
| Walker 2d | 8ms |
| Hopper | 8ms |
| Ant | 50ms |
| Half Cheetah | 50ms |
| Dota 2 1v1 (Berner et al., ) | 67ms |
| Dota 2 5v5(Berner et al., ) | 80ms |
| GT Sophy(Wurman et al., ) | 23-30ms |
| Drone Racing (Kaufmann et al., ) | 10ms |
| Humans | $<150$ms |

The primary reason for this difficulty is the implicit assumption in RL that the environment and the agent operate at a constant timestep. Consequently, in embodied agents, all components—sensors, compute units, and actuators—are synchronized to operate at the same frequency. Typically, this frequency is limited by the speed of computation in artificial agents (Katz et al., ). As a result, robots often require fast onboard computing hardware (CPU or GPU) to achieve higher control frequencies (Margolis et al., ; Li et al., ; Haarnoja et al., ).

In contrast, biological agents achieve precise and seemingly fast control using much slower hardware (biological neurons). This is possible because biological agents effectively decouple the computation frequency from the actuation frequency, allowing them to achieve high actuation frequencies even with slow computational speeds. Consequently, biological agents demonstrate robust, adaptive, and efficient control.

To allow the RL agent to observe and react to changes in the environment quickly, RL algorithms are forced to set a high frequency. Even in completely predictable environments, when the agent learns to walk or move, a small timestep is required to account for the actuation frequency required for the task, but it is not necessary to observe the environment as often or compute new actions as frequently. RL algorithms suffer from many problems such as low sample efficiency, failure to learn tasks with sparse rewards, jerky control, high compute cost, and catastrophic failure due to missing inputs. We hypothesize that this behavior level gap between RL and humans can be bridged by bridging the gap in the underlying process.

Towards that end, we propose Sequence Reinforcement Learning (SRL), a model for action sequence learning based on the role of the basal ganglia (BG) and the prefrontal cortex (PFC). Our model learns open-loop control utilizing a slow hardware and low attention, and hence also low energy. Additionally, the algorithm utilizes a simultaneously learned model of the environment during its training but can act without it for fast and cheap inference. We demonstrate the algorithm achieves competitive performance on difficult continuous control tasks while utilizing a fraction of observations and calls to the policy. To the best of our knowledge, SRL is the first to achieve this feat. To further quantify this result and set a benchmark for control at slow frequencies, we introduce the Frequency Averaged Score (FAS) and demonstrates that SRL achieves significantly higher FAS than Soft-actor-critic (SAC) (Haarnoja et al., ). Additionally, we demonstrate that on complex environments (with high state and action dimensions), SRL also beats model-based online planning on FAS. Finally, we discuss the available evidence in neuroscience that has inspired our biologically plausible algorithm.

## 2 Necessity of Sequence Learning: Frequency, Delay and Response Time

To perform any control task, the agent requires the following three components: Sensor, Processor/Computer, Actuator. In the traditional RL framework, all three components act at the same frequency due to the common timestep. However, this is not the case in biological agents that have different sensors of varying frequencies that are often faster than the compute frequency or the speed at which the brain can process the information (Borghuis et al., ). Additionally, in order to afford fast and precise control, the actuator frequency is also much faster than the compute frequency (see Figure 9 in Appendix).

It is difficult to exactly measure the speed of computation of the brain. However, one clue lies in the reaction time of humans. The reaction time is the sum of the amount of compute time and the signal transmission delay from the sensor to the processor and the processor to the actuator. It should be noted that delay and the speed/frequency of compute are independent of each other and require different solutions. For example, the brain uses predictive behavior instead of a reactive behavior. However, even with predictive behavior, it is imperative that brain has the ability to actuate at high-frequency with a neurons. This requires a different solution: muscle memory or action chunking described in Section 6. In artificial agents, implementing a similar solution will allow RL algorithms to be deployed into edge devices and cheaper and slower hardware with minimal impact on performance. Delay has been the focus of many previous works (Chen et al., , ; Derman et al., ), however, adapting to low frequency compute remains an open problem.

## 3 Related Work

### 3.1 Model-Based Reinforcement Learning

Model-Based Reinforcement Learning (MBRL) algorithms leverage a model of the environment, which can be either learned or known, to enhance RL performance (Moerland et al., ). Broadly, MBRL algorithms have been utilized to:

- 1.
Improve Data Efficiency: By augmenting real-world data with model-generated data, MBRL can significantly enhance data efficiency (Yarats et al., ; Janner et al., ; Wang et al., ).
- 2.
Enhance Exploration: MBRL aids in exploration by using models to identify potential or unexplored states (Pathak et al., ; Stadie et al., ; Savinov et al., ).
- 3.
Boost Performance: Better learned representations from MBRL can lead to improved asymptotic performance (Silver et al., ; Levine and Koltun, ).
- 4.
Transfer Learning: MBRL supports transfer learning, enabling knowledge transfer across different tasks or environments (Zhang et al., ; Sasso et al., ).
- 5.
Online Planning: Models can be used for online planning with a single-step policy (Fickinger et al., ). However, this approach increases model complexity as each online planning step requires an additional call to the model, making it nonviable for energy and computationally constrained agents like the brain and robots.

Compared to online planning, our algorithm maintains a model complexity of zero after training, eliminating the need for any model calls post-training. This significantly reduces the computational and energy requirements, making it more suitable for practical applications in constrained environments. Additionally, model-based online planning is less biologically plausible than SRL. Wiestler and Diedrichsen () demonstrated that the activations in the motor cortex reduce after skill learning, suggesting that the brain gets more efficient at performing the task after learning. In contrast, model-based online planning does not reduce in the compute and model complexity, but rather might increase in complexity as we perform longer sequences. SRL, on the other hand, has a model complexity of zero after training and thus is biologically plausible based on this observed phenomenon.

### 3.2 Model Predictive Control

Similar to model-based reinforcement learning, Model Predictive Control (MPC) utilizes a model of the system to predict and optimize future behavior. In the context of modern robotics, MPC has been effectively applied to trajectory planning and real-time control for both ground and aerial vehicles. MPC has been applied to problems like autonomous driving (Gray et al., ) and bipedal control (Galliker et al., ). However, MPC requires an accurate dynamics model of the system. This limits its practicality to systems that have already been sufficiently modeled. In contrast, SRL does not start with any knowledge of the environment.

Additionally, similar to current RL, MPC requires very fast operational timesteps for practical application. For example, Galliker et al. () implemented walker at 10 ms, Farshidian et al. () implemented a four legged robot at 4 ms and Di Carlo et al. () implemented the MIT Cheetah 3 at 33.33 ms.

### 3.3 Macro-Actions

Reinforcement Learning (RL) algorithms that utilize macro-actions demonstrate many benefits, including improved exploration and faster learning (McGovern et al., ). However, identifying effective macro-actions is a challenging problem due to the curse of dimensionality, which arises from large action spaces. To address this issue, some approaches have employed genetic algorithms (Chang et al., ) or relied on expert demonstrations to extract macro-actions (Kim et al., ). However, these methods are not scalable and lack biological plausibility.

In contrast, our approach learns macro-actions using the principles of RL, thus requiring little overhead while combining the flexibility of primitive actions with the efficiency of macro-actions.

### 3.4 Action Repetition and Frame-skipping

To overcome the curse of dimensionality while gaining the benefits of macro-actions, many approaches utilize frame-skipping and action repetition, where macro-actions are restricted to a single primitive action that is repeated. Frame-skipping and action repetition serve as a form of partial open-loop control, where the agent selects a sequence of actions to be executed without considering the intermediate states. Consequently, the number of actions is linear in the number of time steps (Kalyanakrishnan et al., ; Srinivas et al., ; Biedenkapp et al., ; Sharma et al., ; Yu et al., ).

For instance, FiGaR (Sharma et al., ) shifts the problem of macro-action learning to predicting the number of steps that the outputted action can be repeated. TempoRL (Biedenkapp et al., ) improved upon FiGaR by conditioning the number of repetitions on the selected actions. However, none of these algorithms can scale to continuous control tasks with multiple action dimensions, as action repetition forces all actuators and joints to be synchronized in their repetitions, leading to poor performance for longer action sequences.

## 4 Sequence Reinforcement Learning

Figure: Figure 1: The Sequence Reinforcement Learning (SRL) model. The SRL takes inspiration from the function of the basal ganglia (BG) (Top/Orange) and the prefrontal cortex (PFC) (Bottom/Blue). We train an actor with a gated recurrent unit that can produce sequences of arbitrary lengths given a single state. This is achieved by utilizing a critic and a model that acts at a finer temporal resolution during training/replay to provide an error signal to each primitive action of the action sequence.
Refer to caption: extracted/5937786/images/HSPmodel.png

Based on the insights presented in Section 6, we introduce a novel reinforcement learning model capable of learning sequences of actions (macro-actions) by replaying memories at a finer temporal resolution than the action generation, utilizing a model of the environment during training.

#### Components

The Sequence Reinforcement Learning (SRL) algorithm learns to plan ”in-the-mind” using a model during training, allowing the learned action-sequences to be executed without the need for model-based online planning. This is achieved using an actor-critic setting where the actor and critic operate at different frequencies, representing the observation/computation and actuation frequencies, respectively. Essentially, the critic is only used during training/replay and can operate at any temporal resolution, while the actor is constrained to the temporal resolution of the slowest component in the sensing-compute-actuation loop. Denoting the actor’s timestep as $t^{\prime}$ and the critic’s timestep as $t$, our algorithm includes three components:

$$ $\begin{split}\text{Model }&:s_{t+1}=\textbf{m}_{\phi}(s_{t},a_{t})\\ \text{Critic }&:q_{t}=\textbf{q}_{\psi}(s_{t},a_{t})\\ \text{Actor }&:a_{t^{\prime}}=a_{t^{\prime}},a_{t^{\prime}+t},a_{t^{\prime}+2t }..\sim\pi_{\omega}(s_{t^{\prime}})\end{split}$ (1) $$

We denote individual actions in the action sequence generated by actor using the notation $\pi_{\omega}(s_{t^{\prime}})_{t}$ to represent the action $a_{t^{\prime}+t}$

- 1.
Model: Learns the dynamics of the environment, predicting the next state $s_{t+1}$ given the current state $s_{t}$ and primitive action $a_{t}$.
- 2.
Critic: Takes the same input as the model but predicts the Q-value of the state-action pair.
- 3.
Actor: Produces a sequence of actions given an observation at time $t^{\prime}$. Observations from the environment can occur at any timestep $t$ or $t^{\prime}$, where we assume $t^{\prime}>t$. Specifically, in our algorithm, $t^{\prime}=Jt$ where $J>1;J\in\mathbb{Z}$.

Each component of our algorithm is trained in parallel, demonstrating competitive learning speeds.

We follow the Soft-Actor-Critic (SAC) algorithm (Haarnoja et al., ) for learning the actor-critic. Exploration and uncertainty are critical factors heavily influenced by timestep size and planning horizon. Many model-free algorithms like DDPG (Lillicrap et al., ) and TD3 (Fujimoto et al., ) explore by adding random noise to each action during training. However, planning a sequence of actions over a longer timestep can result in additive noise, leading to poor performance during training and exploration. The SAC algorithm addresses this by maximizing the entropy of each action in addition to the expected return, allowing our algorithm to automatically lower entropy for deeper actions farther from the observation.

#### Learning the Model

The model is trained to minimize the Mean Squared Error of the predicted states. For a trajectory $\tau=(s_{t},a_{t},s_{t+1})$ drawn from the replay buffer $\mathcal{D}$, the predicted state is taken from $\tilde{s}_{t+1}\sim\textbf{m}\phi(s_{t},a_{t})$. The loss function is:

$$ $\mathcal{L}_{\phi}=\mathbb{E}_{\tau\sim\mathcal{D}}(\tilde{s}_{t+1}-s_{t+1})^{2}$ (2) $$

For this work, the model is a feed-forward neural network with two hidden layers. In addition to the current model $\textbf{m}_{\phi}$, we also maintain a target model $\textbf{m}_{\phi^{-}}$ that is the exponential moving average of the current model.

#### Learning Critic

The critic is trained to predict the Q-value of a given state-action pair $\tilde{q}_{t}=\textbf{q}_{\psi}(s_{t},a_{t})$ using the target value from the modified Bellman equation:

$$ $\hat{q}_{t}=r_{t}+\gamma\mathbb{E}_{a_{t+1}\sim\pi_{\omega}(s_{t+1})_{0}}[ \textbf{q}_{\psi^{-}}(s_{t+1},a_{t+1})-\alpha\log\pi_{\omega}(a_{t+1}|s_{t+1})]$ (3) $$

Here, $\textbf{q}_{\psi^{-}}$ is the target critic, which is the exponential moving average of the critic. Following the SAC algorithm, we train two critics and use the minimum of the two $\textbf{q}_{\psi^{-}}$ values to train the current critics. The loss function is:

$$ $\mathcal{L_{\psi}}=\mathbb{E}_{\tau\sim\mathcal{D}}[(\tilde{q_{t}}_{k}-\hat{q} _{t})^{2}]\forall k\in 1,2$ (4) $$

Both critics are feed-forward neural networks with two hidden layers. It should be noted that while the actor utilizes the model during training, the critic does not train on any data generated by the model, thus the critic training is model-free and grounded on the real environment states.

#### Learning Policy

The SRL policy utilizes two hidden layers followed by a Gated-Recurrent-Unit (GRU) (Cho et al., ) that takes as input the previous action in the action sequence, followed by two linear layers that output the mean and standard deviation of the Gaussian distribution of the action. This design allows the policy to produce action sequences of arbitrary length given a single state and the last action.

A naive approach to training a sequence of actions would be to augment the action space to include all possible actions of the sequence length. However, this quickly leads to the curse of dimensionality, as each sequence is considered a unique action, dramatically increasing the policy’s complexity. Additionally, such an approach ignores the temporal information of the action sequence and faces the difficult problem of credit assignment, with only a single scalar reward for the entire action sequence.

To address these problems, we use different temporal scales for the actor and critic. The critic assigns value to each primitive action of the action sequence, bypassing the credit assignment problem caused by the single scalar reward. However, using collected state-action transitions to train the action sequence is impractical, as changing the first action in the sequence would render all future states inaccurate. Thus, the model populates intermediate states, which the critic then uses to assign value to each primitive action in the sequence.

Therefore, given a trajectory $\tau=(a_{t-1},s_{t},a_{t},s_{t+1})$, we first produce the $J$-step action sequence using the policy: $\tilde{a}_{t:t+J}\sim\pi_{\omega}(s_{t})$. We then iteratively apply the target model to get the intermediate states $\tilde{s}_{t+1:t+J-1}$. Finally, we use the critic to calculate the loss for the actor as follows:

$$ $\mathcal{L}_{\omega}=\mathbb{E}_{\tau\sim\mathcal{D}}\bigg{[}\alpha\log\pi_{ \omega}(\tilde{a}_{t}|s_{t})-\textbf{q}_{\psi}(s_{t},\tilde{a}_{t})+\sum_{j=1} ^{J}\alpha\log\pi_{\omega}(\tilde{a}_{t+j}|\tilde{s}_{t+j})-\textbf{q}_{\psi}( \tilde{s}_{t+j},\tilde{a}_{t+j})\bigg{]}$ (5) $$

## 5 Experiments

### Overview

We evaluate our SRL approach on several continuous control tasks, comparing it against the SAC baseline. Our focus is on environments with multi-dimensional actions, ranging from the simple LunarLanderContinuous (2 action dimensions) to the complex Humanoid environment (17 action dimensions). This allows us to highlight the benefits of SRL over traditional action repetition approaches. We utilize the OpenAI gym (Brockman et al., ) implementation of the MuJoCo environments (Todorov et al., ).

### Experiemental Setup

We train SRL with four different action sequence lengths (ASL), $J={2,4,8,16}$, referred to as SRL-$J$. During training, SRL is evaluated based on its $J$ value, processing states only after every $J$ actions. All hyperparameters are identical between SRL and SAC, except for the actor update frequency: SRL updates the actor every 4 steps, while SAC updates every step. Thus, SAC has four more actor update steps compared to SRL. Additionally, SRL learns a model in parallel with the actor and critic.

We presents the learning curves of SRL and SAC across 10 continuous control tasks in the appendix. We observe that SRL outperforms SAC in six out of ten tasks. Notably, SRL-16 achieves competitive performance on LunarLander, Hopper, Reacher and Swimmer tasks, showcasing the algorithm’s capability to learn long action sequences from scratch. Surprisingly, SRL also outperforms SAC in the Humanoid environment with fewer inputs and actor updates while concurrently learning a model, demonstrating the efficacy of the algorithm on environments with higher action dimensions. It should be noted that the learning curves presented for SRL-$J$ take in states every $J$ steps, thus SRL is evaluated on a disadvantage in the learning curves.

### Frequency-Averaged Score

Figure: Figure 2: Performance of SRL and SAC at different Action Sequence Lengths (ASL) for the Hopper environment. All markers are averaged over 5 trials, with the error bars representing standard error. The Frequency-Averaged Score (FAS) is calculated as the area under the curve of this plot.
Refer to caption: extracted/5937786/images/Hopper-v2_FAS.png

Transitioning from simulation to real-world implementation (Sim2Real) in control systems is challenging because deployment introduces computational stochasticity, leading to variable sensor sampling rates and inconsistent end-to-end delays from sensing to actuation (Sandha et al., ). This gap is not captured by the mean reward or return that is the norm in current RL literature. To address this, we introduce Frequency-Averaged Score (FAS) that is the normalized area under the curve (AUC) of the performance vs. decision frequency plot. Figure 2 demonstrates the plot over which FAS is calculated for the Hopper environment. We provide similar plots for all environments in the Appendix. The FAS captures the overall performance of the policy at different decision frequencies, timesteps or macro-action lengths. A High FAS indicates that the policy performance generalizes across:

- 1.
Decision frequencies
- 2.
Observation frequencies
- 3.
Timestep sizes

The Frequency Averaged Score (FAS) is developed to assess the overall performance of a single policy across multiple frequency settings, rather than evaluating algorithmic efficacy by training and testing separate policies for each individual frequency. A high FAS reflects a policy’s robustness and adaptability, indicating that it can be effectively deployed across varying frequencies without the need for retraining.

**Table 2: Mean Frequency-Averaged Score (FAS) and standard deviation for different environments with SAC and various SRL configurations (rounded to two decimals, highest value highlighted). Each value is averaged over 5 trials. We see that SRL provides a significant improvement over SAC.**
| Environment | SAC | SRL-2 | SRL-4 | SRL-8 | SRL-16 |
| --- | --- | --- | --- | --- | --- |
| Lunar Lander | 0.20 $\pm$ 0.09 | 0.12 $\pm$ 0.09 | 0.52 $\pm$ 0.11 | 0.63 $\pm$ 0.21 | 0.89 $\pm$ 0.05 |
| Hopper | 0.04 $\pm$ 0.02 | 0.09 $\pm$ 0.02 | 0.21 $\pm$ 0.04 | 0.36 $\pm$ 0.06 | 0.44 $\pm$ 0.04 |
| Walker2d | 0.07 $\pm$ 0.01 | 0.11 $\pm$ 0.02 | 0.24 $\pm$ 0.05 | 0.26 $\pm$ 0.06 | 0.21 $\pm$ 0.10 |
| Ant | -0.05 $\pm$ 0.04 | 0.05 $\pm$ 0.01 | 0.26 $\pm$ 0.09 | 0.38 $\pm$ 0.14 | 0.47 $\pm$ 0.11 |
| HalfCheetah | 0.01 $\pm$ 0.01 | 0.05 $\pm$ 0.01 | 0.12 $\pm$ 0.02 | 0.20 $\pm$ 0.01 | 0.25 $\pm$ 0.01 |
| Humanoid | 0.06 $\pm$ 0.01 | 0.07 $\pm$ 0.00 | 0.17 $\pm$ 0.02 | 0.27 $\pm$ 0.04 | 0.38 $\pm$ 0.05 |
| InvPendulum | 0.05 $\pm$ 0.02 | 0.08 $\pm$ 0.01 | 0.16 $\pm$ 0.04 | 0.18 $\pm$ 0.04 | 0.23 $\pm$ 0.06 |
| InvDPendulum | 0.02 $\pm$ 0.00 | 0.04 $\pm$ 0.01 | 0.05 $\pm$ 0.02 | 0.02 $\pm$ 0.02 | 0.02 $\pm$ 0.00 |
| Reacher | 0.65 $\pm$ 0.07 | 0.87 $\pm$ 0.01 | 0.89 $\pm$ 0.00 | 0.91 $\pm$ 0.00 | 0.92 $\pm$ 0.00 |
| Swimmer | 0.19 $\pm$ 0.05 | 0.32 $\pm$ 0.05 | 0.37 $\pm$ 0.18 | 0.31 $\pm$ 0.03 | 0.39 $\pm$ 0.16 |

Table 2 presents the Frequency Averaged Score (FAS) metric for SAC and SRL across varying action lengths. In most environments, SRL-16 demonstrates strong and consistent performance across a broad range of frequencies. However, in the Walker2d-v2 and InvertedDoublePendulum-v2 environments, SRL struggles with learning longer action sequences. We hypothesize that this challenge arises from higher modeling errors in these environments, and believe that improved environmental models could enhance these results—a direction we leave for future exploration. Notably, SAC performed poorly across all environments, underscoring the limitations of traditional RL methods in adapting to frequency changes and reinforcing the importance of our approach.

In order to further validate the utility of FAS, we test all the policies (SAC and SRL) in a stochastic timestep environment. The timestep (time until next input) is randomly chosen from a uniform distribution of integers in [1,16] after each decision. This is a more realistic setting as it tests the performance of the policy in when the frequency is not constant. Each policy is evaluated over 10 episodes with stochastic timesteps.

Figure: Figure 3: Avg. Reward vs FAS of different policies on stochastic timestep evaluations. Each marker is averaged over 10 episodes.
Refer to caption: extracted/5937786/images/Stochastic/Ant.png

Figure 3 plots the performance of each policy vs. its individual FAS. As shown, FAS and the performance on a stochastic timestep environment has a high Pearson correlation ratio of 0.95, thus validating the utility of FAS as a metric to measure generalized performance of a policy across timesteps and frequencies. We provide similar plots for all environments in the appendix.

### Comparison to Model-based Online Planning

In addition to action repetition, model-based online planning is another approach that allows the RL agent to reduce its observational frequency. However, it often requires a highly accurate model of the environment and incurs increased model complexity due to the use of the model during control. Despite these challenges, comparing SRL to model-based online planning is essential since it is useful when the actor cannot produce long sequences of actions and does not require the hyper-parameter $J$. With access to an accurate model of the environment, the agent’s performance might generalize to arbitrary ASL.

Since SRL incorporates a model of the environment that is learned in parallel, we compare the performance of the SRL actor utilizing the actor-generated action sequences against model-based online planning, where the actor produces only a single action between each simulated state.

**Table 3: Comparison of the FAS of SRL and corresponding model-based online planning policies across different environments.**
| Environment | SRL | Online Planning | State Space | Action Space |
| --- | --- | --- | --- | --- |
| Lunar Lander | 0.89 $\pm$ 0.05 | 0.79 $\pm$ 0.08 | 8 | 2 |
| Hopper | 0.44 $\pm$ 0.04 | 0.59 $\pm$ 0.19 | 11 | 3 |
| Walker2d | 0.26 $\pm$ 0.06 | 0.20 $\pm$ 0.05 | 17 | 6 |
| Ant | 0.47 $\pm$ 0.11 | 0.34 $\pm$ 0.08 | 27 | 8 |
| HalfCheetah | 0.25 $\pm$ 0.01 | 0.19 $\pm$ 0.02 | 17 | 6 |
| Humanoid | 0.38 $\pm$ 0.05 | 0.18 $\pm$ 0.03 | 376 | 17 |
| InvPendulum | 0.23 $\pm$ 0.06 | 0.63 $\pm$ 0.10 | 4 | 1 |
| InvDPendulum | 0.04 $\pm$ 0.01 | 0.10 $\pm$ 0.07 | 11 | 1 |
| Reacher | 0.92 $\pm$ 0.00 | 0.95 $\pm$ 0.00 | 11 | 2 |
| Swimmer | 0.39 $\pm$ 0.16 | 0.43 $\pm$ 0.14 | 8 | 2 |

Figure: Figure 4: Performance of SRL and model based online planning on different ASL. All markers are averaged over 5 trials. The error bars show standard error.
Refer to caption: extracted/5937786/images/HumanoidPlanning.png

Table 3 compares the FAS score SRL to online planning using the same model in online planning versus the action sequences generated by the SRL policy. We see that SRL can learn action sequences that perform better than model-based online planning in many environments. Notably, SRL performs better in environments with larger action and state space dimensions (See Figure 4 for example of Humanoid). Such environments are harder to model. Thus, SRL can leverage inaccurate models to learn accurate action sequences, further reducing the required computational complexity during training. We hypothesize that this superior performance is due to the fact that the actor learns a $J$-step action sequence concurrently, while online planning only produces one action at a time. Consequently, SRL is able to learn and produce long, coherent action sequences, whereas single-step predictions tend to drift, similar to the ”hallucination” phenomenon observed in transformer-based language models.

## 6 Neural Basis for Sequence Learning

Unlike artificial RL agents, learning in the brain does not stop once an optimal solution has been found. During initial task learning, brain activity increases as expected, reflecting neural recruitment. However, after training and repetition, activity decreases as the brain develops more efficient representations of the action sequence, commonly referred to as muscle memory (Wiestler and Diedrichsen, ). This phenomenon is further supported by findings that sequence-specific activity in motor regions evolves based on the amount of training, demonstrating skill-specific efficiency and specialization over time (Wymbs and Grafton, ).

The neural basis for action sequence learning involves a sophisticated interconnection of different brain regions, each making a distinct contribution:

- 1.
Basal ganglia (BG): Action chunking is a cognitive process by which individual actions are grouped into larger, more manageable units or ”chunks,” facilitating more efficient storage, retrieval, and execution with reduced cognitive load (Favila et al., ). Importantly, this mechanism allows the brain to perform extremely fast and precise sequences of actions that would be impossible if produced individually. The BG plays a crucial role in chunking, encoding entire behavioral action sequences as a single action (Jin et al., ; Favila et al., ; Jin and Costa, ; Berns and Sejnowski, , ; Garr, ). Dysfunction in the BG is associated with deficits in action sequences and chunking in both animals (Doupe et al., ; Jin and Costa, ; Matamales et al., ) and humans (Phillips et al., ; Boyd et al., ; Favila et al., ). However, the neural basis for the compression of individual actions into sequences remains poorly understood.
- 2.
Prefrontal cortex (PFC): The PFC is critical for the active unbinding and dismantling of action sequences to ensure behavioral flexibility and adaptability (Geissler et al., ). This suggests that action sequences are not merely learned through repetition; the PFC modifies these sequences based on context and task requirements. Recent research indicates that the PFC supports memory elaboration (Immink et al., ) and maintains temporal context information (Shahnazian et al., ) in action sequences. The prefrontal cortex receives inputs from the hippocampus.
- 3.
Hippocampus (HC) replays neuronal activations of tasks during subsequent sleep at speeds six to seven times faster. This memory replay may explain the compression of slow actions into fast chunks. The replayed trajectories from the HC are consolidated into long-term cortical memories (Zielinski et al., ; Malerba et al., ). This phenomenon extends to the motor cortex, which replays motor patterns at accelerated speeds during sleep (Rubin et al., ).

## 7 Discussion, Limitations and Future Work

We introduce the Sequence Reinforcement Learning (SRL) algorithm, a biologically plausible model for sequence learning. It represents a significant step towards achieving robust control at brain-like speeds. The key contributions of SRL include its ability to generate long sequences of actions from a single state, its resilience to reduced input frequency, and its lower computational complexity per primitive action.

The current RL framework encourages synchrony between the environment and the components of the agent. However, the brain utilizes components that act at different frequencies and yet is capable of robust and accurate control. SRL provides an approach to reconcile this difference between neuroscience and RL, while remaining competitive on current RL benchmarks. SRL offers substantial benefits over traditional RL algorithms, particularly in the context of autonomous agents such as self-driving cars and robots. By enabling operation at slower observational frequencies and providing a gradual decay in performance with reduced input frequency, SRL addresses critical issues related to sensor failure and occlusion, and energy consumption. Additionally, SRL generates long sequences of actions from a single state, which can enhance the explainability of the policy and provide opportunities to override the policy early in case of safety concerns. SRL also learns a latent representation of the action sequence, which could be used in the future to interface with large language models for multimodal explainability and even hierarchical reinforcement learning and transfer learning.

#### Future Work

We believe the SRL model contributes to both artificial agents and the study of biological control. Future work will incorporate biological features like attention mechanisms and knowledge transfer. Additionally, SRL can benefit from existing Model-Based RL approaches as it naturally learns a model of the world. In the Appendix, we demonstrate preliminary results of generative replay in the latent space. We believe that this is a promising direction to significantly improve upon the results in the paper.

In deterministic environments, a capable agent should achieve infinite horizon control for tasks like walking and hopping from a single state. This is an important research direction that is currently underexplored, as many environments are partially observable or have some degree of stochasticity. Current approaches rely on external information at every state, which increases energy consumption and vulnerability to adversarial or missing inputs. Truly autonomous agents will need to implement multiple policies simultaneously, and simple tasks like walking can be performed without input states if learned properly. Our future work will focus on extending the action sequence horizon until deterministic tasks can be performed using a single state and implementing a mechanism to dynamically pick the action sequence horizon based on context and predictability of the state. Serotonin is an important neuromodulator that has been demonstrated to signal the availability of time and resources in the brain to enable the decision on the planning horizon and the use of compute (Doya et al., ). In the future, we hope to introduce a mechanism to replicate the effect of serotonin in SRL to modulate the sequence length automatically based on context.

## 8 Conclusion

In this paper, we introduced Sequence Reinforcement Learning (SRL): a model based action sequence learning algorithm for model free control. We demonstrated the improvement of SRL over existing framework by testing it over various control frequencies. Furthermore, we introduces the Frequency-Averaged-Score (FAS) metric to measure the robustness of a policy across different frequencies. Our work is the first to achieve competitive results on continuous control environments at low control frequencies and serves as a benchmark for future work in this direction. Finally, we demonstrated directions for future works including comparison to model-based planning, generative replay and connections to neuroscience.

## Appendix A Appendix

### A.1 SRL Algorithm

Figure: Algorithm 1 Sequence Reinforcement Learning

### Hyperparameters

The table below lists the hyperparameters that are common between every environment used for all our experiments for the SAC and SRL algorithms:

**Table 4: List of Common hyperparameters**
| Hyperparameter | Value | description |
| --- | --- | --- |
| Hidden Layer Size | 256 | Size of the hidden layers in the feed forward networks of Actor, Critic, Model and Encoder networks |
| Updates per step | 1 | Number of learning updates per one step in the environment |
| Target Update Interval | 1 | Inverval between each target update |
| $\gamma$ | 0.99 | Discount Factor |
| $\tau$ | 0.005 | Update rate for the target networks (Critic and Model) |
| Learning Rate | 0.0003 | Learning rate for all neural networks |
| Replay Buffer Size | $10^{6}$ | Size of the replay buffer |
| Batch Size | 256 | Batch size for learning |
| Start Time-steps | 10000 | Initial number of steps where random policy is followed |

**Table 5: List of environment-specific hyperparameters**
| Environment | max Timestep | Eval frequency |
| --- | --- | --- |
| LunarLanderContinuous-v2 | 500000 | 2500 |
| Hopper-v2 | 1000000 | 5000 |
| Walker2d-v2 | 1000000 | 5000 |
| Ant-v2 | 5000000 | 5000 |
| HalfCheetah-v2 | 5000000 | 5000 |
| Humanoid-v2 | 10000000 | 5000 |

### A.2 Implementation Details

Due to its added complexity during training, SRL requires longer wall clock time for training when compared to SAC. We performed a minimal hyperparameter search over the actor update frequency parameter on the Hopper environment (tested values: 1, 2, 4, 8, 16). All the other hyperparamters were picked to be equal to the SAC implementation. We also did not perform a hyerparameter search over the size of GRU for the actor. It was picked to have the same size as the hidden layers of the feed forward network of the actor in SAC. The neural network for the model was also picked to have the same architecture as the actor from SAC, thus it has two hidden layers with 256 neurons. Similarly the encoder for the latent SRL implementation was also picked to have the same architecture. For the latent SRL implementation we also add an additional replay buffer to store transitions of length 5, to implement the temporal consistency training for the model. This was done for simplicity of the implementation, and it can be removed since it is redundant to save memory.

All experiments were performed on a GPU cluster the Nvidia 1080ti GPUs. Each run was performed using a single GPU, utilizing 8 CPU cores of Intel(R) Xeon(R) Silver 4116 (24 core) and 16GB of memory.

We utilize the pytorch implementation of SAC ([https://github.com/denisyarats/pytorch_sac](https://github.com/denisyarats/pytorch_sac)) [Yarats and Kostrikov, ].

### Learning Curves

Figure: (a)
Refer to caption: extracted/5937786/images/Learning_Curves/LunarLander.png

### A.3 Plots for Frequency Averaged Scores

Figure 6 shows the plots for FAS. The ASL of 1 in the figure represents the performance of each policy in the standard reinforcement learning setting. We can see that SRL is competitive with SAC on ASL of 1 on all environments tested. Larger $H$ results in better robustness at longer ASLs but it often comes at the cost of lower performance at shorter ASLs.

Additionally, as the FAS reflects, SRL is also significantly more robust across different frequencies than standard RL (SAC).

Figure: (a)
Refer to caption: extracted/5937786/images/ASLs/LunarLander.png

### A.4 Plots for FAS vs. Stochastic Timestep Performance

In Figure 7, we present the plots for FAS vs performance for all environments. For all environments except InvertedDoublePendulum-v2, we see a high correlation. InvertedDoublePendulum-v2 is a difficult problem at slow frequency and demonstrates poor performance of less than 200, thus it does not correlate to FAS.

Figure: (a)
Refer to caption: extracted/5937786/images/Stochastic/LunarLander.png

### A.5 Generative Replay in Latent Space

Figure: (a)
Refer to caption: extracted/5937786/images/WalkerLatentLC.png

Previous studies have shown that generative replay benefits greatly from latent representations [Van de Ven et al., ]. Recently, Simplified Temporal Consistency Reinforcement Learning (TCRL) [Zhao et al., ] demonstrated that learning a latent state-space improves not only model-based planning but also model-free RL algorithms. Building on this insight, we introduced an encoder to encode the observations in our algorithm.

Following the TCRL implementation, we use two encoders: an online encoder $\textbf{e}_{\theta}$ and a target encoder $\textbf{e}_{\theta^{-}}$, which is the exponential moving average of the online encoder:

$$ $\begin{split}\text{Encoder}&:e_{t}=\textbf{e}_{\theta}(s_{t})\end{split}$ (6) $$

Thus, the model predicts the next state in the latent space. Additionally, we introduce multi-step model prediction for temporal consistency. Following the TCRL work, we use a cosine loss for model prediction. The model itself predicts only a single step forward, but we enforce temporal consistency by rolling out the model $H$-steps forward to predict $\tilde{e}_{t+1:t+1+H}$.

Specifically, for an $H$-step trajectory $\tau=(z_{t},a_{t},z_{t+1})_{t:t+H}$ drawn from the replay buffer $\mathcal{D}$, we use the online encoder to get the first latent state $e_{t}=\textbf{e}_{\theta}(o_{t})$. Then conditioning on the sequence of actions $a_{t:t+H}$, the model is applied iteratively to predict the latent states $\tilde{e}_{t+1}=\textbf{m}_{\phi}(\tilde{e}_{t},a_{t})$. Finally, we use the target encoder to calculate the target latent states $\hat{e}_{t+1:t+H+1}=\textbf{e}_{\theta^{-}}(o_{t+1:t+1+H})$.
The Loss function is defined as:

$$ $\mathcal{L}_{\theta,\phi}=\mathbb{E}_{\tau\sim\mathcal{D}}\bigg{[}\sum_{h=0}^{ H}-\gamma^{h}\bigg{(}\frac{\tilde{e}_{t+h}}{||\tilde{e}_{t+h}||_{2}}\bigg{)}^{ T}\bigg{(}\frac{\hat{e}_{t+h}}{||\hat{e}_{t+h}||_{2}}\bigg{)}\bigg{]}$ (7) $$

We set $H=5$ for our experiments. Both the encoder and the model are feed-forward neural networks with two hidden layers.

We provide preliminary results for the Walker environment. Utilizing the latent space for generative replay significantly improved performance, making it competitive even at 16 steps (128ms) (Figure 8).

We also provide the TempoRL [Biedenkapp et al., ] algorithm as a benchmark as it is an algorithm that successfully reduces the number of decisions per episodes. TempoRL is designed to dynamically pick the best frameskip (for performance), therefore we report the avg. action sequence length for TempoRL.

### A.6 Clarification Figure

Figure: Figure 9: Illustration of the control process in an RL agent, comprising three key components: observation, computation, and actuation. In a standard RL framework, these components typically operate at the same frequency, with each observation leading to a single action after a computation pass. However, the sequence learner can achieve faster actuation by generating multiple primitive actions per observation. It’s important to note that during training, the observation frequency must be at least equal to the actuation frequency and, after training, must match the computation frequency.
Refer to caption: extracted/5937786/images/Frequencies.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Please address the issues outlined in the weaknesses section. The effectiveness of the proposed method is significantly limited by the low quality of the experiments, particularly the empirical results. It is essential to include extensive comparisons with related works to more convincingly demonstrate the method’s effectiveness, beyond merely relying on the proposed frequency metric.
soundness: 2
strengths: - The research motivation of the paper, specifically the comparison of decision patterns and frequencies between RL and humans, is compelling.
- The paper establishes a strong connection to biological fundamentals, providing relevant examples and insights throughout.
- The main idea—designing an RL algorithm inspired by biological principles, where each component operates at different frequencies—is novel, and the storyline leading up to the experiment section is smooth and easy to follow.
summary: This paper proposes a new model-based RL method that enables a lower frequency for action decision-making, slower than the frequency of sensing and execution. To accomplish this, the proposed method, Sequence RL (SRL), predicts a sequence of actions at each decision step, allowing for a lower decision frequency while employing a high-frequency dynamic model learner and action-value estimator. This setup mimics biological behavior, where the brain’s computation frequency is lower than that of sensing and actuating processes. Additionally, the paper introduces a new metric, FAS, to assess an RL method’s robustness and adaptability across different frequency settings, demonstrating that SRL achieves a high FAS score.
weaknesses: ## Lack of related works
- This paper primarily focuses on its connection to biological insights and motivation but overlooks relevant efforts in the RL literature addressing frame-skipping, action repetition, long-horizon exploration, and action correlations. Several studies, such as [1,2,3,4], have explored similar topics from different perspectives. Although these works are not biologically motivated, their contributions are highly relevant to this paper and should not be ignored.

- The way of using GRUs in sequence action generation is very similar to the work in [4], please consider adding discussion.

## Technical Issues
- From line 252 to line 259, it is unclear that why using max entropy RL techniques, such as SAC, can address the issue of additive noise and automatically lower entropy for deeper actions arther from the observation.  

- Inproper declaration from line 292 to 296. There are several works that focus on credit assignment for action sequences, such as [2, 5].

- Does the current work fully address the issues listet in line 93? Namely the sparse rewards, jerky control, high compute cost, and catastrophic failure due to missing inputs. If not, it is better to remove this misleading content. 

## Poor Experiment Quality (Critical Issue for this Paper)
- The empirical results only include SAC as a baseline method, which was introduced in 2018. This is clearly insufficient. All the methods mentioned above have demonstrated advantages over SAC by employing different techniques for modeling action sequences or predicting temporally correlated actions. These approaches should theoretically achieve good FAS scores as well. Please consider including some of these methods to enrich the experiments and enhance the technical rigor of the paper.

- Even with SAC as the only baseline, the proposed method, SRL, only outperforms SAC in 6 out of 10 tasks. Despite its advantage in the proposed FAS metric, SRL's performance over SAC is marginal and unconvincing.

- In 5 out of 10 tasks (InvPendulum, Hopper, InvDPendulum, Reacher, and Swimmer), the Online Planning method outperforms SRL, contradicting the description in the caption of the "second Table 3" on page 9.

- Many important results are placed in the appendix, while the main paper is disproportionately occupied by the introduction to biological concepts.

## Excessive Discussion of Biological Concepts
- Section 6 should be either removed entirely or moved to the appendix, as it detracts from the paper’s main focus on learning representation rather than biology.

## Minor issues
- It is better to use different math symbols to distinguish the macro action and per-step action, especially in equation 1. Both sides used $a_{t'}$
- There are two "Table 3" captions in the paper: one on page 8 and another on page 9.
- The layout of figures and text in the paper is of low quality and requires adjustments and proofreading. For example, the title of Section 6 is oddly placed.

## References
[1] Raffin, Antonin, Jens Kober, and Freek Stulp. "Smooth exploration for robotic reinforcement learning." Conference on robot learning. PMLR, 2022.

[2] Li, Ge, et al. "Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning." ICLR 24.

[3] Chiappa, Alberto Silvio, et al. "Latent exploration for reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

[4] Zhang, Haichao, Wei Xu, and Haonan Yu. "Generative planning for temporally coordinated exploration in reinforcement learning." ICLR 22.

[5] Ni, Tianwei, et al. "When do transformers shine in rl? decoupling memory from credit assignment." Advances in Neural Information Processing Systems 36 (2024).

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Like SAC, SRL is using batched experience replay. Can you elaborate on how this interferes with biological plausibility of the method?
2. What is the $\alpha$ that shows up in eq. 3? What does the $\alpha \log \pi$ term represent?
soundness: 3
strengths: The introduced SRL method is explained well and is easy to understand while also showing significant improvement in the introduced FAS score.  Experimental evaluation is good and the usefulness of the FAS score is adequately demonstrated. The proposed SRL framework is well motivated using recent neuroscientific discoveries.
summary: This model introduces Sequence Reinforcement Learning (SRL), a framework inspired by biological learning agents that allows prediction of action sequences of variable length. The method builds on the Soft-Actor-Critic algorithm, comprising a policy and a state-action-value function. Additionally, a model of the environment is trained and used for updating the policy. The policy includes a GRU RNN allowing the computation of arbitrary length action sequences starting from a single observation. While the model and the critic are trained on truly experienced trajectories only, training of the policy is done by predicting intermediate states using the model and the critic for assigning state-action-values. The method is evaluated on several continuous control tasks. The authors introduce a metric they call Frequence-Averaged Score (FAS) which is defined as the area under the curve of the reward vs decision frequency plot, and find that the SRL framework achieves significantly higher FAS-scores compared to the SAC baseline. They show that this metric is useful in predicting the average reward achieved on a randomized timestep version of the environment, arguing that a policy trained with SRL is better equipped for bridging the sim-to-real gap. The paper ends with a comprehensive comparison to structures found in the mammalian brain.
weaknesses: Demonstrating the improvement in sim-to-real transfer would add to the quality of the paper.
Likewise, including methods that use action repetition and macro-actions could be an interesting addition.

Minor mistakes:
- Line 104: demonstrate**s**
- Line 376: performance of the policy **in** when the frequency is not constant
- Line 535: we introduce**s** the Frequency-Averaged-Score (FAS) metric

Finally, I can't say that I fully agree with the statement "simple tasks like walking can be performed without input states if learned properly". Biological agents also use a range of inputs to maintain gait.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - What variations of SRL are included in “all the policies” in line 374? If e.g., SRL-2 is included how would it provide open-loop actions for 16 steps if 16 is sampled from the uniform distribution in line 376?
- The setting on Figure 2 when considering the left-most action sequence length appears similar to an MPC setting for the SRL-J agents, where performance over model-rollouts is learned but only the first action is executed before predicting a new sequence. Could you elaborate on the performance delta of SRL-8/16 to SAC? Is this due to poor model fit?
- Could you elaborate on what you mean by the following sentence: “The SAC algorithm addresses this by maximizing the entropy of each action in addition to the expected return, allowing our algorithm to automatically lower entropy for deeper actions farther from the observation.” How does this lower the entropy of “deeper” actions?
- Is there a good argument of why Gym-v2 versions of the tasks were chosen?
soundness: 3
strengths: The paper investigates an interesting problem, considering learning of temporally consistent open-loop action sequences that require fewer queries to the policy during deployment. While the approach is reliant on the quality of the underlying learned model, this enables learned MPC-like control at one end (deterministic 1-step) while also accommodating stochastic sampling periods by using open-loop actions until the next query is possible. The authors evaluate the method across several environments, and compare across 5 seeds against the strong SAC baseline. The paper is generally well-written and clearly motivated.
summary: This paper proposes Sequence Reinforcement Learning (SRL) as a method to lower effective control frequency by predicting multi-step open-loop action sequences. The action sequences are optimized on model rollouts at the given low-level frequency, while the policy only needs to be queried at a lower rate during deployment. The policy can then be deployed over the full open-loop horizon, or re-queried at intermediate points, where total reward obtained across action sequence lengths favors longer SRL variations over SAC.
weaknesses: - The selection of baselines and or environments should be expanded, as the paper makes a quantitative argument of performance. Other model-free agents (e.g. D4PG, etc.) as well as model-based agents would help to better put the results into perspective. It could furthermore be interesting to run these experiments on the DeepMind Control suite as well, as Gym and DMC tasks have interesting differences in their underlying dynamics and resulting agent performance.
- It would be very informative to see how SAC performs when learning actions at the same rates as SRL-J, e.g. by applying action repeats of J steps. If “SAC-J” would significantly underperform SRL-J, it would provide a stronger argument for adding complexity via model learning.
- Potential inspiration for additional real-world-inspired environment modifications can be found in [1]
- I partially agree with the statement in line 334 that SRL-J is at a disadvantage. However, the agent still uses access to J=1 information via model and critic learning and so the learning curves do provide an accurate comparison across time steps.
- Instead of a learned model, it would be interesting to see an ablation with the ground-truth dynamics to determine how well SRL-J works w/o this “handicap” (ideal model)
- The sentence starting in line 160 highlights that SRL has a model complexity of zero after training. It should be made more clear that this actually applies to many MBRL algorithms (e.g. references in lines 138-143).
- Line 204: consider providing a brief summary of the insights of Section 6 here, or moving Section 6 forward. Otherwise it’s difficult to follow insights that have not been presented, yet.
- Sentence repetition in abstract lines 17-19
- Minor: line 49 —> missing period; line 265 —> model not subscripted

[1] G. Dulac-Arnold, N, Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. "An empirical investigation of the challenges of real-world reinforcement learning." arXiv, 2020.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: * In line 098 the claim “Our model learns open-loop control utilizing a slow hardware and low attention, and hence also low energy.” is made. Why should that be true?
* In line 150 the claim that online planning is not feasible in robotics or the brain is made. In my experience this is not true. How did the authors come to this conclusion?
* What is meant by “reducing actor sample complexity” in line 028 in the abstract? Sample complexity usually refers to the amount of transitions needed for training.
* In the paragraph starting in line 050, the authors compare the performance of RL agents on MuJoCo tasks with the performance of humans on complex tasks and relate to reaction times. Why would these settings be comparable as the embodiment, action representation, and tasks are completely different?
* In table one the < sign in the last row should be >.
* What is meant by online planning in the paper? And how is it separate from MPC? It is not clear to me what online planning means.
* In line 054 there seems to be a typo, a redundant “the shows”.
* In line 169 the statement that MPC is limited to systems that have already been modeled accurately is made. However, algorithms like TD-MPC2 use MPC successfully with (partially) learned models.
* The next paragraph states that MPC requires very short timesteps to work. However, that many robotics papers use high frequencies does not mean that it is strictly required. For example, [1] does MPC over skills which has a very low frequency.
* The training curves in the appendix are plotted on top of each other such that it is almost impossible to make out some of the algorithms. Would it be possible to improve these plots?
* Would it be possible to visualize the action sequences predicted by SRL compared to those of SAC?
* In line 516 the claim “In deterministic environments, a capable agent should achieve infinite horizon control for tasks like walking and hopping from a single state.” is made. This is infeasible in environments with complex, possibly unstable dynamics since errors do not disappear but can instead compound and blow up. I would recommend revising this paragraph.

[1] Shi, Lucy Xiaoyang, Joseph J. Lim, and Youngwoon Lee. "Skill-based model-based reinforcement learning." arXiv preprint arXiv:2207.07560 (2022).
soundness: 3
strengths: The paper relates problems that robotics researchers encounter in a creative way with insights from neuroscience. The issue of high computational demands of modern RL and imitation learning algorithms is quite relevant for the robotics community. The writing is furthermore clear and easy to understand. The proposed algorithm is simple, and easy to understand and implement.
summary: The paper argues that the application of current reinforcement learning (RL) methods to real-world problems is hampered by the assumption that the actor can be executed at a high frequency (matching that of the actuators). Depending on the architecture of the policy, this can lead to a high computational burden. The authors take inspiration from action chunking observed in the brain to propose an RL algorithm that directly predicts action sequences (or macro actions). After training, it can thus run at slower frequencies than the environment MDP.  Experiments on several simulated continuous control benchmarks demonstrate good performance when predicting macro actions at frequencies lower than that of the MDP, as well as for variable stochastic time steps. The paper furthermore discusses connections of macro actions to neuroscience.
weaknesses: The paper shifts its motivation from real-time control for robotics to biological plausibility over the course of the paper. The requirements of robotics and biological plausibility are not aligned well, however. Running policies at high frequencies is not necessarily an issue in robotics as long as the computational resources are sufficient. Furthermore, RL algorithms like Soft Actor Critic (SAC) usually do not aim at being biologically plausible, and the use of backpropagation through time in SRL could be viewed as not biologically plausible.

In particular, the small MLP policies trained with SAC should run at about 1 ms on modern hardware (or thereabouts). What is more, even when running them with an action repeat corresponding to a policy execution frequency of 50 Hz, the resulting policies usually perform fine [1]. Hence, the computational load of the experimental set up does not pose a challenge to modern hardware and is therefore not well aligned with the motivation. 

What is furthermore absent from the discussion of the results is actual inference times. According to the abstract, reducing the time it takes to produce an action is a main motivation for the paper. However, it is entirely unclear why computing the first action of the macro action should be faster with the proposed algorithm than with vanilla SAC. Hence, it is unclear which advantage the algorithm in its current form offers in a real-world scenario where primitive actions have to be supplied at a fixed frequency to the actuators. It might be possible to run at a higher frequency if a delay is introduced to account for the time needed to calculate the first action of a macro action. Discussing this trade off would be interesting.

The motivation of the paper moreover hinges on the claim that existing methods cannot deal well with low control frequencies in the presence of multiple action dimensions (line 197). In my opinion, this claim is unsubstantiated for two reasons: (i) There is a complete lack of baselines targeted at low control frequencies in the experiments. At the very least showing results for SAC with action repeat is an absolute necessity to put the results into perspective, in my opinion. (ii) The two prior works mentioned (FiGAR and TempoRL) do have experiments on continuous control environments with multiple action dimensions in which they do achieve competitive performance. They should also be included as baselines (TempoRL makes an appearance in the appendix and performs well but is not shown in the main text). The evaluation of SAC is furthermore entirely unfair as it was trained without any action repetition but is then evaluated at a substantially lower frequency. This cannot work and is not a relevant baseline (unlike SAC with action repeat, training with randomized time steps, FiGAR, and TempoRL). For these reasons, I think the experimental results lack relevant baselines and cannot be evaluated properly without them. I would suggest adding these baselines (in the main text).

The field of hierarchical RL is only mentioned in passing in the paper but offers many relevant approaches to easing the computational requirements for control. A higher level (or manager) modulates the lower level (or worker) which produces primitive actions. As the worker may have a simpler architecture, it can run at a higher frequency, while the manager runs at a lower frequency. The worker can be both open loop or closed loop. In practice, simple PD controllers are often used in conjunction with low-frequency policies in robotics and often provide good results. Discussing the hierarchical RL literature to some extent therefore seems necessary. 

In line 055 the claim “When RL agents are constrained to human-like reaction times, even state-of-the-art algorithms struggle to perform in simple environments.” is made. This needs a citations or experiments to back this up.

[1] Allshire, Arthur, et al. "Transferring dexterous manipulation from gpu simulation to a remote real-world trifinger." 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.

Meta-review: This paper addresses the implicit requirement of current RL usage in real-world environments where observations and action frequencies are all assumed to be synchronized with the decision-making algorithm’s frequency, which is difficult to achieve in real time at a high frequency. The paper proposes to mitigate it by learning an open-loop action sequence control with a low decision frequency, where the action sequence is learned with a background planner. The experiments show that the proposed method achieves good performance with a low decision frequency on tasks where RL algorithms typically operate at a high decision frequency. As such, I recommend accepting this paper.

The paper seems to have missed a highly related work: Karimi et al. (2022) “Variable-Decision Frequency Option Critic.” But the claims of this paper still hold, introducing a learned model to learn the action sequence, whereas in the other paper, the open-loop control seems to be not learned.

Decision: Accept (Poster)