Title: Achieving Precise Control with Slow Hardware: Model-Based Reinforcement Learning for Action Sequence Learning
Venue: NeurIPS 2024
Accepted: no
Keywords: Action Sequence Learning

Basal Ganglia

Prefrontal Cortex

Reinforcement Learning

Model Based
Abstract: Current reinforcement learning (RL) models are often claimed to explain animal behavior. However, they are designed for artificial agents that sense, think, and react much faster than the brain, and they tend to fail when operating under human-like sensory and reaction times. Despite using slow neurons, the brain achieves precise and low-latency control through a combination of predictive and sequence learning. The basal ganglia is hypothesized to learn compressed representations of action sequences, allowing the brain to produce a series of actions for a given input. We present the Hindsight-Sequence-Planner (HSP), a model of the basal ganglia and the prefrontal cortex that operates under "brain-like" conditions: slow information processing with quick sensing and actuation. Our "temporal recall" mechanism is inspired by the prefrontal cortex's role in sequence learning, where the agent uses an environmental model to replay memories at a finer temporal resolution than its processing speed while addressing the credit assignment problem caused by scalar rewards in sequence learning. HSP employs model-based training to achieve model-free control, resulting in precise and efficient behavior that appears low-latency despite running on slow hardware. We test HSP on various continuous control tasks, demonstrating that it not can achieve comparable performance 'human-like' frequencies by relying on significantly fewer observations and actor calls (actor sample complexity).

1

Abstract

Current reinforcement learning (RL) models are often claimed to explain animal behavior. However, they are designed for artificial agents that sense, think, and react much faster than the brain, and they tend to fail when operating under human-like sensory and reaction times. Despite using slow neurons, the brain achieves precise and low-latency control through a combination of predictive and sequence learning. The basal ganglia is hypothesized to learn compressed representations of action sequences, allowing the brain to produce a series of actions for a given input. We present the Hindsight-Sequence-Planner (HSP), a model of the basal ganglia and the prefrontal cortex that operates under "brain-like" conditions: slow information processing with quick sensing and actuation. Our "temporal recall" mechanism is inspired by the prefrontal cortex's role in sequence learning, where the agent uses an environmental model to replay memories at a finer temporal resolution than its processing speed while addressing the credit assignment problem caused by scalar rewards in sequence learning. HSP employs model-based training to achieve modelfree control, resulting in precise and efficient behavior that appears low-latency despite running on slow hardware. We test HSP on various continuous control tasks, demonstrating that it not can achieve comparable performance 'human-like' frequencies by relying on significantly fewer observations and actor calls (actor sample complexity).

1 Introduction

Biological and artificial agents must learn behaviors that maximize rewards to thrive in complex environments. Reinforcement learning (RL), a class of algorithms inspired by animal behavior, facilitates this learning process (1). The connection between neuroscience and RL is profound. The Temporal Difference (TD) error, a key concept in RL, effectively models the firing patterns of dopamine neurons in the midbrain (2; 3; 4). Additionally, a longstanding goal of RL algorithms is to match and surpass human performance in control tasks (5; 6; 7; 8; 9; 10)

However, most of these successes are achieved by leveraging large amounts of data in simulated environments and operating at speeds orders of magnitude faster than biological neurons. For example, the default timestep for the Humanoid task in the MuJoCo environment (11) in OpenAI Gym (12) is 15 milliseconds. In contrast, human reaction times range from 150 milliseconds (13) to several seconds for complex tasks (14). When RL agents are constrained to human-like reaction times, even state-of-the-art algorithms struggle to perform in simple environments.

The primary reason for this difficulty is the implicit assumption in RL that the environment and the 33

agent operate at a constant timestep. Consequently, in embodied agents, all components-sensors, 34

compute units, and actuators-are synchronized to operate at the same frequency. Typically, this 35 frequency is limited by the speed of computation in artificial agents (15). As a result, robots often 36 require fast onboard computing hardware (CPU or GPU) to achieve higher control frequencies 37 (16; 17; 18). 38

39

40

41

42

43

In contrast, biological agents achieve precise and seemingly fast control using much slower hardware. This is possible because biological agents effectively decouple the computation frequency from the actuation frequency, allowing them to achieve high actuation frequencies even with slow computational speeds. Consequently, biological agents demonstrate robust, adaptive, and efficient control.

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

To allow the RL agent to observe and react to changes in the environment quickly, RL algorithms are forced to set a high frequency. Even in completely predictable environments, when the agent learns to walk or move, a small timestep is required to account for the actuation frequency required for the task, but it is not necessary to observe the environment as often or compute new actions as frequently. As a result, RL algorithms suffer from many problems such as low sample efficiency, failure to learn tasks with sparse rewards, jerky control, high compute cost, and catastrophic failure due to missing inputs.

In this work, we propose Hindsight-Sequence-Planner (HSP), a model for sequence learning based on the role of the basal ganglia (BG) and the prefrontal cortex (PFC). Our model learns open-loop control utilizing a slow hardware and low attention, and hence also low energy. Additionally, the algorithm utilizes a simultaneously learned model of the environment during its training but can act without it for fast and cheap inference. We demonstrate the algorithm achieves competitive performance on difficult continuous control tasks while utilizing a fraction of observations and calls to the policy. To the best of our knowledge, HSP is the first to achieve this feat.

2 Neural Basis for Sequence Learning

Unlike artificial RL agents, learning in the brain does not stop once an optimal solution has been found. During initial task learning, brain activity increases as expected, reflecting neural recruitment. However, after training and repetition, activity decreases as the brain develops more efficient representations of the action sequence, commonly referred to as muscle memory (19). This phenomenon is further supported by findings that sequence-specific activity in motor regions evolves based on the amount of training, demonstrating skill-specific efficiency and specialization over time (20).

The neural basis for action sequence learning involves a sophisticated interconnection of different brain regions, each making a distinct contribution:

Basal ganglia (BG): Action chunking is a cognitive process by which individual actions are grouped into larger, more manageable units or "chunks," facilitating more efficient storage, retrieval, and execution with reduced cognitive load (21). Importantly, this mechanism allows the brain to perform extremely fast and precise sequences of actions that would be impossible if produced individually. The BG plays a crucial role in chunking, encoding entire behavioral action sequences as a single action (22; 21; 23; 24; 25; 26). Dysfunction in the BG is associated with deficits in action sequences and chunking in both animals (27; 28; 29) and humans (30; 31; 21). However, the neural basis for the compression of individual actions into sequences remains poorly understood.

Prefrontal cortex (PFC): The PFC is critical for the active unbinding and dismantling of action sequences to ensure behavioral flexibility and adaptability (32). This suggests that action sequences are not merely learned through repetition; the PFC modifies these sequences based on context and task requirements. Recent research indicates that the PFC supports memory elaboration (33) and maintains temporal context information (34) in action sequences. The prefrontal cortex receives inputs from the hippocampus.

Hippocampus (HC) replays neuronal activations of tasks during subsequent sleep at speeds six to seven times faster. This memory replay may explain the compression of slow actions into fast chunks. The replayed trajectories from the HC are consolidated into long-term cortical memories (35; 36). This phenomenon extends to the motor cortex, which replays motor patterns at accelerated speeds during sleep (37).

3 Related Work 87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

3.1 Model-Based Reinforcement Learning

Model-Based Reinforcement Learning (MBRL) algorithms leverage a model of the environment, which can be either learned or known, to enhance RL performance (38). Broadly, MBRL algorithms have been utilized to:

Improve Data Efficiency: By augmenting real-world data with model-generated data, MBRL can significantly enhance data efficiency (39; 40; 41).

Enhance Exploration: MBRL aids in exploration by using models to identify potential or unexplored states (42; 43; 44).

Boost Performance: Better learned representations from MBRL can lead to improved asymptotic performance (45; 46).

Transfer Learning: MBRL supports transfer learning, enabling knowledge transfer across different tasks or environments (47; 48).

Online Planning: Models can be used for online planning with a single-step policy (49). However, this approach increases model complexity as each online planning step requires an additional call to the model, making it nonviable for energy and computationally constrained agents like the brain and robots.

Compared to online planning, our algorithm maintains a model complexity of zero after training, eliminating the need for any model calls post-training. This significantly reduces the computational and energy requirements, making it more suitable for practical applications in constrained environments. Additionally, the performance of online planning algorithms relies heavily on the accuracy of the model. In contrast, our approach can leverage even an inaccurate model to learn a better-performing policy than online planning, using the same model.

3.2 Macro-Actions

Reinforcement Learning (RL) algorithms that utilize macro-actions demonstrate many benefits, including improved exploration and faster learning (50). However, identifying effective macroactions is a challenging problem due to the curse of dimensionality, which arises from large action spaces. To address this issue, some approaches have employed genetic algorithms (51) or relied on expert demonstrations to extract macro-actions (52). However, these methods are not scalable and lack biological plausibility.

In contrast, our approach learns macro-actions using the principles of RL, thus requiring little overhead while combining the flexibility of primitive actions with the efficiency of macro-actions.

3.3 Action Repetition and Frame-skipping

To overcome the curse of dimensionality while gaining the benefits of macro-actions, many approaches utilize frame-skipping and action repetition, where macro-actions are restricted to a single primitive action that is repeated. Frame-skipping and action repetition serve as a form of partial open-loop control, where the agent selects a sequence of actions to be executed without considering the intermediate states. Consequently, the number of actions is linear in the number of time steps (53; 54; 55; 56; 57).

For instance, FiGaR (56) shifts the problem of macro-action learning to predicting the number of steps that the outputted action can be repeated. TempoRL (55) improved upon FiGaR by conditioning the number of repetitions on the selected actions. However, none of these algorithms can scale to continuous control tasks with multiple action dimensions, as action repetition forces all actuators and joints to be synchronized in their repetitions, leading to poor performance for longer action sequences.

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

Action Sequence

BG

Actor (GRU)

PFC

Critic

Model

Intermediate

states

reward

Environment

state

4 Hindsight Sequence Planner

Based on the insights presented in Section 2, we introduce a novel reinforcement learning model capable of learning sequences of actions (macro-actions) by replaying memories at a finer temporal resolution than the action generation, utilizing a model of the environment during training.

Components

The Hindsight-Sequence-Planner (HSP) algorithm learns to plan "in-the-mind" using a model during training, allowing the learned action-sequences to be executed without the need for model-based online planning. This is achieved using an actor-critic setting where the actor and critic operate at different frequencies, representing the observation/computation and actuation frequencies, respectively. Essentially, the critic is only used during training/replay and can operate at any temporal resolution, while the actor is constrained to the temporal resolution of the slowest component in the sensing-compute-actuation loop. Denoting the actor's timestep as t ′ and the critic's timestep as t , our algorithm includes three components:

We denote individual actions in the action sequence generated by actor using the notation π ω ( s t ′ ) t to represent the action a t ′ + t

Model : Learns the dynamics of the environment, predicting the next state s t +1 given the current state s t and primitive action a t .

Critic : Takes the same input as the model but predicts the Q-value of the state-action pair.

Actor : Produces a sequence of actions given an observation at time t ′ . Observations from the environment can occur at any timestep t or t ′ , where we assume t ′ > t . Specifically, in our algorithm, t ′ = Jt where J > 1; J ∈ Z .

Each component of our algorithm is trained in parallel, demonstrating competitive learning speeds.

We follow the Soft-Actor-Critic (SAC) algorithm (58) for learning the actor-critic. Exploration and 154 uncertainty are critical factors heavily influenced by timestep size and planning horizon. Many 155 model-free algorithms like DDPG (59) and TD3 (60) explore by adding random noise to each action 156 during training. However, planning a sequence of actions over a longer timestep can result in additive 157 noise, leading to poor performance during training and exploration. The SAC algorithm addresses this 158 by maximizing the entropy of each action in addition to the expected return, allowing our algorithm 159 to automatically lower entropy for deeper actions farther from the observation. 160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

Learning the Model

The model is trained to minimize the Mean Squared Error of the predicted states. For a trajectory τ = ( s t , a t , s t +1 ) drawn from the replay buffer D , the predicted state is taken from ˜ s t +1 ∼ m ϕ ( s t , a t ) .

The loss function is:

For this work, the model is a feed-forward neural network with two hidden layers. In addition to the current model m ϕ , we also maintain a target model m ϕ -that is the exponential moving average of the current model.

Learning Critic

The critic is trained to predict the Q-value of a given state-action pair ˜ q t = q ψ ( s t , a t ) using the target value from the modified Bellman equation:

Here, q ψ -is the target critic, which is the exponential moving average of the critic. Following the SAC algorithm, we train two critics and use the minimum of the two q ψ -values to train the current critics. The loss function is:

Both critics are feed-forward neural networks with two hidden layers.

Learning Policy

The HSP policy utilizes two hidden layers followed by a Gated-Recurrent-Unit (GRU) (61) that takes as input the previous action in the action sequence, followed by two linear layers that output the mean and standard deviation of the Gaussian distribution of the action. This design allows the policy to produce action sequences of arbitrary length given a single state and the last action.

A naive approach to training a sequence of actions would be to augment the action space to include all possible actions of the sequence length. However, this quickly leads to the curse of dimensionality, as each sequence is considered a unique action, dramatically increasing the policy's complexity. Additionally, such an approach ignores the temporal information of the action sequence and faces the difficult problem of credit assignment, with only a single scalar reward for the entire action sequence.

To address these problems, we use different temporal scales for the actor and critic. The critic assigns value to each segment of the action sequence, bypassing the credit assignment problem caused by the single scalar reward. However, using collected transitions to train the action sequence is impractical, as changing the first action in the sequence would render all future states inaccurate. Thus, the model populates intermediate states, which the critic then uses to assign value to each primitive action in the sequence.

Therefore, given a trajectory τ = ( a t -1 , s t , a t , s t +1 ) , we first produce the J -step action sequence using the policy: ˜ a t : t + J = π ϕ ( s t ) . We then iteratively apply the target model to get the intermediate states ˜ s t +1: t + J -1 . Finally, we use the critic to calculate the loss for the actor as follows:

5 Experiments

Overview

We evaluate our HSP approach on several continuous control tasks, comparing it against the SAC baseline and the TempoRL algorithm (55). Our focus is on environments with multi-dimensional actions, ranging from the simple LunarLanderContinuous (2 action dimensions) to the complex Humanoid environment (17 action dimensions). This allows us to highlight the benefits of HSP over traditional action repetition approaches. We utilize the OpenAI gym (62) implementation of the MuJoCo environments (11).

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

Experiemental Setup

We train HSP with four different action sequence lengths (ASL), J = 2 , 4 , 8 , 16 , referred to as HSPJ . During training, HSP is evaluated based on its J value, processing states only after every J actions. All hyperparameters are identical between HSP and SAC, except for the actor update frequency: HSP updates the actor every 4 steps, while SAC updates every step. Thus, SAC has four more actor update steps compared to HSP. Additionally, HSP learns a model in parallel with the actor and critic.

Learning Curves

LunarLanderContinuous-v2

Hopper-v2

300

3500

5000

200

3000

4000

Avg. Return

100

Avg. Return

2500

Avg. Return

SAC

2000

3000

-100

HSP-2

1500

2000

HSP-4

1000

-200

HSP-8

500

1000

-300

HSP-16

OF

0

0.1

0.2

0.30.4

0

0.20.40.60.8

0

0.20.40.60.8

Million Steps

Million Steps

Million Steps

HalfCheetah-v2

Humanoid-v2

6000

15k

8000

Avg. Return

Avg. Return

Avg. Return

4000

10k

6000

4000

2000

5k

2000

0

1

2

3

4

0

1

2

3

4

0

2

4

6

8

Million Steps

Million Steps

Million Steps

Figure 6 presents the learning curves of HSP and SAC across six continuous control tasks. We observe that HSP outperforms SAC in four out of six tasks (excluding Ant and HalfCheetah). Notably, HSP-16 achieves competitive performance on LunarLander and Hopper tasks, showcasing the algorithm's capability to learn long action sequences from scratch. Surprisingly, HSP also outperforms SAC in the Humanoid environment with fewer inputs and actor updates while concurrently learning a model, demonstrating the efficacy of the algorithm on environments with higher action dimensions.

Action Sequence Length (ASL) Performance

Learning curves alone do not fully capture HSP's performance and benefits. For instance, HSP-16 shows poor performance on Ant in the learning curve, yet it demonstrates competitive performance when tested on shorter action sequences. Figure 3 presents the performance of trained algorithms across different action sequence lengths (ASL).

We select the largest J that shows competitive performance (greater than 75% of the SAC when 220 evaluated on primitive actions) for each environment and test it for sequence lengths up to 30. For 221 SAC and HSP, we fix the length of action sequences while TempoRL is designed to dynamically pick 222 the best ASL, therefore we report the avg. action sequence length for TempoRL. HSP demonstrates 223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

LunarLanderContinuous-v2

Hopper-v2

300

3500

5000

Avg. Reward

200

Avg. Reward

2500

3000

Avg. Reward

4000

100

2000

3000

0-

1500

2000

-100

·HSP

1000

·SAC

500

1000

-200

·TempoRL

1o

0-

0

10

20

30

0

100

200

0

100

200

Action Sequence Length

Action Sequence Length (ms)

Action Sequence Length (ms)

HalfCheetah-v2

Humanoid-v2

6000

15k

8000-

Avg. Reward

Avg. Reward

理

Avg. Reward

4000

10k

6000

4000

2000

5k

2000

0-

0

+0

：：

11

0

500

1000

1500

0

500

1000

1500

0

200

400

Action Sequence Length (ms)

Action Sequence Length (ms)

Action Sequence Length (ms)

competitive performance on longer action sequences, approaching human-like reaction times in some environments. Unlike SAC, which fails with action sequences of 2 or 3, HSP shows a gradual degradation of performance. Additionally, HSP generalizes well in environments like LunarLander and Ant, even though the actor is trained only on sequence lengths of 16.

Comparing HSP to TempoRL, we find that TempoRL prefers shorter repetitions and struggles in more difficult environments. TempoRL does not incentivize longer actions and suffers from the curse of dimensionality to some extent, as it needs to learn the number of repetitions for each unique state-action pair. Furthermore, action repetitions are not suitable for multi-dimensional actions, as they force synchronized repetition across all actuators resulting in poor performance in environments with high action dimensions like Ant and HalfCheetah environments.

Comparison to Model-based Online Planning

In addition to action repetition, model-based online planning is another approach that allows the RL agent to reduce its observational frequency. However, it often requires a highly accurate model of the environment and incurs increased model complexity due to the use of the model during control. Despite these challenges, comparing HSP to model-based online planning is essential since it is useful when the actor cannot produce long sequences of actions and does not require the hyper-parameter J . With access to an accurate model of the environment, the agent's performance might generalize to arbitrary ASL.

Since HSP incorporates a model of the environment that is learned in parallel, we compare the 242 performance of the HSP actor utilizing the actor-generated action sequences against model-based 243 online planning, where the actor produces only a single action between each simulated state. 244

Humanoid-v2

8000

HSP

OnlinePlanning

Reward

6000

4000

2000

0

0

200

400

Action Sequence Length (ms)

Figure 4 shows the performance of online planning using the model in HSP versus the action 245 sequences generated by the HSP policy. We see that HSP can learn action sequences that perform 246 better than model-based online planning using the same model. Thus, HSP can leverage inaccurate 247 models to learn accurate action sequences, further reducing the required computational complexity 248 during training. We hypothesize that this superior performance is due to the fact that the actor learns 249 a J -step action sequence concurrently, while online planning only produces one action at a time. 250 Consequently, HSP is able to learn and produce long, coherent action sequences, whereas single-step 251 predictions tend to drift, similar to the "hallucination" phenomenon observed in transformer-based 252 language models. 253

Generative Replay in Latent Space 254

5000

5000

4000

Return

4000

3000

SAC

Rer

3000

R

bi

2000

HSP-2

2000

HSP-16

HSP-4

SAC

1000

HSP-8

1000

TempoRL

HSP-16

0

0

0.2

0.4

0.6

0.8

0

100

200

Million Steps

Action Sequence Length (ms)

Previous studies have shown that generative replay benefits greatly from latent representations (63). 255 Recently, Simplified Temporal Consistency Reinforcement Learning (TCRL) (64) demonstrated 256 that learning a latent state-space improves not only model-based planning but also model-free RL 257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

algorithms. Building on this insight, we introduced an encoder to encode the observations in our algorithm. We provide the complete implementation details in the Appendix.

We did not observe any benefits of using the encoder and temporal consistency for HSP in most environments (results in the appendix). However, for the Walker environment, utilizing the latent space for generative replay significantly improved performance, making it competitive even at 16 steps (128ms) (Figure 5).

6 Discussion, Limitations and Future Work

We introduce the Hindsight-Sequence-Planner (HSP) algorithm, a biologically plausible model for sequence learning. It represents a significant step towards achieving robust control at brain-like speeds. The key contributions of HSP include its ability to generate long sequences of actions from a single state, its resilience to reduced input frequency, and its lower computational complexity per primitive action.

The current RL framework encourages synchrony between the environment and the components of the agent. However, the brain utilizes components that act at different frequencies and yet is capable of robust and accurate control. HSL provides an approach to reconcile this difference between neuroscience and RL, while remaining competitive on current RL benchmarks. HSP offers substantial benefits over traditional RL algorithms, particularly in the context of autonomous agents such as self-driving cars and robots. By enabling operation at slower observational frequencies and providing a gradual decay in performance with reduced input frequency, HSP addresses critical issues related to sensor failure and occlusion, and energy consumption. Additionally, HSP generates long sequences of actions from a single state, which can enhance the explainability of the policy and provide opportunities to override the policy early in case of safety concerns. HSP also learns a latent representation of the action sequence, which could be used in the future to interface with large language models for multimodal explainability and even hierarchical reinforcement learning and transfer learning.

Limitations

Despite its advantages, HSP has some limitations. It shows slightly reduced performance in the Ant and HalfCheetah environments, which we believe can be mitigated through improved models and hyperparameter tuning. HSP also requires more computational resources during training due to the parallel training of an environment model and introduces more hyperparameters, particularly the training ASL ( J ). In this work, we do not optimize the neural network architecture of the actor to reduce the compute, as a result, the total compute per primitive action is still larger than SAC. However, we believe producing a sequence of actions will be more efficient than producing a single primitive action per state after optimization. Larger ASL values may not perform well in stochastic environments. Moreover, HSP currently uses a constant ASL, but ideally, the ASL should adapt based on the environment's predictability.

Future Work

We believe the HSP model contributes to both artificial agents and the study of biological control. Future work will incorporate biological features like attention mechanisms and knowledge transfer. Additionally, HSP can benefit from existing Model-Based RL approaches as it naturally learns a model of the world. In deterministic environments, a capable agent should achieve infinite horizon control for tasks like walking and hopping from a single state. This is an important research direction that is currently underexplored, as many environments are partially observable or have some degree of stochasticity. Current approaches rely on external information at every state, which increases energy consumption and vulnerability to adversarial or missing inputs. Truly autonomous agents will need to impl ement multiple policies simultaneously, and simple tasks like walking can be performed without input states if learned properly. Our future work will focus on extending the action sequence horizon until deterministic tasks can be performed using a single state and implementing a mechanism to dynamically pick the action sequence horizon based on context and predictability of the state. Serotonin is an important neuromodulator that has been demonstrated to signal the availability of time and resources in the brain to enable the decision on the planning horizon and the use of compute (65). In the future, we hope to introduce a mechanism to replicate the effect of serotonin in HSP.

310

311

312

313

314

315

316

317

318

References

R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press, 2018.

W. Schultz, P. Dayan, and P. R. Montague, 'A neural substrate of prediction and reward,' Science , vol. 275, pp. 1593-1599, 1997.

W. Schultz, 'Neuronal reward and decision signals: From theories to data,' Physiological Reviews , vol. 95, pp. 853-951, 7 2015.

J. Y. Cohen, S. Haesler, L. Vong, B. B. Lowell, and N. Uchida, 'Neuron-type-specific signals for reward and punishment in the ventral tegmental area,' Nature 2012 482:7383 , vol. 482, pp. 85-88, 1 2012.

OpenAI, C. Berner, G. Brockman, B. Chan, V. Cheung, P. D˛ ebiak, C. Dennison, D. Farhi, 319 Q. Fischer, S. Hashme, C. Hesse, R. Józefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, 320 H. P. d. O. Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tang, 321 F. Wolski, and S. Zhang, 'Dota 2 with large scale deep reinforcement learning,' 12 2019. 322

J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock323 hart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver, 'Mastering atari, go, chess and shogi 324 by planning with a learned model,' Nature 2020 588:7839 , vol. 588, pp. 604-609, 12 2020. 325

E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza, 326 'Champion-level drone racing using deep reinforcement learning,' Nature 2023 620:7976 , 327 vol. 620, pp. 982-987, 8 2023. 328

P. R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capo329 bianco, A. Devlic, F. Eckert, F. Fuchs, L. Gilpin, P. Khandelwal, V. Kompella, H. C. Lin, 330 P. MacAlpine, D. Oller, T. Seno, C. Sherstan, M. D. Thomure, H. Aghabozorgi, L. Barrett, 331 R. Douglas, D. Whitehead, P. Dürr, P. Stone, M. Spranger, and H. Kitano, 'Outracing cham332 pion gran turismo drivers with deep reinforcement learning,' Nature 2022 602:7896 , vol. 602, 333 pp. 223-228, 2 2022. 334

335

O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi,

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre,

T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard,

D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring,

Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu,

Hassabis, C. Apps, and D. Silver, 'Grandmaster level in starcraft ii using multi-agent reinforcement learning,' Nature 2019 575:7782 , vol. 575, pp. 350-354, 10 2019.

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, 'Human-level control through deep reinforcement learning,' Nature 2015 518:7540 , vol. 518, pp. 529-533, 2 2015.

E. Todorov, T. Erez, and Y. Tassa, 'Mujoco: A physics engine for model-based control,' in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 5026-5033, IEEE, 2012.

M. Towers, J. K. Terry, A. Kwiatkowski, J. U. Balis, G. d. Cola, T. Deleu, M. Goulão, A. Kallinteris, A. KG, M. Krimmel, R. Perez-Vicente, A. Pierré, S. Schulhoff, J. J. Tai, A. T. J. Shen, and O. G. Younis, 'Gymnasium,' Mar. 2023.

A. Jain, R. Bansal, A. Kumar, and K. Singh, 'A comparative study of visual and auditory reaction times on the basis of gender and physical activity levels of medical first year students,' International journal of applied and basic medical research , vol. 5, no. 2, pp. 124-127, 2015.

R. Limpert, Brake design and safety . SAE international, 2011.

B. Katz, J. D. Carlo, and S. Kim, 'Mini cheetah: A platform for pushing the limits of dynamic 356 quadruped control,' Proceedings - IEEE International Conference on Robotics and Automation , 357 vol. 2019-May, pp. 6295-6301, 5 2019. 358

[9]

G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal, 'Rapid locomotion via re359 inforcement learning,' International Journal of Robotics Research , vol. 43, pp. 572-587, 4 360 2024. 361

362

363

364

Q. Li, G. Dong, R. Qin, J. Chen, K. Xu, and X. Ding, 'Quadruped reinforcement learning without explicit state estimation,' in 2022 IEEE International Conference on Robotics and Biomimetics (ROBIO) , pp. 1989-1994, IEEE, 2022.

365

366

367

368

369

370

371

372

373

374

T. Haarnoja, B. Moran, G. Lever, S. H. Huang, D. Tirumala, J. Humplik, M. Wulfmeier, S. Tunyasuvunakool, N. Y. Siegel, R. Hafner, et al. , 'Learning agile soccer skills for a bipedal robot with deep reinforcement learning,' arXiv preprint arXiv:2304.13653 , 2023.

T. Wiestler and J. Diedrichsen, 'Skill learning strengthens cortical representations of motor sequences,' Elife , vol. 2, p. e00801, 2013.

N. F. Wymbs and S. T. Grafton, 'The human motor system supports sequence-specific representations over multiple training-dependent timescales,' Cerebral cortex , vol. 25, no. 11, pp. 4213-4225, 2015.

N. Favila, K. Gurney, and P. G. Overton, 'Role of the basal ganglia in innate and learned behavioural sequences,' Reviews in the Neurosciences , vol. 35, no. 1, pp. 35-55, 2024.

X. Jin, F. Tecuapetla, and R. M. Costa, 'Basal ganglia subcircuits distinctively encode the 375 parsing and concatenation of action sequences,' Nature neuroscience , vol. 17, no. 3, pp. 423376 430, 2014. 377

378

379

X. Jin and R. M. Costa, 'Shaping action sequences in basal ganglia circuits,' Current opinion in neurobiology , vol. 33, pp. 188-196, 2015.

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

G. S. Berns and T. J. Sejnowski, 'How the basal ganglia make decisions,' in Neurobiology of decision-making , pp. 101-113, Springer, 1996.

G. S. Berns and T. J. Sejnowski, 'A computational model of how the basal ganglia produce sequences,' Journal of cognitive neuroscience , vol. 10, no. 1, pp. 108-121, 1998.

E. Garr, 'Contributions of the basal ganglia to action sequence learning and performance,' Neuroscience & Biobehavioral Reviews , vol. 107, pp. 279-295, 2019.

A. J. Doupe, D. J. Perkel, A. Reiner, and E. A. Stern, 'Birdbrains could teach basal ganglia research a new song,' Trends in neurosciences , vol. 28, no. 7, pp. 353-363, 2005.

X. Jin and R. M. Costa, 'Start/stop signals emerge in nigrostriatal circuits during sequence learning,' Nature , vol. 466, no. 7305, pp. 457-462, 2010.

M. Matamales, Z. Skrbis, M. R. Bailey, P. D. Balsam, B. W. Balleine, J. Götz, and J. BertranGonzalez, 'A corticostriatal deficit promotes temporal distortion of automatic action in ageing,' ELife , vol. 6, p. e29908, 2017.

J. G. Phillips, E. Chiu, J. L. Bradshaw, and R. Iansek, 'Impaired movement sequencing in patients with huntington's disease: a kinematic analysis,' Neuropsychologia , vol. 33, no. 3, pp. 365-369, 1995.

L. Boyd, J. Edwards, C. Siengsukon, E. Vidoni, B. Wessel, and M. Linsdell, 'Motor sequence chunking is impaired by basal ganglia stroke,' Neurobiology of learning and memory , vol. 92, no. 1, pp. 35-44, 2009.

C. F. Geissler, C. Frings, and B. Moeller, 'Illuminating the prefrontal neural correlates of action sequence disassembling in response-response binding,' Scientific Reports , vol. 11, no. 1, p. 22856, 2021.

M. A. Immink, M. Pointon, D. L. Wright, and F. E. Marino, 'Prefrontal cortex activation during 402 motor sequence learning under interleaved and repetitive practice: a two-channel near-infrared 403 spectroscopy study,' Frontiers in Human Neuroscience , vol. 15, p. 644968, 2021. 404

D. Shahnazian, M. Senoussi, R. M. Krebs, T. Verguts, and C. B. Holroyd, 'Neural representa405 tions of task context and temporal order during action sequence execution,' Topics in Cognitive 406 Science , vol. 14, no. 2, pp. 223-240, 2022. 407

408

409

410

M. C. Zielinski, W. Tang, and S. P. Jadhav, 'The role of replay and theta sequences in mediating hippocampal-prefrontal interactions for memory and cognition,' Hippocampus , vol. 30, no. 1, pp. 60-72, 2020.

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

P. Malerba, K. Tsimring, and M. Bazhenov, 'Learning-induced sequence reactivation during sharp-wave ripples: a computational study,' in Advances in the Mathematical Sciences: AWM Research Symposium, Los Angeles, CA, April 2017 , pp. 173-204, Springer, 2018.

D. B. Rubin, T. Hosman, J. N. Kelemen, A. Kapitonava, F. R. Willett, B. F. Coughlin, E. Halgren, E. Y. Kimchi, Z. M. Williams, J. D. Simeral, et al. , 'Learned motor patterns are replayed in human motor cortex during sleep,' Journal of Neuroscience , vol. 42, no. 25, pp. 5007-5020, 2022.

T. M. Moerland, J. Broekens, A. Plaat, C. M. Jonker, et al. , 'Model-based reinforcement learning: A survey,' Foundations and Trends® in Machine Learning , vol. 16, no. 1, pp. 1-118, 2023.

D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pineau, and R. Fergus, 'Improving sample efficiency in model-free reinforcement learning from images,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, pp. 10674-10681, 2021.

M. Janner, J. Fu, M. Zhang, and S. Levine, 'When to trust your model: Model-based policy optimization,' Advances in neural information processing systems , vol. 32, 2019.

J. Wang, W. Li, H. Jiang, G. Zhu, S. Li, and C. Zhang, 'Offline reinforcement learning with reverse model-based imagination,' Advances in Neural Information Processing Systems , vol. 34, pp. 29420-29432, 2021.

D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, 'Curiosity-driven exploration by selfsupervised prediction,' in International conference on machine learning , pp. 2778-2787, PMLR, 2017.

B. C. Stadie, S. Levine, and P. Abbeel, 'Incentivizing exploration in reinforcement learning with deep predictive models,' arXiv preprint arXiv:1507.00814 , 2015.

N. Savinov, A. Raichuk, R. Marinier, D. Vincent, M. Pollefeys, T. Lillicrap, and S. Gelly, 'Episodic curiosity through reachability,' arXiv preprint arXiv:1810.02274 , 2018.

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. , 'Mastering the game of go without human knowledge,' nature , vol. 550, no. 7676, pp. 354-359, 2017.

S. Levine and V. Koltun, 'Guided policy search,' in International conference on machine learning , pp. 1-9, PMLR, 2013.

A. Zhang, H. Satija, and J. Pineau, 'Decoupling dynamics and reward for transfer learning,' arXiv preprint arXiv:1804.10689 , 2018.

R. Sasso, M. Sabatelli, and M. A. Wiering, 'Multi-source transfer learning for deep model-based reinforcement learning,' arXiv preprint arXiv:2205.14410 , 2022.

A. Fickinger, H. Hu, B. Amos, S. Russell, and N. Brown, 'Scalable online planning via reinforcement learning fine-tuning,' Advances in Neural Information Processing Systems , vol. 34, pp. 16951-16963, 2021.

A. McGovern, R. S. Sutton, and A. H. Fagg, 'Roles of macro-actions in accelerating reinforcement learning,' 1997.

Y.-H. Chang, K.-Y. Chang, H. Kuo, and C.-Y. Lee, 'Reusability and transferability of macro actions for reinforcement learning,' ACM Transactions on Evolutionary Learning and Optimization , vol. 2, no. 1, pp. 1-16, 2022.

H. Kim, M. Yamada, K. Miyoshi, T. Iwata, and H. Yamakawa, 'Reinforcement learning in 453 latent action sequence space,' in 2020 IEEE/RSJ International Conference on Intelligent Robots 454 and Systems (IROS) , pp. 5497-5503, IEEE, 2020. 455

456

457

458

S. Kalyanakrishnan, S. Aravindan, V. Bagdawat, V. Bhatt, H. Goka, A. Gupta, K. Krishna, and V. Piratla, 'An analysis of frame-skipping in reinforcement learning,' ArXiv , vol. abs/2102.03718, 2021.

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

A. Srinivas, S. Sharma, and B. Ravindran, 'Dynamic action repetition for deep reinforcement learning,' in AAAI , 2017.

A. Biedenkapp, R. Rajan, F. Hutter, and M. Lindauer, 'Temporl: Learning when to act,' in International Conference on Machine Learning , pp. 914-924, PMLR, 2021.

S. Sharma, A. Srinivas, and B. Ravindran, 'Learning to repeat: Fine grained action repetition for deep reinforcement learning,' ArXiv , vol. abs/1702.06054, 2017.

H. Yu, W. Xu, and H. Zhang, 'Taac: Temporally abstract actor-critic for continuous control,' Advances in Neural Information Processing Systems , vol. 34, pp. 29021-29033, 2021.

T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. , 'Soft actor-critic algorithms and applications,' arXiv preprint arXiv:1812.05905 , 2018.

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, 'Continuous control with deep reinforcement learning,' arXiv preprint arXiv:1509.02971 , 2015.

S. Fujimoto, H. Hoof, and D. Meger, 'Addressing function approximation error in actor-critic methods,' in International conference on machine learning , pp. 1587-1596, PMLR, 2018.

K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, 'Learning phrase representations using rnn encoder-decoder for statistical machine translation,' arXiv preprint arXiv:1406.1078 , 2014.

G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, 'Openai gym,' 2016.

G. M. Van de Ven, H. T. Siegelmann, and A. S. Tolias, 'Brain-inspired replay for continual learning with artificial neural networks,' Nature communications , vol. 11, no. 1, p. 4069, 2020.

Y. Zhao, W. Zhao, R. Boney, J. Kannala, and J. Pajarinen, 'Simplified temporal consistency reinforcement learning,' in International Conference on Machine Learning , pp. 42227-42246, PMLR, 2023.

K. Doya, K. W. Miyazaki, and K. Miyazaki, 'Serotonergic modulation of cognitive computa484 tions,' Current Opinion in Behavioral Sciences , vol. 38, pp. 116-123, 2021. 485

D. Yarats and I. Kostrikov, 'Soft actor-critic (sac) implementation in pytorch.' https:// 486 github.com/denisyarats/pytorch_sac , 2020. 487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are provided in Section 6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a complete algorithm and list of hyperparameters in the appendix. Additionally we also released the code and trained models.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

Answer: [Yes]

Justification: We include code to reproduce the results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide code, and list of all hyperparameters in appendix

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars are reported for all results presented.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: A description of compute resources used is provided in the appendix

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See section 6

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite original papers for each algorithm and envrionment used

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The documentation for the released code is provided

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

A Appendix / supplemental material 797

A.1 HSP Algorithm 798

Algorithm 1: Hindsight Sequence Planner

800

801

802

803

Hyperparameters

The table below lists the hyperparameters that are common between every environment used for all our experiments for the SAC and HSP algorithms:

A.2 Implementation Details

Due to its added complexity during training, HSP requires longer wall clock time for training when 804 compared to SAC. We performed a minimal hyperparameter search over the actor update frequency 805 parameter on the Hopper environment (tested values: 1, 2, 4, 8, 16). All the other hyperparamters 806 were picked to be equal to the SAC implementation. We also did not perform a hyerparameter search 807 over the size of GRU for the actor. It was picked to have the same size as the hidden layers of the feed 808 forward network of the actor in SAC. The neural network for the model was also picked to have the 809 same architecture as the actor from SAC, thus it has two hidden layers with 256 neurons. Similarly 810 the encoder for the latent HSP implementation was also picked to have the same architecture. For the 811 latent HSP implementation we also add an additional replay buffer to store transitions of length 5, 812 to implement the temporal consistency training for the model. This was done for simplicity of the 813 implementation, and it can be removed since it is redundant to save memory. 814

All experiments were performed on a GPU cluster the Nvidia 1080ti GPUs. Each run was performed 815 using a single GPU, utilizing 8 CPU cores of Intel(R) Xeon(R) Silver 4116 (24 core) and 16GB of 816 memory. 817

818

819

We utilize the pytorch implementation of SAC ( https://github.com/denisyarats/pytorch_ sac ) (66).

820

821

822

823

824

825

826

827

828

829

830

831

A.3 Latent State Space Experiments

Following the TCRL implementation, we use two encoders: an online encoder e θ and a target encoder e θ -, which is the exponential moving average of the online encoder:

Thus, the model predicts the next state in the latent space. Additionally, we introduce multi-step model prediction for temporal consistency. Following the TCRL work, we use a cosine loss for model prediction. The model itself predicts only a single step forward, but we enforce temporal consistency by rolling out the model H -steps forward to predict ˜ e t +1: t +1+ H .

Specifically, for an H -step trajectory τ = ( z t , a t , z t +1 ) t : t + H drawn from the replay buffer D , we use the online encoder to get the first latent state e t = e θ ( o t ) . Then conditioning on the sequence of actions a t : t + H , the model is applied iteratively to predict the latent states ˜ e t +1 = m ϕ (˜ e t , a t ) . Finally, we use the target encoder to calculate the target latent states ˆ e t +1: t + H +1 = e θ -( o t +1: t +1+ H ) . The Loss function is defined as:

834

Here, we provide complete learning curves for the latent space HSP.

LunarLanderContinuous-v2

Hopper-v2

300

3500

200

3000

5000

Avg. Return

100

Avg. Return

2500

Avg. Return

4000

SAC

2000

3000

100

HSP-2

1500

2000

HSP-4

1000

-200

HSP-8

500

1000

-300

HSP-16

0

D

0

0.1

0.2

0.3

0.4

0

0.2

0.4

0.60.8

0

0.20.40.60.8

Million Steps

Million Steps

Million Steps

HalfCheetah-v2

6000

15k

Avg. Return

Avg. Return

4000

10k

2000

5k

0

2

3

4

0

1

2

3

4

Million Steps

Million Steps

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem that commonly in RL, small reaction times and high action frequencies are required, which is not the case for computations in the brain. As a more accurate model, the authors propose an RL method that learns an internal model to improve performance in high-latency applications.\n\nSTRENGTHS:\nThe paper rightfully describes the issue that, usually, RL methods operate under different conditions than the human brain (higher latencies/lack of information). Developing a biologically plausible model that can deal with higher latencies and lower perceptual frequencies would be beneficial to get a better understanding of the human brain and for realizing certain applications in robotics. The approach of learning an internal model guiding the actions, if no observations are present, seems to follow intuition and could be promising. I would therefore rate the problem as relevant and interesting, and the general model (at least on a computational level - this could be stated more clearly in the paper) as \"plausible\".\n\nThe problem and approach (besides my points in the Weaknesses/Questions sections) are clearly described. The method is evaluated on a wide range of six control problems, showing that the model can produce reasonable control signals.\n\nWEAKNESSES:\nI found a few parts of the paper difficult to follow (mainly the policy and evaluation section, see also my questions for this), and the paper could benefit from improvements in these parts. In particular, I am uncertain whether the approach introduced in the paper fits the problem description. It seems that the approach targets the setting of high action frequency but not delays of perception.\n\nFurthermore, I had problems understanding how the policy of the proposed approach works. The authors seem to define a probabilistic policy ($\\pi_\\omega$) defined by parameters $\\omega$ with the past previous action as input (line 177). In Eq. 5, which describes the loss for learning the policy, the previous action does not seem to appear. For this loss, they need state-action pairs from a deterministic policy $\\pi_\\psi$. For a final evaluation, I would need clarification on this (see questions).\n\nAlso, the experiment section was not easy to follow. I believe it would be valuable to give an overview at the beginning of the section about how the evaluation is structured and state the goals of each conducted experiment (linked to the motivation of the paper). As an example, I think (please correct me if I am mistaken) that the comparison to SAC was conducted to show that the proposed method can learn a controller that, even with the limitation of a lower frequency, does not compromise much performance. I am a bit unclear as to why the proposed method outperforms SAC (what it does in 4/6 tasks), as to my understanding the strength of the method should be in the specific scenario where hardware is more similar to the human brain. That the proposed method achieves higher performance even in this standard scenario suggests to me rather a lack of appropriate parameter tuning. Also, the setting of the ASL subsection and Online planning section should be stated more clearly (see questions).\n\nIn lines 104-105, it is confusing to me to imply that the proposed method in comparison to model-based RL has the advantage that a model is not needed after training. The main purpose of applying model-based RL is to learn a model with the option to replan after training. Model-free RL which could be more similar for this application is not mentioned.\n\nSection 3.2. is about macro actions but it does not even provide in a single sentence an overview of what it is. The authors claim that an advantage of their approach is that it uses the principles of RL. Based on my understanding the concepts of hierarchical reinforcement learning and movement primitives would be very relevant here but are not discussed.\n\n\n## Minor:\n- Line 17 \"not\"\n- Line 163 $\\psi$ should be subscript\n- Line 202 Typo in \"Experiemental\"\n- I find it confusing that parentheses are used for both equations and citations. In most papers, therefore, for citations square brackets are used.\n- The plots do not use well the space in the paper (large whitespaces between subplots, the graphs could be made wider), and font sizes between plots differ significantly. Legends are usually integrated into the first subplot only but could be put, e.g., next to the plots to make this information more obvious.\n- The Readme of the provided code seems to be incomplete. The code cannot be directly used to reproduce the figures of the paper.\n\nQUESTIONS:\n- Line 31-32: \"When RL agents are constrained to human-like reaction times, even state-of-the-art algorithms struggle to perform in simple environments.\" Is there a reference for this statement?\n- I am a bit unclear about which exact problem / setting the proposed method addresses. Is it the delayed perception of the state as mentioned in the introduction? Then, why in the evaluation do the authors repeat the same actions instead of introducing a delay? Also, it seems to me that the proposed method can act on the current state for the first action in the action sequence (eq. 5). Or, is it the frequency of action execution? Then, why is the method motivated by the introduction of human latencies in decision-making? I could imagine that stating that humans operate under lower frequencies than autonomous systems might not be accurate.\n- Is the model probabilistic or deterministic? In (1) it seems deterministic, while in line 163 it seems like a state is sampled from a distribution.\n- In the section \"learning policy\": Where is the deterministic policy $\\pi_\\psi$ described and where do the state-action pairs come from?\n- For the generative replay in latent space / Figure 5. Are the plots for the right figure provided also for the other environments?\n- Why does in Fig. 6 HSP outperform SAC in 4/6 tasks? To my understanding, the SAC implementation there does not force any latencies or action repetitions.\n- How does the approach of SAC with action repetition work? Is the repetition already introduced for learning? Why do you work with action repetition and not use a more coarse-grained discretization?\n- How does the model-based online planning approach work in detail? What algorithm and approach do you use? \n- What is the purpose of the experiment \"Generative Replay in Latent Space\" and what are the implications of the results?\n\nLIMITATIONS:\nI think so.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Hindsight-Sequence-Planner (HSP), a reinforcement learning (RL) model inspired by the brain's ability to achieve precise control using slow neurons. The model aims to mimic human-like sensory and reaction times by leveraging an environmental model for sequence learning. HSP demonstrates competitive performance with fewer observations and actor calls compared to faster RL models. The model is evaluated on various continuous control tasks, showing robust performance even with longer action sequences.\n\nSTRENGTHS:\n1. The concept of mimicking brain-like conditions in RL models is innovative and offers a fresh perspective on sequence learning. The integration of temporal recall and sequence learning inspired by neural mechanisms is novel.\n2. The use of a temporal recall mechanism allows for fine-tuned action sequence learning despite operating on slower hardware.\n3. HSP demonstrates competitive performance across various continuous control tasks, showcasing its robustness and adaptability.\n4. The experiments cover a range of continuous control tasks and provide comparisons with state-of-the-art models like Soft Actor-Critic (SAC), highlighting HSP’s efficiency.\n\nWEAKNESSES:\n1. The paper lacks specific numerical performance comparisons to quantify improvements over baseline models.\n2. Sections like \"Learning the Model\" and \"Learning Critic\" need further elaboration to highlight their specific contributions and novelty.\n3. The broader implications and potential real-world applications of HSP are not fully discussed.\n4. There is insufficient discussion on how HSP handles situations with highly inaccurate model predictions.\n5. The related work section lacks details on \"Macro-Actions,\" the scalability issues of current methods, and the meaning of \"principles.\"\n\nQUESTIONS:\n1. Can the authors provide more specific numerical comparisons of HSP's performance gains over baseline models in the experimental section?\n2. How does HSP handle situations where the model predictions are highly inaccurate?\n3. Could the authors elaborate on the potential real-world applications where HSP would be particularly beneficial?\n4. Lower computation and scalability in HSP are contributions; how can these two metrics be quantified?\n\nLIMITATIONS:\n1. The authors mention the reliance on an inaccurate model but could provide more in-depth analysis on how this affects different types of tasks?\n2. The scalability of HSP to very large action spaces and high-dimensional state spaces is not thoroughly discussed.\n3. Typo Issue: In Section Abstract: demonstrating that it not can achieve comparable performance at ‘human-like’ frequencies by relying on significantly fewer observations and actor calls. -->demonstrating that it can achieve comparable performance at ‘human-like’ frequencies by relying on significantly fewer observations and actor calls.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed HSP, a bio-mimic framework for learning-based control. Motivated by human brains, HSP can deal with observation and computation in different frequencies by making the \"actor\" produce action sequences, similar to the functioning pattern of ganglia and the prefrontal cortex in human brains. HSP employs a model-based training approach to achieve model-free control, resulting in precise behavior despite running on slow hardware. The authors demonstrate the performance of HSP on various continuous control tasks.\n\nSTRENGTHS:\nThis paper is well structured with clear figures, and the motivation and introduction are interesting, bridging the gap between biological and artificial reinforcement learning systems. The empirical results across various continuous control tasks are impressive as well. Overall, the work presents a promising direction for developing more efficient and adaptable RL algorithms that could have broad implications for robotics and other real-world applications.\n\nWEAKNESSES:\n* One major concern is the performance of the proposed framework. The experiment results do not show a significantly better performance than traditional RL methods, and the performances are worse when $J=16$ in most cases. The latent space variant of HSP shows improvements only in one environment (Walker2d). Further investigation into why this approach doesn't generalize well to other environments would be valuable.\n* The novelty of the proposed framework is also questionable. In control literature, especially trajectory optimization control, producing action sequences is commonly used, and a similar technique could be viewed as a variant of model predictive control (MPC). It would be better if the authors could provide a detailed comparison with existing control algorithms (like MPC) in the experiment section. \n* The comparison with model-based online planning is somewhat limited. A more comprehensive comparison with state-of-the-art model-based RL methods would provide better context for HSP's contributions.\n* Just a small suggestion - The title of the paper mentions \"hardware,\" so I expect to see some real-world control experiments in the paper. It would be nice if the author could really demonstrate the control performance using \"slow hardware\" like Raspberry Pi or even slower platform.\n\nQUESTIONS:\n* How does HSP perform in environments with stochasticity? Traditional RL and control methods could flexibly correct errors due to high control frequency. Could you provide results or insights on how the algorithm might be adapted for such scenarios?\n* The current implementation uses a fixed action sequence length. Have you considered mechanisms for dynamically adjusting the sequence length based on the task or environment complexity?\n* Can you provide more detailed analysis of the computational costs of HSP compared to baseline methods, especially during inference?\n* The paper doesn't specify the computational resources required for training the models. Given that training RL models could be computationally intensive, could you provide more details about this?\n\nSome typos: \n* Equation (1): $a_t$ to $a_{t^\\prime}$\n* Line 272: HSL to HSP\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}