Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate
Authors: Fan-Ming Luo
Zuolin Tu
Zefang Huang
Yang Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Recurrent Off-Policy Reinforcement Learning
Partially Observable Markov Decision Process
Primary area: reinforcement_learning

Abstract: Real-world decision-making tasks are usually partially observable Markov decision processes (POMDPs), where the state is not fully observable. Recent progress has demonstrated that recurrent reinforcement learning (RL), which consists of a context encoder based on recurrent neural networks (RNNs) for unobservable state prediction and a multilayer perceptron (MLP) policy for decision making, can mitigate partial observability and serve as a robust baseline for POMDP tasks. However, prior recurrent RL algorithms have faced issues with training instability. In this paper, we find that this instability stems from the autoregressive nature of RNNs, which causes even small changes in RNN parameters to produce large output variations over long trajectories. Therefore, we propose **R**ecurrent Off-policy RL with Context-**E**ncoder-**S**p**e**cific **L**earning Rate (RESeL) to tackle this issue. Specifically, RESeL uses a lower learning rate for context encoder than other MLP layers to ensure the stability of the former while maintaining the training efficiency of the latter. We integrate this technique into existing off-policy RL methods, resulting in the RESeL algorithm. We evaluated RESeL in 18 POMDP tasks, including classic, meta-RL, and credit assignment scenarios, as well as five MDP locomotion tasks. The experiments demonstrate significant improvements in training stability with RESeL. Comparative results show that RESeL achieves notable performance improvements over previous recurrent RL baselines in POMDP tasks, and is competitive with or even surpasses state-of-the-art methods in MDP tasks. Further ablation studies highlight the necessity of applying a distinct learning rate for the context encoder. Code is available at https://github.com/FanmingL/Recurrent-Offpolicy-RL.

Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Sppiife Learning Rate
Fan-Ming Luo!.2 Zuolin Tu Zefang Huang'
Yang 12+
National Key Laboratory for Novel Software Technology. Nanjing University, China School Artificial Intelligence, Nanjing University, China
Polixin.ai
luofm@lamda .nju. edu. en, zuolin. tu@polixir.aa ai, zf frank. huang@gmail com, yuy@nju.ead edu.
Abstract
Real-world decision-making tasks are usually partially observable Markov decision processes (POMDPs), where the state not fully observable. Recent progress has demonstrated that recurrent reinforcement learning (RL), which consists of a context encoder based on recurrent neural networks (RNNs) for unobservable state prediction and multilayer perceptron (MLP) policy for decision making, can mitigate partial observability and serve as a robust baseline for POMDP tasks. However, prior recurrent RL algorithms have faced issues with training instability. this paper, we find that this instability stems from the autoregressive nature of RNNs, which causes small changes in RNN parameters produce large output variations over long trajectories. Therefore, we propose Recurrent Off-policy RL with Context-Encoder- this issue. Specifically, RESeL uses lower learning rate for context encoder than other MLP layers ensure the stability of the former while maintaining the training efficiency of the latter. We integrate this technique into existing off- policy RI methods, resulting in the RESel. algorithm. We evaluated RESel. in 18 POMDP tasks, including classic, meta-RL, and credit assignment scenarios, well five MDP locomotion tasks. The experiments demonstrate significant improvements in training stability with RESeL. Comparative results show that RESeL achieves notable performance improvements over previous recurrent RL baselines in POMDP tasks, and is competitive with or even surpasses state-of- the-art methods in MDP tasks. Further ablation studies highlight the necessity of applying a distinct learning rate for the context encoder. Code is available at htttps://githu: com/ wtw.Paannnnnt//Peeurrre----ffeeo
Learning Rate (RESeL) to tackle
Introduction
In many real-world reinforcement learning (RL) tasks (1) complete state observations often unavailable due to limitations such as sensor constraints, cost considerations, or task-specific require- ments. For example, visibility be obstructed by obstacles in autonomous driving [2] 3 robotic manipulation 44, and measuring ground friction may be infeasible when controlling quadruped robots complex terrain (5). These scenarios common and typically conceptualized Partially Observable Markov Decision Processes (POMDPs). Traditional RI. struggles with POMDPs due to the lack of essential state information (6).
mainstream class of POMDP RL algorithms infers unobservable states by leveraging historical observation information, either explicitly or implicitly (6). This often requires memory-augmentt
"Yang Yu is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

network architectures, such as recurrent neural networks (RNNs) 7 8 and Transformers 19. Re- placing standard networks in RL algorithms with these memory-based structures has proven effective Particularly, recurrent RI. [I]], which employs an RNN-based context encoder to extract unobservable hidden states and an MIP policy to make decisions based on both current observation and hidden states, demonstrates robust performance in POMDP tasks. Compared to Transformer-basee RL [133, recurrent RL offers lower inference time complexity, making it highly applicable, especially in resource-constrraine terminal controllers.
in solving POMDP problems [10 12 13
Despite these advantages, recurrent RL faces a significant challenge: while advanced RNN archi- tectures 7 8 [4] can effectively address gradient explosion issues, training in recurrent RL often remains more unstable compared to MLP-based RL, particularly when handling long sequence lengths. This instability can lead to poor policy performance and even training divergence [[S]. Although existing methods usually avoid training with full-length trajectories by using shorter tra- jectory segment [II, this practice introduces distribution shift due to the inconsistency between the sequence lengths used during training and deployment, ESCP [T6] addressed this by truncating the history length during deployment to match the training sequence length, but this may limit policy performance due to restricted memory length. Additionally, some methods introduce auxiliary losses to aid context encoders in learning specific information. For instance, they train alongside transition model, forcing the outputs of context encoder to help minimize the prediction error of this model [1718] 18, 19]. However, these methods require the RNN to learn specific information, which may limit the RNN's potential and may only be applicable to a subset of tasks.
In this work, we found that, with the autoregressive property of RNNs, the output variations caused by parameter changes are amplified as the sequence length increases. As RNN parameters change, even slight variations in the RNN output and hidden state at the initial step can become magnified in subsequent steps. This occurs because the altered hidden state is fed back into the RNN at each step, causing cumulative output variations. Our theoretical analysis shows that these output variations grow with the sequence length and eventually converge. The amplified output variations can lead to instability in the RL process. For instance, in off-policy RL algorithms, the bootstrapped update target of the Q-function may fluctuate significantly, resulting in unstable Q-function training. To avoid training instability caused by excessive amplification of output variations, we propose Recurrent Learning Rate (RESeL). Specifically, we employ a lower learning rate for the RNN context encoder while keeping the learning rate for the other MLP layers unchanged. This strategy ensures efficient training for MLPs, which do not experience the issue of amplified output variations.
Off-policy RL with
Based on the SAC framework (20), and incorporating context-encoder-speifff learning rate as well as the ensemble-Q mechanism from REDQ (21] for training stabilization, we developed the practical RESeL algorithm. We empirically evaluated RESeL across 18 POMDP tasks, consisting of classic POMDP, meta-RL, credit assignment scenarios, well as five MDP locomotion tasks. In our experiments, we first observed the increasing RNN output variations over time and demonstrated that the context-emooder--pecii learning rate can mitigate the amplification issue, significantly enhancing the stability and efficacy of RL training. Comparative results indicate that RESeL. achieves notable performance improvements over previous recurrent RL methods in POMDP tasks and is competitive with, or even surpasses, state-of-the-ar (SOTA) RL. methods in MDP tasks. Further ablation studies highlight the necessity of applying distinct learning rate for the context encoder.
2 Background
Partially Observable Markov Decision Processes (POMDPs)
[22 6 enhance MDPs for scenarios with limited state visibil- ity, addressing a number of real-world decision-making chal- lenges with imperfect or uncertain data. A POMDP is defined by A, O, P, O, Y, Pa), consisting of state space S, action space A, observation space O, state transition function P. obser- vation function O, reward function r, discount factor y, and initial state distribution Pn- Unlike MDPs, agents POMDPs receive observations E O instead of direct state information, requiring a belief state-a a probability distribu- tion over S-to to inform decisions. The objective to search for a policy x to maximize the expected rewards, while factoring in observation and transition uncertainties.
MLP Policy
RNN
MLP
Context Encoder
Figure 1: A simple recurrent
policy architecture

---

Recurrent RL 23 involves employing RNN-based policy or critic models within the framework of RL. Figure provides a commonly utilized recurrent policy architecture. This architecture has two main components. First, context encoder processes the current observation o, the last action á, the reward O and the RNN hidden state. Then, an MLP policy uses the context embedding and to generate actions, facilitating the extraction of non-observable states into a context embedding. We can train the recurrent policy by incorporating it into any existing RL algorithm. During training, recurrent RL requires batch sampling and loss computation on seaecctoryybyttajecctee basis.
3 Related Work
In this work, we focus on recurrent RL methods [24 25 23 which leverage CNNNbased models for RL tasks. These methods have demonstrated robust performance in POMDP [00 22] and meta-
RL tasks [25] (6). having shown notable success in practical applications such as robotic motion
control 27 28 29 and MOBA games
30
Recurrent RI. utilizes RNNs such as GRUs 8. LSTMs [7]. and more recently, SSMs 12 to construct policy and critic models. Typically, in recurrent actor-critic RI,, both policy and critic adopt RNN architectures [23] [I]. When full state information is available during training, an MLP critic can also be used instead [29], receiving the full state as its input. In this work, we do not assume that complete state information can be accessed, so policy and critic models are both RNN structures.
The most direct way to train RNN models in recurrent RL is combining RNN structure with existing RL algorithms, such as being integrated into on- polpiolly methods like PPO [28] 30, B1]
and TRPO 25 32 as well off-policy methods like TD3 [1] and SAC 23 This work follows the line of recurrent off-policy RL [1 23 as we believe that existing recurrent off-policy RI. algorithms have not yet achieved their maximum potential.
Despite the simplicity of direct integration, however, training RNNs often suffers from the training instability issue 15), especially in long sequence length. Previous methods usually use truncated trajectory to train RNNs [.. However, deploying recurrent models with full-trajectory lengths can result in a mismatch between training and deployment. ESCP [16] addressed this issue by using the same sequence lengths in both training and deployment by truncating the historical length in deployment scenarios, but this could impair policy performance due to the restricted memory length. On the other hand, many studies also employed auxiliary losses in addition to the RL loss for RNN training [19.. These losses aim to enable RNNs stably extract some certain unobservable state either implicitly or explicitly. For instance, they might train the RNNs to predict transition parameters (33, distinguish between different tasks [16] 34], accurately predict state transitions [17 18 19.. These methods require the RNN to learn specific information, which may limit the RNN's potential and may only be applicable a subset of tasks, reducing the algorithmic generalizability.
RESeL directly trains recurrent models using off- off-policy RL due to its simplicity, flexibility, and potential high sample efficiency. The most relevant study to RESeL is (II), which focused on improving recurrent off-policy RL by analyzing the hyperparameters, network design, and algorithm choices. More in depth, RESeL studies the instability nature of previous methods and enhances the stability of recurrent off-policy RI. through a specifically designed learning rate.
4 Method
In this section, we address the training stability challenges of recurrent RI. We will first elaborate on our model architecture in Sec. 4.1 and introduce how we address the stability issue in Sec. 4.2 Finally, we will summarize the training procedure of RESeL in Sec. 4.3
4.1 Model Architectures
The architectures of RESeL policy and critic models depicted in Fig. 2 Specifically. RESeL policy initially employs MLPs as pre-encoders to map the current and last-step observations (o and ò respectively), actions (@), and rewards (f) into hidden spaces with equal dimensions, ensuring balanced representation of various information types. These pre-encoded inputs are then concatenated
"The inclusion of reward (f) in policy inputs is optional and depends on the RL task throughout the paper.
3

---

Pre- Encodert
Policy Architecture
Critic Architecture
@
Contest Encoder (CA)
MLP Palkyy
Contect Erender (C0)
RNN
7
Bridden State
6AN
KNN
MLP
concatir on
Figure 2: Policy and critic architectures of RESeL.
into vector and inputted into context encoder. For the context encoder, RESeL utilizes RNN such as GRU 8 and Mamba [14, to mix sequential information and extract non-observable state information from the observable context. The MLP policy comprises a two-layer architecture, each layer consisting of 256 neurons. The architecture of the pre-encoder and context encoder for the RESeL critic mirrors that of the policy. To improve training stability, RESel. adopts the ensemble-Q technique from REDQ [21], employing 8 MLP critic models, and each one of them comprises two hidden layers with 256 neurons.
4.2 Stabilizing Training with Context-Enccoeer-Spiiiie Learning Rate
To locate the source of instability, we first formalize the RNN and its sequence generation process. An RNN network (2,, h2) X' x H + } H is defined as: Vt, he41 f (az, hy), where represents the network parameters, 2t, Yt. and h4 are the RNN input, output, and hidden state, respectively. For simplicity,
denote Yt fy (Iz, he) and h++1 fM (1z, h,). Given an input sequence { .T....,,TT.. TT) of length T + 1 and generated by f is denoted as (yu. ....... ... yr }. Let O' be the parameter neighboring @ after one- step gradient update. The output sequence produced by f is denoted as {b6+ Vi....,.f. With certain assumptions, we can derive Proposition]] [ which bounds the difference between 3/2 and v/ at any time step f.
initial hidden state ho. the output sequence
Proposition 1. Assuming 1 and od both satisfy Lipschitz continuity, i.e., for all E0,00, X. h, h E H. there exist constants Kn E [0, 1) and Kg E R such that:
(1(,1) - fh(,,,1) (.h.))-fh(e,
Kr. ||A 1'll.
f (x, h')
Kg l h'l,
and for all x E X. h H. the output differences between the RNN parameterized by and 1 are bounded by a constant E R.
(((r)))) fh (a. 11111115(2.1)) - (2,
max
the RNN output difference at t-th step is bounded by
1 Ki Kg amplification factor
llve will
Vt 2 0.
(I)
1 Kr.
Refer to Appendix A for a detailed proof. Proposition focuses on the case where K₁ [0, 1). as most RNN models adhere to this condition. GRU [8] and LSTM 7 meet this requirement through the use of sigmoid activation functions, while Mamba [4 achieves by constraining the values in the hidden state's 's transition matrix.
In Eq. I represents the maximum variation of the model output when the model parameters are modified and both the model input and hidden state remain constant, i.e., the first-step output variation. However, from Eq. (T). we observe that due the variability in the hidden state, the final variation in the model output is amplified. This amplification factor is minimal at / = 0, being 0. K,
As increases, this amplification factor gradually increases and ultimately converges to B = I-k A Additionally, it can be verified that as t increases, the upper bound of the average variation in network output E'=1 l/g: víll also increases with time step and eventually converges to Be + (proved

---

in Appendix B which is again amplified by B. This conclusion indicates that, compared to the case of sequence length of 1, in scenarios with longer sequence lengths, the average variations in the RNN output induced by gradient descent are amplified.
This issue is different from the gradient explosion that commonly occurs in RNNS over long se- quences [7]. Proposition| [i that the variation in network output is amplified by a constant factor, rather than undergoing exponential explosion. This type of instability is not very significant in supervised learning but can lead to instability in RL processes. For instance, in off-policy RL algo- rithms, the optimization target of the Q-function minimizes the difference with a bootstrapped target. If the network output varies greatly. this target will fluctuate dramatically, making the Q- Q-function training unstable. which in turn can lead overall training instability.
To ensure stable training, it is required to use a smaller learning rate to counterbalance the amplified variations in network output caused by long sequence lengths, smaller learning rate help reduce 6. However, the output variations MLPs are not amplified in the same way as in RNNs. MLPs are also set with small learning rate, their learning efficiency could be significantly reduced, as the learning rate may become too slow for MLPs, resulting in inefficient training and poor overall performance. Consequently, we propose to use a context-encoder-speeiff learning rate for the context encoder. In our implementation, we use smaller learning rate LRce for the context encoder, which contains the RNN architecture. For other layers, we use a normal learning rate LRother- e.g. x 10 Applying different learning rates to different modules is similar to Two-Tine Tioeecsee Network [35]. which has been shown improve the training convergence.
4.3 Training Procedure of RESeL
Based on the SAC framework 20 combining context-encoder-speciff learning rate and the ensemble-Q mechanism from REDQ 21, we developed the practical RESeL algorithm. Specifically, we initially configure context- weattt-nnooooscciift.. optimizers for both the policy and value models. After each interaction with the environment, RESel. samples batch of data from the replay buffer, containing several full-length trajectories. This batch is used to compute the critic loss according to REDQ, and the critic network is optimized using its mentexx-encoder-speecfff optimizer. Notably, unlike REDQ, we did not adopt a update-to-data ratio greater than 1. Instead of that, update the network model once per interaction. The policy network is updated every two critic updates. During the policy update, the policy loss from REDQ is used, and the policy network is optimized with its cantexttencoder-speeiff optimizer. Here, we delay the policy update by one step to allow more thoroughly training of the critic before updating the policy, which is inspired by TD3 (36] to improve the training stability. The detailed algorithmic procedure is provided in Appendix C and Algorithm
5 Experiments
To validate our argument and compare RESeL with other algorithms, we conducted series of valida- tion and comparison experiments across various POMDP environments. We primarily considered three types of environments: classical partially observable environments, meta-RL. environments, and credit assignment environments. To assess the generalizablity of RESeL. to MDP tasks, we also test RESeL in five MuJoCo [77 locomotion tasks with full observation provided. We implement the policy and critic models with a parallelizable RNN, i.e., Mamba [14] to accelerate the training process. A detailed introduction of the network architecture can be found in Appendix D.3.2
In all experiments, the RESeL algorithm was repeated six times using different random seeds, i.e. 1-6. All experiments were conducted on a workstation equipped with an Intel Xeon Gold 5218R CPU, four NVIDIA RTX 4090 GPUs, and 250GB of RAM, running Ubuntu 20.04. For more detailed experimental settings, please refer to Appendix
5.1 Training Stability
The updates of the RNN lead to large variations in the model output. In Sec. 4.2 we observe that the variations in model output induced by a model update are amplified. this part, we quantify the amplification factor, assess its impact on RL training, and discover how a context-encoder-speene learning rate addresses this issue. We use POMDP task, specifically WalkerBITT--vd to validate argument. We loaded a policy model trained by RESeL and updated it with various LRçe and
5

---

Zo0m-in
action variations after one-siep updale
0.4
0.8
actoon variation
0.3
6
0.2
0.1
0.0
20
40
250
1000
lout
rallout stop
1Ra LR
(R 10-1
(Pcx 0 LRoa
LRat
UPawe 20
Figure 3: Action variations the rollout step increases after a single gradient update with different values of LRCE and LRother. Action variation refers to the change in policy output after the gradient update compared to its output before the update, using the same input sequences.
LRother for a one-step gradient update. The output differences for all rollout steps between the pre- and post-update models are presented in Fig. [ The right panel shows the variations in model output as the rollout step increases, while the left panel zooms on the rollout steps from 0 to 75.
In the zoom-in panel, comparing the brown-circle and rellow-triangl curves reveals that at the first rollout step, the two curves very close. This indicates that with the same learning rate, identical input, and hidden states, the updates of RNN and MLP have almost the same impact on the model's output. However, as the rollout steps increase, the brown-cirele curve gradually rises, increasing by 400% at the 20-th step. This observation aligns with Proposition D demonstrating that changes in the hidden state lead to increasing variations in the model's output. With further increases in rollout length, the right panel shows that the brown-cirele curve converges at around 0.4 after 250 steps, indicating that the amplification of the output variations eventually converges. This also meets Ultimately, the increase in action variations is approximately tenfold.
Proposition I
This action variation magnitude is equivalent a learning rate increased twentyfold. At this point, the learning rate reaches 0.006, which is typically avoided in RL scenarios due the risk of training instability. Conversely, reducing LRCE 10 (the purple-diamong curve) significantly suppresses the amplification of variations. The right panel shows that the orange and purple curves remain at similar levels until the final time step.
MLP policy network (the gray-dashed curve) with
Large output variations lead to instability in RL training. To investigate how the large variations influence the RL process, visualized the gradient norm and value function loss during training for two POMDP tasks, as shown in Figs. and 5 We fixed LR = 3 x 10 and compared the training processes of LRcF 10 (red line) and 3 10 (orange line). For the latter, we applied gradient clipping: otherwise, training would diverge and stop early. The orange line in Fig. 4 shows the norm of the policy gradient before gradient rescaling, where significant oscillations appear in the gradient in the later stages of training. The orange gradient norm is eventually scaled to 0.5, consistently smaller than the red line, which does not show oscillations. This suggests that the late-stage instability of the orange line does not stem from large gradient magnitudes.
On the other hand, as seen in Fig- [ the value loss for the orange line remains consistently high, with extremely large values appearing in later stages. This is due to large variations in the RNN output, which lead to an unstable bootstrapped update target for the value function. Consequently, the value loss diverges, causing the abnormal gradient norm observed in Fig. 4 The results in Figs, 4 and reveal the unique challenges of using RNN structures in RL, where traditional RNN stabilization techniques may no longer he effective.
learning rate improves training stability, while traditional RNN stabilization techniques fall short. To further examine the impact of a lower learning rate on the stability of RL training, conducted experiments across several POMDP tasks. Building on the setup in Fig. we further evaluated a variant without the gradient clipping technique and another that replaces gradient clipping with a truncation of recurrent backpropagation steps to 32 (green line). For reference, we also included a variant where the CE output is entirely masked to zero (purple line).
Using context-encoder-speeeee
Comparing the vanilla RNN-based RL (blue line) with the purple line shows that introducing an RNN can partially address partial observability issues and improve policy performance. However, the blue line still exhibits instability, even triggering early stopping due to outliers. Setting specific

---

learning rate for the CE (red line) significantly enhances training stability, indicating that reducing the RNN learning rate indeed improves overall RL training stability. However, we also found that traditional RNN stabilization techniques do not reliably improve RI. training stability. For example, truncating the number of recurrent backpropagation steps (green line) can still lead to early stopping, while clipping the gradient norm (orange line) prevents early stopping but does not improve the policy performance. This is because these two stabilization techniques can only suppress gradient explosions in the RNN, keeping the RNN gradients within a normal range. However, the output of the RNN can still vary significantly, which continues to cause instability in RL.
HallChheetanBLT-Www
Walker T-44000
HaIfChseta/BLTVV0
WalkerB LTVV
6.0
12
10*
4.5
3 10P
10
3.0
10a
10'
1.5
3
10
0.0
timestep
1e6
timestep
1eb
1eb
106
LRca - 10- no horgradd_.ggoog norm_clip
LRca 1019, no. grad narm clip
LRcc 3 10-1, max, grod nomm-o.s
LRzz 100 10 max, grad norm=0.5
Figure 4: L2-norm of the policy gradient for different RNN stabilization approaches.
Figure 5: Value losses in log-scall for differ- ent RNN stabilization approaches.
AniBLTA--V0
HallCheetta@LTTVv00
HopperBLTA-v0
WolkerBLTAV-00
3000
2000
1800
2000
1500
retu
1200
1000
1600
1000
600
800
500
1000
0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
timestep
1e6
timestep
1c6
timestep
1e6
timestep
1e6
1Rc-10
1R x 10 grad _tspsttppppp grad_step_
truncation urd__ste__trunration 1R x 10-4, grad max max__arr=003. norm=0.5
ND
1R 10
Figure 6: Learning curves in four POMDP tasks with different learning rates and RNN stabilization techniques, shaded with one standard error. We fixed LRother x 10-4, Some learning curves in AntBLT-V and HalfCheetahBLL-V are incomplete as some runs encountered infinite or NaN outputs.
5.2 Performance Comparisons
In this part, we compare RESeL with previous methods in various tasks. The detailed introduction of the baselines and tasks can be found in Appendix D.2 and Appendix [.1 More comparative results can be found in Appendix E.3
Classic POMDP Tasks. We first compare RESeL with previous baselines in four PyBullet locomotion environments: AntBLT, HalfCheetahBLI, HopperBLT, and WalkerBLT. To create partially observable tasks, obscure part of the state information as done in previous studies [18, 11 151, preserving only position (-P tasks) or velocity (-V tasks) information from the original robot observations. We benchmark RESeL against prior model-free recurrent RL (MF-RNN) [I]] VRM (18), and GPIDE 38 GPIDE, which extracts historical features inspired by the principle of PID controller, the state-of-the-art (SOTA) method for these tasks.
39
The comparative results are shown in Fig. 7 RESeL demonstrates significant improvements over previous recurrent RL methods (MF-RNN, PPO-GRU, and A2C-GRU) in almost all tasks, except for HalfCheetahBLLT-P where its performance is close to that of MF-RNN. These results highlight the advantage of RESeL in classic POMDP tasks over previous recurrent RL methods. Furthermore, RESeL outperforms GPIDE in most tasks, establishing as the new SOTA method. The learning curves of RESeL are more stable than that of GPIDE, suggesting that fine fine-grained feature design could also introduce training instability, while a stably trained RNN can even achieve superior performance.
Dynamics-Randomizze Randomized Tasks. Instead of directly obscuring part of the immediate state, meta-RI. considers different type of POMDP. In meta-RL scenarios (26), the agent learns across various
7

---

AntBLT-P-40
AntBLT-A-0
HallCheetah@LT-P-00
HallCheetahBLTT--0
3000
2000
3000
2400
1500
1500
1600
1500
1000
800
500
0
1500
1500
0.5
0.0
1.5
1.5
timestep
timestep
timestep
1o6
timestep
1e6
DIpperBLT-P-v0
HopperBLT-Vvv
WalkeraLT-Pvvd
WalkerBLT-Vvv
3200
2400
1800
2400
1600
1600
1600
800
800
800
600
0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
timestep
1e6
timestep
1e6
timestep
166
timestep
1e6
RESeL (ours) PPO-GRU
MF-RNN
GPIDE-ESS
A2C-GRU
SAC-Transformer
TD3-MLP
VAM
Figure 7: Learning curves shaded with one standard error in classic POMDP tasks.
tasks with different dynamics reward functions 16 where the parameters of the dynamics and reward functions are not observable. We first compare RESeL with previous methods in the dynamics randomization meta-RL tasks. Following previous meta-RL work [16], we randomized the gravity in MuJoCo environments [37]. We created 60 dynamics functions with different gravities, using the first 40 for training and the remaining for testing. The gravity is unobservable, requiring the agents to infer it from historical experience. We compare RESeL to various meta-RL RLL methods in these tasks, including ProMP [4), ESCP [16], OSI [33], EPI [41, and PEARL 42). Notably, ESCP, EPI, and OSI use recurrent policies with different auxiliary losses for RNN training. ESCP and PEARL are previous SOTA methods for these tasks. The comparative results shown in Fig. 8
We find that in Ant, Humanoid, and Walker2d, RESeL demonstrates significant improvement over all other methods. In Hopper, RESeL performs on par with ESCP, while surpassing other methods. Specifically, RESeL outperforms SAC-RNN by a large margin, further highlighting its advantages. Moreover, ESCP, EPI, and OSI use RNNs to extract dynamics-relater information, with ESCP having inferred embeddings highly related to the true environmental gravity. The ad advantages of RESeL over these methods suggest that the context encoder of REScL may extract not only gravity (see Appendix E.T but also other factors that help the agent achieve higher returns.
DMLAnt.aravty-22
DML WwLLHatCChetah.coo ggawity
DN_ flopper_sravttyee
DM, Hlumanoodddsgoo jwu_lmmmtrs_gavvtyyy graviyy
DM. WW__Walerrez.po graviiyy
7500
3000
6000
4000
5000
2000
2000
2500
1000
2000
1500
timestep
106
timestep
166
timestep
166
timestep
1c6
timestep
166
RESeL (ours)
SAC-MLP
SAC-GRU
ESCP
PEARL
EP
051
ProMP
Figure 8: Learning curves shaded with one standard error in dynamics-randomizz tasks.
Classic meta-RI. Tasks. We further compare RESeL. with baselines in other meta-RI. tasks, particu- larly the tasks possessing varying reward functions, sourced from [19 I.. In AntDir, Dir, and HalfCheetahVel. the robots' target moving direction target velocity are changeable and unobservable. Agents must infer the desired moving direction or target velocity from their histori- cal observations and rewards. Wind is non-locomotion meta-RL RL task with altered dynamics. We compare RESeL to previous me methods, including RL2 25), VariBad
19] and MF-RNN (TT.
The learning curves of these methods are shown in Fig. 9 In the AntDir and HalfCheetahDir tasks, we found that RESeL can converge within 5M steps. Our results show that RESeL demonstrates high sample efficiency and superior asympiotic performance in AntDir, CheetahDir, and HalfCheetah Vel. In Wind, RESeL. performs comparably to MF-RNN, as this task less complex. These findings suggest that RESeL effectively generalizes to POMDP tasks with hidden reward functions.
8

---

AntDirvv
Cirvv
HallCheetahVel-v0
Wind-v0
2400
4500
60
-200
return
1800
3000
40
1200
400
1500
20
600
6000
0
2
2
6
timestep
1e7
timestep
1c7
timestep
16
timestep
1e5
RESeL (ours)
MF-RNN
SAC-MLP
VariBad-Onpolicy
OffiaaiyO
Figure 9: Learning curves shaded with one standard error in meta-RL tasks.
Credit Assignment Tasks. A notable application of recurrent RL is solving credit assignment problem [T]] [5]. We also compare RESeL with MF-GPT [15] and MF-RNN [I] in
Key-to-Door
SUCC8SS rete
a credit assignment task, namely Key-to-Door Door. In this task,
0.5
it is required to assign reward obtained at the last-step an early action. We compared RESel. with algorithms MF-
GPT and MF-RNN. As did in [15] we test RESeL with credit assignment length of (60, 120, 250, 500]. the hardness of the task grows as the length increases. In this task, the methods are evaluated by success rate. The results are shown in Fig. TO The success rate of RESeL matches the previous SOTA MF- GPT in tasks with [60,120] lengths, closed to 100% success rate. For harder tasks with the lengths being [250, 500], RESeL reached higher success rates than the others. The results indicate that RESeL outperforms transformer-based methods in handling challenging credit assignment tasks.
100
200
300
400
500
(Easy)
Credit assignment length (Hardd)
Re.. (bum)
- MI-GPT
JINN
Figure 10: Success rate in the key- to-door task with different credit as- signment lengths.
Classic MuJoCo I Locomotion Tasks. Finally, we would like to discover how RESeL. performs in MDP tasks. We adopt five classic MuJoCo tasks and compare RESeL with previous RL methods, e.g., SAC [20]. TD3 [36], and TD7 43). As the current SOTA method for these environment, TD7 introduces several enhancements based on TD3, e.g., a representation learning method and checkpoint trick, which are not existing in RESeL. The results, listed in Table show that RESeL is comparable to TD7, showing only 4.8% average performance drop. This demonstrates that RESeL is effective enough to nearly match the performance of the most advanced MDP algorithm. In Hopper and Walker, RESeL even surpasses TD7 by a significant margin. By comparing RESeL. with SAC, we find RESeL. is superior to SAC in all tasks and can improve SAC by 34.2% in average. These results indicate that RESeL. can also be effectively extended to MDP tasks with notable improvement.
Table I: Average performance on the classic MuJoCo tasks at 5M time steps + standard error.
TD3
SAC
TQC
TD3+OFE
TD7
RESeL (ours)
HallCheetah--2 v2
14337
191
15526 1 697
17459 1 258
16399 + 164
18165 : 255
16730 432
Hopper.v2
3682
83
3167 485
3462 1 818
3423
584
4075 1 225
4408 5
Walker2d-v2
5078
313
56881
329
6137 + 1194
6379 + 332
7:997 + 454
5004 150
Ant-2
5589
758
4615 2022
6329
1. 1510
8547 1 84
10133
1. 968
8006 63
Humanoid-v2
5:133
245
6555 279
8361 1 1364
8951 :: 246
10281 : 588
10490
381
Average
6821
7109
8350
8779
10010
9532
5.3 Sensitivity and Ablation Studies
Sensitivity Studies. In this section, we analyze the impact of LRCE and LRother with experiments on the WalkerBLT-V task. Initially, we fixed LRother = 3 x 10 and varied LRCE. The final returns for different LRCE values are shown in Fig. Ta The figure demonstrates that the highest model performance is achieved when LRCE is between 5 x 10-6 to 10-5, Both larger and smaller LRCE values result in decreased performance. Notably, an overly small LRcE is preferable to an overly large one, as a small learning rate can slow down learning efficiency, whereas large learning rate can destabilize training, leading to negative outcomes. Next, we fixed LRCF = 10-5 and varied LRother. The resulting final returns are presented in Fig. 11b Here, the model achieves the highest score with T.Rother = 3 10-4, while other values for R.Rother yield lower scores. Figs. 11a and lb together indicate that the optimal values for LRCE and LRother are not of the same order of
9

---

2000
WalkerBLT-V-v., LR 10-4
2000
WalkerBLT-V-vv. LR 10
WalkerBLT-V-40. LRce-LRottee
1000
1500
return
retum
1000
1000
500
800
Log. my
Loc-oe
125-43
mt
Roghe
LRcr and LRower
(a) Fixing LRother, varying LRCE. (b) Fixing LRCE, varying LRother- (c) LRCE = LRother varying both.
Figure 11: Sensitivity studies of varied learning rates terms of the average final return.
HallCheetahaLT-P.00
HallCheetah@LTiV-0
WalkeralT-P-va
WalkerBLT-V--D
3000
2400
2000
2000
1800
1500
return
1000
1200
1000
600
11000
-1500
0
0
2
2
6
timestep
1e5
timestep
1e5
timestep
1e5
timestep
1e5
Mamba
RESeL-GRU
RESel Transformer
MF- (GRU)
SAC Jransffanforrmrr
Figure 12: Learning curves shaded with I standard error with different RNN architectures.
magnitude. Additionally. it can also be found from Fig. Tcc that setting LRcE equally significantly degrades policy performance. This emphasizes the importance of using different learning rates for context encoder and other layers. We extended the experiments from Figs. [laland and [[c to additional seven POMDP tasks, with results presented in Fig. 20 in Appendix E.6 The conclusions drawn from Fig. 20 are largely consistent with those from Figs. 11a and 11c. indicating that the impact of the context-encoder-specffee learning rate on training is relatively generalizable.
Ablation Studies. We then explore whether the context-encooder-peeifff learning rate applies to other RNN or Transformer architectures 9. We integrated a GRU ( or Transformer as the context encoder while keeping all other settings constant. Due to the high computational cost of training GRUs and Transformers on full-length trajectories, we tested the RESeL variant with these architectures on four classic POMDP tasks for 0.75M time steps. Figure 12 displays the learning curves for various variants. MF-RNN [ and SAC-Transformer (38] serve as baselines based on GRUs and Transformers from prior studies.
We observe that the context-encoder-spccffee learning rate also improves and stabilizes the per- formance of the GRU and Transformer variants. Our findings show that both RESeL-GRU and RESeL-Trransformer perform comparably to RESeL-Mamba on three out of four tasks. These re- sults suggest that the performance gains of RESeL are not solely attributed to an advanced RNN architecture, and that different arhitectures may be suited to different tasks. Additional sensitivity experiments and ablation studies on context length can be found in Appendices E.4 to E.6
6 Conclusions and Limitations
In this paper, we proposed RESeL., an efficient recurrent off- off-policy RL. algorithm. RESeL. tackles the training stability issue existing in previous recurrent RL methods. RESeL uses difference learning rates for the RNN context encoder and other fully-connected layers to improve the training stability of RNN and ensure the performance of MLP. Experiments in various POMDP benchmarks showed that RESeL can achieve surpass previous SOTA across a wider variety of tasks, including classic POMDP tasks, meta-RL tasks, and credit assignment tasks.
Limitations. We also notice that there are several noticeable limitations concerning this work: (1) We have shown that RESeL can even surpass SOTA methods some MDP tasks, but it is unclear how the RNN helps the policy improve the performance. (2) RESeL introduces one more hyper-parament i..., L.RCE- would be nice to explore the relationship between optimal LRCE and I. or automatically parameter-tume tunnnng strategy for LRCE-
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Fundamental Research Program for Young Scholars (PhD Candidates) of the National Science Foundation of China (623B2049) and the Jiangsu Science Foundation (BK20243039). The authors thank the anonymous reviewers and Ms. Qianqian Cheng for their constructive feedback on this paper.
References
[1] Richard Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, 1998.
[2] Henrik Bey, Maximilian Tratz, Moritz Sackmann, Alexander Lange, and Jöm Thielecke. Tutorial on sumpling-basee mlann-ng for automated driving. In Proceedings of the International Conference on Vehicle Technology and Intelligent Transport Systems, Prague, Czech Republic. 2020.
[3] Zachary Sunberg and Mykel J. Kochenderfer. Improving automated driving through POMDP planning with human internal states. IEEE Transactions Intelligent Transportation Systems, 23(11):20033-20885 2022.
[4] Jing-Cheng Pang, Si-Hang Yang. Xiong-Hui Chen, Xinyu Yang, Yang Yu, Mas Ma, Ziqi Guo, Howard Yang, and Bill Huang. Object-oriented option framework for roboties manipulation in clutter. In Proceedings of the IEEE/RSJ International Conference Intelligent Robots and Systems, Detroit, MI, 2023.
[5] Zhongyu Li, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath. Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control. CoRR, abs/2401.16889, 2024.
[6] Mikko Lauri, David Hsu, and Joni Pajarinen. Partially observable markov decision processes in robotics: A survey. JEEE Transactions Robotics, 39(1):21 40, 2023.
[7] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 98):1735- 1780, 1997.
[8] Junyoung Chung, Çaglar Gülgehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014.
[9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, Long Beach, CA, 2017.
[10] Siegmund Duell, Steffen Udluft, and Volkmar Sterzing. Solving partially observable reinforce- ment learning problems with recurrent neural networks. In Neural Networks: Tricks of the Trade Second Edition, volume 7700, pages 709-733. Springer. 2012.
[11] Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many pomdps. Proceedings of the International Conference on Machine Learning, Baltimore, MD, 2022.
[12] Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal M. P. Behbahani. Structured state space models for in-context reinforcement learning. In Advances Neural Information Processing Systems 36, New Orleans, L.A, 2023.
[13] Jake Grigsby, Linxi Fan, and Yuke Zhu. AMAGO: scalable in-context reinforcement learning for adaptive agents. Proceedings of the International Conference on Learning Representations, Vienna, Austria, 2024.
[14] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, 10s0222 2023.

---

[15] Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Luc Bacon. When do Transformers shine in RL? Decoupling memory from credit assignment. In Advances in Neural Information Processing Systems 36, New Orleans, LA, 2023.
[16] Fan-Ming Luo, Shengyi Jiang, Yang Yu, Zongzhang Zhang, and Yi-Feng Zhang. Adapt to environment sudden changes by learning a context sensitive policy. In Proceedings of the AAAI Conference on Artificial Intelligence, virtual, 2022.
[17] Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment probing interaction policies. In Proceedings of the International Conference Learning Representations, New Orleans, LA, 2019.
[18] Dongqi Han, Kenji Doya, and Jun Tani. Variational recurrent models for solving partially observ- able control tasks. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
[19] Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: very good method for bayes-adaptive deep RL via meta-learning. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
[20] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning, Stockholmsmässan, Sweden, 2018.
[21] Xinyue Chen, Che Wang. Zijian Zhou, and Keith W. Ross. Randomized ensembled double Q-learning: Learning fast without a model. In Proceedings of the International Conference on Learning Representations, virtual, 2021.
[22] Matthijs T. Spaan. Partially Observable Markov Decision Processes, pages 387-414. Springer
Berlin Heidelberg, Berlin, Heidelberg, 2012.
[23] Zhihan Yang and Hai Nguyen. Recurrent off-policy baselines for memory- based continuous control. CoRR, abs/21 12628, 2021.
[24] Daan Wierstra, Alexander Förster, Jan Peters, and Jürgen Schmidhuber. Solving deep memory POMDPs with recurrent policy gradients. In Proceedings of the International Conference on Artificial Neural Networks, Porto, Portugal, 2007.
[25] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL Fast reinforcement learning via slow reinforcement learning. CoRR, 1.02799, 2016.
[26] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In Proceedings of the International Conference on Machine Learning. Sydney, Australia, 2017.
[27] Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26), 2019.
[28] Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of the Conference on Robot Learning, London, UK, 2021.
[29] Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In Proceedings of the International Conference on Robotics and Automation, Brisbane, Australia, 2018
[30] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang. Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre, Ziyu
12

---

Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature, 575(7782)3350-333 2019.
[31] Steven D. Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. Popgym: Benchmarking partially observable reinforcement learning. Proceedings of the International Conference on Learning Representations, Kigali, Rwanda, 2023.
[32] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust re- gion policy optimization. In Proceedings of the International Conference on Machine Learning. Lille, France, 2015.
[33] Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. Robotics: Science and Systems XIII, Cambridge, MA, 2017.
[34] Haotian Fu, Hongyao Tang, Jianye Hao, Chen Chen, Xidong Feng, Dong Li, and Wulong Liu. Towards effective context for meta-reinforcement learning: an approach based on contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, virtual, 2021.
[35] Wesley Chung, Somjit Nath, Ajin Joseph, and Martha White. Two-timescale networks for nonlinear value function approximation. In Proceedings of the 7th International Conference on Learning Representationss New Orleans, LA, 2019.
[36] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the International Conference Machine Learning, Stockholmsmässan, Sweden, 2018.
[37] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Portugal, 2012.
[38] Ian Char and Jeff Schneider. PID-Inspired inductive biases for deep reinforcement learning in partially observable control tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, 2023.
[39] Nicolas Minorsky, Directional stability of automatically steered bodies. Journal of the American Society for Naval Engineers, 34(2):280-309, 1922.
[40] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. ProMP: Proximal meta-policy search. Proceedings of the International Conference on Learning Representations, New Orleans, LA, 2019.
[41] Xue Bin Peng. Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In Proceedings of the International Conference on Robotics and Automation, Brisbane, Australia, 2018
[42] Kate Rakelly, Aurick Zhou, Chelsea Fino, Sergey Levine, and Deirdre Quillen. Efficient off- policy meta-reinfforeement learning via probabilistic context variables. In Proceedings of the International Conference on Machine Learning. Long Beach, CA, 2019.
[43] Scott Fujimoto, Wei-Di Chang, Edward J. Smith, Shixiang Gu, Doina Precup. and David Meger. For SALE: state-action representation learning for deep reinforcement learning. In Advances in Neural Information Processing Systems 36, New Orleans, LA, 2023.
[44] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. In Advances Neural Information Processing Systems 33, virtual, 2020.
[45] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6988-698) 2022.
13

---

[46] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems 28, Montreal, Canada, 2015.
[47] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In Proceedings of the International Conference on Learning Representations. virtual, 2022.
[48] Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. In Proceedings of the International Conference on Learning Reprexentations, Kigali, Rwanda, 2023.
[49] Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico, 2016.
[50] lly Loshchilov and Frank Hutter, Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations, New Orleans, LA, 2019.
[51] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mchta, and João G. M. Araújo. Cleanrl: High-quality single-file implementations deep reinforcement learning algorithms. Journal of Machine Learning Research, 23:274:1-274:18 2022.
[52] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(86)2577--2655, 2008.
[53] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang. and Wojciech Zaremba. Openai gym. CoRR,
2016.
14

---

A Proof Proposition] !
The following proof process primarily based on the proof approach in [44 45). To the best of our knowledge, this is the first time the compounding error analysis method has been applied to study the properties of RNNs.
Let an RNN network (I,, h1) X x H + x H. be formalized as:
yr, h441 = 5 (x*,, ht);
where 0 is the network parameter, Ia is the RNN input, yr is the RNN output, he and h are the RNN hidden states, For simplicity, we denote
1/x (an, (xr,
h441 (1r,
We consider another RNN parameter 0', which is obtained by updating 0 by one gradient step. We make the following assumptions:
1. A1 (Lipschitz continuity): 1 and S both satisfy the Lipschitz continuity. There exist Kr E [0, 1 and Ky E R satisfying:
(1, h) - [x,,,, n1)l|
S K₁ | - WI1, Vo E N, e e - Kn | 1 1/1,, Va E h, e H.,
(A1.1)
| (zr, h) 1h (x, h')
(A1.2)
(x,,)) h) (x h')ll
Ky /|h - I/II, Vx h, h' H. s Ky 1 A' Vo E X, h, H.
(A1.3)
|rg (z, rg (x, h0
(A1.4)
2. A2 (one-step output difference constraint): The output differences of the RNN parameter- ized by 0 and 6' are bounded by a constant E R:
(x,,)))) h) 1h (a, (a. h))
Sc. Vz E H.
(A2.1)
(a,,)) h) - 1 (xx, m>}l se, Va ExE
(A2.2)
With the above assumptions, we now prove Proposition in following. Proof. Ve E X.
Xx h. N/ E H. we have
5h (xr,1))- 12 hi
5% (x, h²) 1h (z, h) + f (z, h) -1 (2,
5% (x, h²) 5 (z,, (z. |1h (x, h) (x, h2)
(2)
Eq. AL.2
Kn. |/h WIl 118 (x,h)-Jfth (2( hi]
Eq A2.1
Kh. |h W/1l +
Similarly, Va E X, h, E H. we also have
3 (z,h1) (2, s Ky 11/- h1ll +
Given an input sequence { x1. Ti. .r) with sequence length T + and an initial hidden state ho- let the output and hidden state sequences produced by f be {yo. V1.... yr and (lr,..., NTT+1 )-
*e only consider K₁, E [0, 1) as most RNN models satisfy this condition. GRU/LSTM and SSMs achieve this condition by adopting a sigmoid activation function and directly limits the value of the transition matrix of the hidden state, respectively. The RNN with a Kr. 1 could diverge during sequence forwarding.
15

---

Similarly, let the output and hidden state sequences produced by f be f36- vi..... v/ and (h1,...,,7.. We can measure the upper bound of the hidden state differences between fo and fo
l hill
|| (20
(20. ho) 5h (zo, ho)
(4)
112
hgll
[1 h1) - fh (I1, (11,
E4 2
Kn || - -t1ll +e
(5)
Eq
KAC tc.
Similarly, we have
2
[/3 - hu2ll 1 (x2, 12) - fh (12, h22)
Σ
Kic.
(6)
1=1
|16 hill 1 ho-1) - fh (tc-1,1//11
Ki,
i=U
Then we can derive the output difference at the t-th step 0): f (zt, hn)
1//e
2/1l
[ (ze, 1e)
Eq
Ky |l
+ hc
(8)
Eq
Ky Ki te.
As < K₁ < 1, Eq. 8 can be further simplified to
Kh 1i v'll s K2 1 - Kh
e + e.
(9)
Eq. 99 also holds for t = 0, i.e., 1lvo - vóol S c. Thus, we finally have
1 K
1lve - vill S Kg
6 + v2
(10)
K
B Average Output Differences Bound
Proposition 2. Following the setting in Proposition 7 the average output difference satisfies
Kg t + Kg (1+ 11(1 Kn)3%
1 Ktt1
1ive 2ill < 1 Kh.
Vt 2 0.
(II)
1=0
Proof. From Eq. 10 we have
llv.
v/ll
S
K
4 Kg
6
Kl.
Kr
K
As
K
8 T=0
K(+1
(1 - (1)
16

---

Algorithm 1: Training Procedure of RESeL
Input:
Environment F; Policy m6: Ensemble critic Qu: Target ensemble critic Qui Batch size BS; Entropy coefficient o;; Target entropy TE;
Initialize context-encoderr-seeiii policy optimizer and critic optimizer; Initialize an empty replay buffer B 4 @ and trajectory T + 0;
2
Initial context â 0, <- 0,f 0;
4 Initial hidden state h + 0;
t 2t- -0;
Obtain initial observation from E;
7 while max_ max_step do (z, h < (o ô, h):
Execute a, F;;
10
Observe new observation 0r+1, reward and terminal signal de
11
Add (or, dz. Tt, 0r+1) to
Update context data à +- dr, Ô t- or, r6
13
de is True then
Insert to B;
15
T 7+00: 0;
Reset context < 0, 0 +- 0:
17
Reset hidden state t- 0;
Reset F,, obtain new observation Oc+11 from E;
19
Sample data batch D = {T, i= 1,2...... with data count Σ, |-| > BS from B;
20
Obtain REDQ 21] critic loss with Q Qu: a, and D following Eq. (T9p
Update Y with the critic optimizer via one-step gradient descent; Softly assign & to vi;;
22
if 1%2 then
24
Obtain REDQ (1 policy loss with na, a, and following Eq. (0))
Update with the policy optimizer via one-step gradient descent; Automatically tune a to control the policy entropy to TE;
26
27
1+1+1
we can get
11 vill
1=0
Ky
6+ Ky Ky Kn t
K
K₁.
Ky
c K2 1 K7+1
(t+)(+++)) C.
1 Kh
C Algorithmic Details
C.1 Detailed Procedure of RESeL
The training procedure of RESeL is summarized in Algorithm 1 In the following part, we elaborate on the form of REDQ (1 losses the setting of recurrent RL. Let the critic Qv a, ô, â, f, hQ): & x 7 be mapping function from the current observation o and action a,
O x A O x A x R last-step observation ô, action å, reward f, and hidden state hę to a 8-dim Q-value vector and the next hidden state hg. The target critic Q ú, Ô, à, f. hQ) shares the same architecture to Qu but parameterized by v/. The policy (o, ô, û, hz)) O A x O x A x R - AA x H is mapping function from the current observation o. last-step observation à, action á, reward and hidden state hy to action distribution and the next hidden state h,,
17

---

Qurget, be the j-th column of Given batch of data D = { = ......., where ;; for: ao: ro. of ri. ....... -1. oz the i-th trajectory in the batch, L; the trajectory length. We then re-formalize t; to tensorized data: O; 04, a, E A O; E O à; E EA
IR_, i; E IR.,, where, for example, of O of the 1-th row of o,. 0 = a 0. P. = as the there no last step at the first time step. The policy receives the tensorized data and outputs action distributions. For simplicity, we formalize the outputs of the policy as the action sampled from the action distribution and its logarithm sampling probability:
ā, log Pt, hr + (o, o, á, F, 0), where à; E E "R',,h" E H. are obtained autoregressively: a log P. h f- Th (o. 08. a., +., 0), a log p h + (o). of. a F. h
(12)
(13)
a log p [+ + (o 6 af. H4, M.) Similarly, the critic model receives the tensorized data and outputs 8-dim Q-values as follows:
Q h/ + Q (o:, a,, o, a. fz, 0),
(14)
where Q: E IR 1488 , H are obtained autoregressively:
Q h1g - Q+ a. o.. a. 0),
Q h2 - QN (o) a o, at fl 12))
(15)
1 h Q (0%. at, 6%, at, F2, 12)
Note that, in Mamba, the process demonstrated in Eq. 13 and Eq. 15) can be parallelized.
In order to compute the target value, we first extend the tensorized data
o + {or O
o + {ō,, o L. à + (á. a, 1-11 }.
(16)
F. - (f., r
r. (0,r:).
where of is the next observation of the last step. of a/ and r/--1 are the observation. action, and reward at the last step, respectively.
The target value can be obtained:
i log Pr 11 11+t1 x- mo ,,, a. ft,
(17)
I++1 r.t + (Q (0, a trrger , aaa.t.
a. 1 .t.00 .0 log Prargets)
1/2
Then, we sample a set M of 2 distinct indices from {0, 1,... 7}. Let Q tit
Quigee, We choose the minimum value from (Ortigen E MM:
Quinee. min JEM
tu Qungen
(18)
where Quirget, B.ct1 We then let y, be the last L, rows of Quinge,
yi = ++ Outngen.
Li + 1].
The REDQ critic loss can be written as
2 Q
LQ
(19)
>
D
L
2
18

---

where Q: is obtained following Eq. ll2 denotes L2-norm. As for the policy loss, we first obtain å, and log P. via Eq. 12 We can get the critic value Q corresponding to the new action a, following Eq. (14):
h2 + Qu ā o, a, F,,
can get the policy loss:
L
sum (Ô: - log
(20)
3 D. L,
where denominator of results from our ensemble size being 8, requiring the averaging of the outputs of the eight crities to compute the policy loss.
C.2 Sample Stacked Batch from Replay Buffer
trajectory index
0
2
4
6
stacked batch
[s, 0,
10
dajs tim
sample batch
22
hidden reset flag
replay buffer
K-step zeros
kernel convid
Figure 13: Illustrating for sampling a stacked batch from replay buffer.
In RESeL, collect several full-length trajectories from the replay buffer batch for policy training. However, in many environments, trajectory lengths could vary lot, making it impossible directly concatenate them into single tensor for subsequent computations. Typically, we pad trajectories to the same length with zeros, resulting in tensor of shape [B. L, C). where B is the batch size, the sequence length, and C' is the tensor dimension. Additionally, we introduce mask tensor of shape [B, L, 1], where elements equal to indicate valid data, and Os indicate padding. We then use modified mean operation:
mask_mean (z, mask) = Σ ( x mask)) sumnkk))))mmmm))))
to prevent padding from biasing the mean. This approach allows for GPU parallelization of all trajectories in the batch, though calculations for padded Os are redundant. If trajectory lengths vary greatly, the proportion of padding can become significant, severely reducing computational efficiency, For instance, in batch with one trajectory of length 1000 and ten trajectories of length 50, there would be 1500 valid data points and 9500 padded points, substantially lowering efficiency.
resulting in Aut-of-GGU-memeee
To address this issue, we adopt the batch sampling method illustrated in Fig. 13 stacking shorter trajectories along the time dimension. This requires the RNN to accept hidden reset flag, which we insert at the beginning of each trajectory to reset the hidden state. Additionally, we insert K steps of zero data between trajectories prevent convolutionld from mixing adjacent trajectory To implement this sampling strategy, we modify the original Mamba implementation adding an input denoting hidden reset flag. When occurring the hidden reset flag
data, as shown in Fig. 13
19

---

with value of 1, the hidden state will be reset to zero. The introduction of hidden reset flag will not influence the parallelizabiltyy of Mamba but can avoid potential redundant computation.
D Experiment Details
D.1 Descriptions of the Environments
D.1.1 Classic POMDP Tasks
These classic POMDP taske t are developed using PyBullet locomotion environments. The envi- ronments include AntBLT, HopperBLT, WalkerBLT, and HalfCheetahBLT. Partial observability is introduced by obscuring parts of the observation. For each environment, two types of tasks are created: one where only joint velocity is observable and another where only joint position observable. We denote these as "-V" for joint velocity information and "-P" for joint position information. In such settings, the agent must reconstruct the missing information, whether velocity or position, based on historical observations.
D.1.2 Senamics-Randomizze Tasks
The Dynamics- Dynamics-Randomiizz Randomizdd Task & environments are based on the MuJoCo environment [37], fol- lowing the training settings in [16]. In these tasks, the environment's gravity is altered using an open-source tool The gravity is sampled as following procedure: We first sample a value a E R uniformly from 3, 3]. Let the original gravity in MuJoCo be Porigias the new gravity is obtained as:
Inew 1.5% Jorigin.
(21)
Before training, we sample 60 values of Inew Of these, 40 are used for training and 20 for testing the learned policies. In these tasks, the agent must infer the gravity based on historical interactions with the environment and adjust its policy accordingly.
D.1.3 Classic meta-RL Tasks
These meta-RL. taske have been utilized various meta-RI. algorithms 42 19
Specifically, in these tasks, the reward function is modified in the Ant and HalfCheetah MuloCo environments. In the 'Vel' tasks, the agent is rewarded based on proximity to a desired velocity. In the 'Dir' tasks, the agent must move in desired direction. The desired velocity and direction information are reflected only in the reward function. In the Wind environment, an external, unobservable disturbance factor (wind speed) influences state transitions. The agent must navigate to goal under varying wind conditions. these tasks, the agent uses past interaction data, including rewards, to infer the underlying reward functions or transition disturbance factors.
D.1.4 Key-to-Door
The Key-to-Door environmen consists of three phases. In the first phase, the agent spawns at random location the map, and key will appear at another location on the map, the agent can touch the key. In the second phase, the agent will have the task of picking up apples. In the third phase, if the agent has touched the key the first phase, it can receive reward: otherwise. it cannot.
The action space is four-dimensional, encompassing the actions of moving forward, backward, left and right. The reward structure is dual-faceted: it includes rewards garnered from the collection of apples during the second phase and the ultimate reward obtained upon door opening in the final phase. The transition function is defined by the positional changes of the agent subsequent to the execution of any of the four actions, with stage transitions dictated by an intrinsic timer, which facilitates a shift to the subsequent stage upon the expiration of the countdown. The optimal policy should locate and touch the key in the initial phase, which is served for the final reward acquisition upon door opening
httpps//ggchab..com//mmmmss////s
parppaarrappaaa _nvs
20

---

in the third phase, while maximizing apple collection during the second phase. The credit assignment length Fig. 10 denotes the time period staying at the second stage.
D.2 Baselines
In the section, we give brief introductions of the baseline algorithms we comparing in our work. We first introduce the baselines in the classic POMDP tasks Fig. 7
Variational Recurrent Model (VRM) [8] VRM utilizes variational RNN [46] to learn transition model, and feeding the RNN hidden state into an MLP policy to obtain the action.
GPIDE (GPIDE-ESS) 38]. Inspired by the PID controller's success, which relies on summing and derivative to accumulate information over time, GPIDE propose two history encoding architectures: one using direct PID features and another extending these principles for general control tasks by consisting of a number of "heads" for accumulating information about the history.
Recurrent Model-free RL (MF-RNN) [T. ME-RNN proposes several design considerations in recurrent off-policy RL methods. MF-RNN analyzed the best choice concerning the context length, RL algorithm, whether sharing the RNN between critic and policy networks,
etc. We used the best results reported in [1] in Fig. 7
The baseline results in Fig. 7 come from two sources: (1) The evaluation results provided in | including MF-RNN, PPO-GRU, SAC-MLP, TD3-MLP, VRM, and A2C-GRU; (2) The results provided in 38
consisting GPIDE-ESS and ArC-rransformer.
The baselines compared in Fig. 8 elaborated as follows:
Environment Probing Interaction Policy (EPI) 17]. This baseline aims to extracting features from the transition. A context encoder is obtained to enable transition model to predict the future state through minimizing the following loss:
CDM
Egf
[| ar, 2/; (r) + -1411/1
(22)
where Tst, Gt, 21: 0T) is transition model parameterized by 0r, =f is the output of the context encoder. In addition, the transition model is also optimized as described in Eq. (22)
Online system identification (OSI) [33. OSI is structured by a context encoder which is used to predict the parameters of the environment dynamics. The architecture of OSI is the same as Fig. OSI adopts a supervised learning loss function to train the context encoder:
C = E || - -allg- where c; is the dynamics parameter such as gravity, 2f is the embedded context.
This approach is design to improve both the the robustness and sensitivity context encoder, based on a contrastive loss. In order to improve the policy robustness in environment where dynamics sudden changes could occurs, ESCP truncate the memory length in both training and deployment phase.
Environment sensitive contextual policy learning (ESCP) 16
Proximal Meta-Policy Search (ProMP) 40 This method learns a pretrained policy model in training phase and finetune deployment phase with few gradient steps. The reported performance of ProMP is obtained by the policies after finetuning.
Soft actor-critic with MLP network (SAC-MLP) (20). An implementation of the soft actor-critic algorithm with MLP policy and critic models.
Probabilistic embeddings for actor-critic RL (PEARL) 442. The context encoder of PEARL
is trained to make a precise prediction of the action-value function while maximize entropy. The loss of the context encoder can be written as:
PEARL CE
Lerte + DKu[p(2 (7) || (0,,
https: hattge///gchhb.b.cca///aaaarrrap
21

---

where Centi is the critic loss in SAC. DxL (p((TT || | I)] denotes the KL divergence between distribution of the embedded context and a standard Gaussian distribution. In p(=[r), denotes the output of the context encoder based on given trajectory. PEARI. is implemented with the officially provided codes
SAC-GRU. This is an upgrade version of SAC with GRU models. It use standard SAC algorithm to optimize an RNN policy.
The baselines compared in Fig.
RL [25]. RL a meta-RL. algorithm that trains meta-learne to efficiently adapt to new tasks by leveraging past experiences. It involves two nested loops: inner loop where the meta-leamer interacts with the environment to learn task-specific policies, and an outer loop where the meta-lcarner updates its parameters based the performance across multiple tasks. The inner loop is accomplished by an RNN.
VariBad 19 VariBAD (Variational Bayes for Adaptive Deep RL) also meta-RL method that uses variational inference to enable fast adaptation to new tasks. It employs Bayesian approach meta-learning, inferring fask-specific posterior distributions over the parameters of the policy and value function. Specifically. it first learns an RNN variational auto-encoder extract the sequential features of the dynamics or tasks. The policy then receives the current state and the output of the context encoder to make adaptive decisions.
The results of the baselines Fig. 9 are sourced from
MF-GPT [15]. i.e. the baseline in Fig. 10 denotes to train a Transformer network, i.e. GPT-2, to solve the memory-baa tasks. The baseline result the baselines in Fig. are from [5
D.3
Implementation Details
D.3.1 State Space Models
In our implementation, we use Mamba to form the context encoder, as it supports processing sequences in parallel. Therefore, we will first introduce state space models and Mamba before discussing the network architecture.
State Space Model (SSM) is sequential model 47 14 48 commonly used to model dynamic systems. The discrete version of an SSM can be expressed as:
h = A + BIg, Y1 Ch+
(23)
A (x)( B -AAA
I) AB,
where A, A. B. and C are parameters; Fe. ye- and h2 represent the input, output, and hidden state at time step 1, respectively. A and B are derived from A, A, and B.
Mamba [1] is an RNN variant that integrates SSM layers to capture temporal information. In Mamba, the parameters A, B, and C are dependent the input æ,, forming an input-dependent selection mechanism. Unlike traditional RNNs which repeatedly apply Eq. (23) p in a for-loop to generate the output sequence, Mamba employs a parallel associative scan, which enables the computation of the output sequence in parallel. This method significantly enhances computational efficiency.
D.3.2 Network Architecture
Pre-encoder. The pre-encoders consist of one-layer MLPs with consistent output dimension of 128.
The context encoder is comprised of one or more Mamba blocks, each flanked by two MIPs. Typically, the MLP before the Mamba block consists of single layer with 256 neurons, except for the Key-to-Door task, where three-layer MLP with 256 neurons per layer is employed.
Context Encoder.
Except in Key-to- Door task, single Mamba block is utilized, with state dimension of 64. The kernel size of the convolution layer fixed at Additionally, for these tasks, a position-wise
M htatpp://gtthbbbcccc///aeerrlees http:::///gthhbb..oom///m///////
22

---

feedforward network, used by Transformer 19, is appended to the end of the Mamba block to slightly enhance stability during training. In Key-to-Door tasks, the convolution layer in Mamba omitted, as the focus shifts to long-term credit assignment, rendering the convolution d unnecessary for capturing short-term sequence features. Conversely, in Key-to-Door tasks with a credit assignment length of 500, four Mamba blocks are employed to improve the long-term memory capacity.
The MLP after the Mamba block also comprises single layer with an output dimension of 128.
MLP Policy/Critic. As detailed in Sec. 4.1 both the MLP policy and critic consist MLPs with two hidden layers, each comprising 256 neurons.
Activations. Linear activation functions are employed for the output of the pre-encoder, context encoder, and the MLP policy/critie. For other layers, the ELU activation function 49 is used.
Optimizer. We use AdamW (0 for RESel. training.
D.3.3 Hyperparameters
The hyperparameters used for RESeL is listed in Table We mainly tuned the learning rates, batch size for each tasks. 7 and last reward as input
determined according to the characteristic of the tasks. The batch size in Table 2 (1000 or 2000) is much larger than that used in previous MLP-based SAC algorithms (128 256) 20)] This is because train on least complete trajectory at time, and in most environments, the maximum trajectory length is 1000. Thus, batch sizes 1000 or 2000 correspond to
trajectories of maximum length.
Table 2: Hyperparameters of REScL.
Attribute
Value
Task
2 x 10
classic MuloCo and classic meta-RL tasks other tasks
context encoder learning rate LRcs
10-5
6 x 10 3 x 10
classic MuJoCo classic meta-RL. tasks
other learning rate LRother for policy
other tasks
2 x 10
classic MuloCo and classic meta-RL tasks
other learning R.atter value
10-3
other tasks
0.9999
Key-to-Door
0.99
other tasks
True False
classic meta-RL tasks
last reward as input
other tasks
2000
classic POMDP tasks
batch size
1000
other tasks
target entropy
1x Dimension of action
all tasks
leamning rate of a
10
all tasks
soft-update factor for target value network
0.995
all tasks
number of the randomly sampled data
5000
all tasks
Hyperparameter Determination Some hyper-parameter (discount factor, reward as input) were chosen based on the nature of the environments, while others (Context encoder LR, policy and value function LR, and batch size) are determined via grid search:
We fixed the learning rate of the context encoder to be 1/30 of the policy learning rate in all tasks. We aim to have the action variation caused by RNN updates to be comparable to that caused by MLP updates, thus fixing the learning rate ratio between the context encoder and MLP. After conducting parameter searches in several environments, as shown in Fig. 20 we find the 1/30 ratio performed well, so we fixed this ratio across all environments.
Context Encoder LR:
We mainly referred to the default learning rate settings in CleanRI. [51], using policy learning rate of 3 10-4 and an action-value function learning rate of 10 10-3 We conducted a grid search between the aforementioned learning rates and those reduced by a factor of 5 (aiming to further stabilize training). We found that smaller
Policy and Value Function LR:
23

---

learning rate was more stable in tasks where the partial observability is not very significant
(e.g., MDPs like classic MuJoCo).
Discount Factor (c): For general tasks, we used common 7 value of 0.99. For environ-
ments requiring long-term memory (e.g.. Key-to-Door), we used a larger of 0.9999.
Reward as Input: The Classic Meta-RL tasks require inferring the true reward function based real-time rewards. Thus, also included real-time reward feedback as imput to the policy and value in these tasks.
conducted a grid search for batch size, mainly exploring sizes that were 1x 2x the maximum trajectory length. We found that in classic POMDP tasks, larger batch size performed better. In other tasks, the advantage large batch size was not significant and could even reduce training speed.
Batch Size:
E More Experimental Results
E.1 Visualization
22
33VE-dmm
Grouiyy
TSME
Figure 14: The t-SNE visualization of the outputs of the context encoder in Halfcheetah with different gravity.
In order to explore what the context encoder has learned, we attempt to conduct in in-depth analysis on the embedded context data. We conducted experiments in the environment of Halfcheetah with varying gravity. We uniformly sample gravity acceleration within the range of [2.90, 33. 13//? resulting in 40 environments with different gravities. For each environment, we used policy learned by RESeL to collect one trajectory, with a length of 1000 steps. Ultimately, we obtained 128ddimensional vector, was saved. Subsequently, we performed t-SNE [52] processing on the 40x 1000x 128 data to map the high-dimensional 128-dimensional data into 2-dimensional data which can assist in visualization analysis.
data of size 40x 1000, where each step's 's output of the encoder,
Finally, we presented the results in Fig. 14 In this figure, the colorbar represents the gravity acceleration magnitude. The x and y axes the figure represent the two-dimensional data results after t-SNE mapping. From the figure, we can distinctly observe the non-random distribution characteristics of colors from left to right. The different data represented by cool and warm colors are not randomly intermingled but rather exhibit certain clustering characteristics. We can find that the embeddings are strongly correlated with gravity acceleration, indicating that RESeL. has recovered the hidden factor of the POMDP environment from the historical interaction data.
E.2 Time Overhead Comparisons
We find the computational efficiency is also a common issue among previous recurrent RL methods, es- pecially in the context of recurrent off- off-policy RL. This issue mainly comes from the poallaaalle nature of traditional RNNs. our work, we tackle this problem by using the SSM structure, i.e. Mamba to improve the computational efficiency. Mamba is able to process sequential data in parallel based on the technique of parallel associative sean. the following parts, we will analyze the time cost of Mamba compared with that of GRU.
24

---

one-step update time (ms)
60
40
20
Mamba
GRU
context encoder type
Figure 15: One-step update time with different types of context encoder.
To evaluate the improvement in training speed of the policy, we conducted experiments utilizing the HalfCheetah environment within Gym [53]. Maintaining consistency across all training con- figurations, we substituted Mamba in the context encoder with MLP and GRU, ensuring identical input/output dimensions for policy training. Each experiment contains 10,000 time steps, during which we measure the average training time per step. Policy training employed full-length trajectories, Le. a fixed sequence length of 1000. The time costs associated with different network architectures are depicted in Fig. 15
The results indicate that Mamba possesses significantly lower time costs compared to GRU, amounting to approximately 25% of the time taken by GRU. Mamba's 's time cost surpasses that of MIPP by approximately 80%; nevertheless, it still presents substantial speedup relative to GRU. These results are based on utilizing a single layer/block.
Table GPU utilization, memory, and time cost with various neural network types. Time denotes the time cost for each update iteration, in HalfCheetah-v2. Normalized time is normalized with the corresponding GRU time cost.
Network Type
GPU Utilization
GPU Memory (MB)
Time (ms)
Normalized Time
Layer/Block
FC
35%
1474
9.5
14.4%
Mamba GRU
33%
1812
17
25.8%
58%
1500
66
100.0%
Layers/Blocks
FC
37%
1496
9.8
9.3%
Mamba
38%
1840
22
21.0%
GRU
75%
1558
105
100.0%
3 Layers/Blocks
FC
37%
1500
10.5
7.1%
Mamba
40%
1912
29
19.7%
GRU
77%
1592
147
100%
Layers/Blocks
FC
38%
1502
5.7%
Mamba
43%
1982
34
17.7%
GRU
78%
1612
192
100.0%
Time Overhead Comparisons with Different Number Layers/Blocks. In Table the GPU utilization and time costs for different numbers of layers are detailed, complementing Fig. [5] The results indicate that the normalized time for Mamba decreases as the layer count increases, suggesting that Mamba becomes more computationally efficient with more blocks, compared with GRU. Additionally, Mamba consistently uses less GPÚ resources than GRU, further demonstrating its superior scalability and efficiency.
25

---

network forward
network backward
30
Mamba
30
GRU
time (ms)
20
time (ms)
20
10
10
0
1000
2000
3000
4000
5000
0
1000
2000
3000 4000
5000
sequence length
sequence length
Figure 16: Network forward/backward time overhead for GRU and Mamba layers.
Time Overhead Comparisons with Different Sequence Lengths. To evaluate how Mamba scales with varying sequence lengths, we compared the forward and backward time costs of Mamba and GRU using artificial data. The results, presented in Fig. T6 show the sequence length the horizontal axis. The left panel illustrates forward inference time, while the right panel depicts backward inference time. The time cost for GRU increases linearly with sequence length in both forward and backward scenarios. In contrast, Mamba's 's time cost remains relatively constant, showing no significant growth in either scenario. These results highlight Mamba's computational efficiency and scalability with respect to sequence length.
E.3 More Comparative Results
E.3.1
POMDP Tasks
Table 4: Average performance the classic POMDP benchmark with gravity changes at 1.5M time
steps + one standard error over seeds.
PPO-GRU
ME-RNN
SAC- SAC-Tranffrrmer
SAC-MLP
TD3-MLP
GPIDE-ESS
VRM
A2C-GRU
AItBLT-P.wd
352
11:77
2507
AuBLT--va
1971
168
1778
1017
2900 170
148
-1817
HulChheuaBBL
2078
195
177 115
HogperBLT
2700
100
1763
140
HooperaLT
2480
438
126
425
2537
1.67
475
WalkerfLLT P0
2284
136
840 272
1150
36
3901
425
160
30
17
215
1..
2471
1053
1482
5.6
264
In Table the corresponding final returns in Fig. are listed detail. We can find that RESeL makes notably enhancement and surpasses the previous SOTA (GPIDE) by 22.9% in terms of the average performance.
E.3.2 Dynamics-Randomizzz Tasks
Table 5: Average performance on the MuJoCo benchmark with gravity changes at 2M time steps + one standard error over 6 seeds.
RESel. (ours)
SAC-MIPP
SAC-GUU
FSCP
PEARI.
EPI
ProMP
Ant- gravity-v2
5949
314
1840
358
758 18
3103 478
1065 J: 293
854 1:2
878 62
1243 1:
HallCheetahe gu g
8705
T33
6225 2: 1273
7022 2:051
7805 = 91
5453 1: 19
6053 1: 582
6874 1:: 147
865 : 101
Hopper gravity-x2
2846
191
192
1680
2680
369
2171
4: 337
2153 08
1756
252
6360
3084
Walker2d-gree gravviity--2
5866
266
3242 +
2098
270
2080 73
4284 + 174
1901 4: 242
1467 224
271 +
Average
5945
3207
2922
11:6
3964
2735
2899
615
In Table 5 the corresponding final returns in Fig are listed in detail. We can find that RESeL makes
notably enhancement and surpasses the previous SOTA (ESCP) by 43.4% terms of the average performance.
26

---

Table Average performance on the classic MuJoCo benchmark at 300k, 1M, and 5M time steps, over 6 trials + standard errois.
Time step
TD3
SAC'
TQC
TD3+FFE
TD7
RESel. (ours)
300k
7715 1: 633
8052 1: 515
7006 + 891
11294 247
15031 1 401
94561:212
HaliCheetah-v2
1M
10574
4: 897
10484 659
12349 + 878
13758 544
17434 155
12327 500
SM
14337
1491
15526 697
17459 1: 258
16596
164
18165 +: 255
16750 432
300k
1289 768
2370 + 626
3251 +: 2.4461
1581 = 682
2948 464
3480 22
Hopper-v2
1M
3226 315
2785 1 634
3526 244
3121 : 506
3512 315
3508 :: 522
5M
3682 1: 83
3167 485
3462
818
3423 584
4075 225
4408 1:
300k
1101 386
1989 :: 500
2812 4: 838
4018 :: 570
5379 328
4373 1: 144
Walker2d-v2
IM
3946 + 292
4314 + 256
5321 + 322
5195 = 512
6097 +: 570
5410 + 176
5078 343
5681 329
6137 #: 1194
6379 +: 332
7397 454
8004 150
300k
1704 655
1478 354
1830 572
6348 :: 441
6171 831
4181 +: 438
Ant-v2
IM
3942 + 10330
3681 + 506
3582 + 1093
7398 = 118
8509 = 422
6295 + 102
5M
5589 758
4615 + 2022
6329 = 1510
8547 84
10133 +: 966
8006 63
300k
1344 365
1997 = 483
3117 910
3181 = 771
5332 714
4578 + 509
Humanoid-v2
IM
5165 145
4909 = 364
6029 531
6032 = 334
7429 153
7280 +: 168
5M
5433 + 245
6555 + 279
8361 = 1364
8951 = 246
10281 + 588
10490 381
E.3.3 Classic MDP tasks
Table 6 provides comprehensive training results for MuJoCo MDP environments, extending the data from Table Specifically. the table includes results recorded at 300k, IM, and 5M time steps. Baseline results are primarily sourced from [43]. In environments such as Hopper, Walker2d, and Humanoid, RESeL achieves or surpasses the previous SOTA performance. Notably, RESeL may underperform compared to previous baselines at the 300k time step. However, its asymptotic perfor- mance eventually matches or exceeds the SOTA. This suggests that incorporating RNN structures might slightly reduce sample efficiency initially, as the observation space expands to include full historical data, requiring more optimization to find optimal policies. Nevertheless, RESeL ultimately leverages the enlarged observation space to achieve superior asymptotic performance.
E.4
Ablation Studies on Context Length
AntBLTAA-0
HalfCheetah8LTA4v00
HopperBuTAv-vo
WalkerBLT-VAvO
3000
2000
2400
2000
1500
1500
1000
1600
1000
1000
500
800
500
1000
0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
timestep
1c6
timestep
1e6
timestep
1e6
timestep
1c6
Tull trajectory
trajectory segment
Figure 17: Ablation studies on the context length.
Since trajectory length (or called context length) in training data is often a critical parameter in previous studies (1I) our work has emphasized the importance of using complete trajectories. Training with trajectory segments but deploying with full-trajectory length can Iead to distribution shift issues. To investigate this, conducted experiments using full trajectory data versus trajectory segments (64 steps). The results are illustrated in Fig. 17
In general, models trained with full trajectory data perform better than those trained with trajec- tory segments. However, in the A and HalfCheetah-V V environments, the performance gap is smaller. The key difference in these tasks is the longer trajectory lengths, especially in HalfCheetahBLLT-V where the trajectory length is fixed at 1000 steps. The data in these tasks are usually cyclic, while in other tasks, agents may drop and terminate trajectories before achieving stable performance. We suspect that strong cyclic data can mitigate distribution shift issues, as trajectory segments containing complete cycle can effectively represent the properties of the full trajectory.
27

---

E,5
More Comparisons of Policy Performance with different RNNs
stachastic policy retun
3000
HallCheeeaaBLC-P-4000
HellCheeela@@LTNV-00
2400
WalkeraBT-P-40
2000
WalkerBBTTVVvv
2000
2000
1800
1000
1000
1200
1000
0
600
500
-1000
2
6
timestep
1e5
timestep
le5
timestep
1e5
timestep
1e5
RESaL-Mamba
RESeL-GRU
RESeL--ranstorman
Figure 18: architectures.
Learning curves in terms of the stochastic policy performance with different RNN
WalkerBLT-V-vO, LRcE=5 x 10-6
2000
1500
return
1000
500
6.0E.08
1.0E-05
6:0E-05
3.0E-04
6.0E 04
12E-00
LRather
Figure 19: Sensitivity studies of the context-encoder-specfii learning rates in terms of the average final return.
The learning curves of the exploration policy with difference RNN architectures depicted in Fig. 18. Compared with Fig. 12 Fig. T8 includes evaluation with exploration noise added to the actions. In Fig. [8] CESeL-GRU shows substantial improvement RESeL-Mamba than that in Fig. suggesting that RESeL-GRU might perform better in tasks involving action disturbances.
E.6 More Sensitivity Studies
Figure 19 serves as a supplementary panel to Fig. In Fig.
we fixed LRCE x 1 and varied LRother from x 10- to 1.2 10-3 The results suggest that the optimal LR is still 0.0003, which significantly larger than LRCE- This finding further supports that the optimal learning rates for LRcE and LRother should be different and not of the same magnitude.
The learning rate sensitivity analysis on the remaining seven POMDP tasks presented in Fig. 20 As shown, the conclusions in Sec. [.3 still hold: when different learning rates are used for the MLP and RNN components and the MIP learning rate fixed, excessively low or high RNN learning rates result in inefficient training. However, the optimal RNN learning rate generally falls between 5.0 x 10 and 1.0 x 10 If the MLP and RNN share the same learning rate, their performance rarely surpasses that of the setting with different learning rates, with the only exception being the HallChectahBLLT-P.. environment, where a single learning rate slightly outperforms the tifferent-learnin--nal setting.
28

---

3200
2400
retum
1600
800
0
AntBLT-P-vO
AntBLT-V-vO
3000
return
2000
1000
0
HalfCheetahBLT--P.00
HalfCheetahBLT-vv.00
3200
2400
retum
1600
800
0
HopperBLT-P-v0
HopperBLT-v-v0
2400
retur
1800
1200
600
WalkerBLT-P-vO
WalkerBLT-V-vO
LRc = 0.0, LR Lother 3.0 14
LRcE 5.0 x 14
LRomer 3.0 10
LRcE 5.0 10-7 LRomer 3.0 10 -
LRcr 1.0 -
Lotner 3.00 10
LRcE = 1.0 x 10 LR = 3,0 10
LRCE = 3.0 x 10 6 L Rother = 3.0x 10-6.
LRc 5.0 x 10
LRother = 3.0
LRCE 6.0 10
LRthhr = 1000
LRCE = 1.0 x 10
LRather 3.0 x 10
LRCE = 1.5 x 10 5 LRoter 1.5x
LRCE = 5.0 x 1
LRother = =3.0 10-4
LRCE 3.0 x 1
LRotner = 3.0 10-5
LRcE 1.0 -4
LRothr 3.0 x 10-4
LRcE 6.0 10
LRothr 6.0 10-5
LRCE = 3.0 x 10-4 LR 3.0x
LRcE = 6.0 10 LR 6.0 x -4
Figure 20: Sensitivity studies of the context-encoder-speee learning rates in terms of the average final return in eight POMDP tasks. The variants with the highest final return are marked with *.
29

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Sec. d
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification:
The proof of Proposition can be found in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide a complete algorithm procedure in Appendix model architecture in Sec. 1. and prppeeppraaeterss in Appendix D
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification: The codes and corresponding hyper-parameters can found https
//github. com/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssicnyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the experiment details and syper-parameters in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Each result of RESeL is repeated with 6 distinct random seeds. The standard errors are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
32

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have analyzed and reported the information the computer resources in Sec. 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::///pocrrss.cc/p/pa////hhht
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of recurrent reinforcement learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
35


Reviews:
[{"review_id": "Fcl1lLhFfu", "reviewer": "Reviewer_89BQ", "summary": "The paper contributes to the important and long-standing issue of representing latent state information for successfully finding optimal RL control policies in the context of partially observable Markov decision processes (POMDP). The major contribution is a newly composed RL algorithm (RESeL), which, by design, enables a thorough study of the impact and importance of the actual learning rate (LR) used for the recurrent (latent) context encoder part of the solution architecture. The author highlight the fact that current state-of-the-art (SOTA) methods commonly use an equal LR for training both the latent representation and the policy/critic networks. \n\nTheir idea of separating the LR between recurrent CE and policy networks, as well as the overall performance benefits w.r.t current SOTA algorithms, is demonstrated by an extensive survey of selected POMDP tasks and benchmarks. These experiments together with theoretical arguments let the authors conclude that using a single  (and often too large) LR parameter approach of previous methods leads to sub-optimal performance results in solving POMDP tasks.", "questions": "l. 127: Why is the last-step observation also needed as input to the MLP if both current obs and context-encoded obs are already fed into the  NN? According to Fig. 2, the last-step obs is fed into the CE only (but not into the MLP)\n\nl. 132: Is there any justification/reasoning behind the choice of the MLP architecture, in particular why a relatively large capacity (2 x 256 neurons) was chosen? Is there an estimate of how much this choice impacts the overall performance results of the policy training?\n\nl. 163-164: It seems to me that the claim of Proposition 1 and the subsequential statements (\"average variations in the RNN output ... are amplified\") are not a novel discovery but have been studied and found by others in similar or different contexts of RNN before. In that case, it would be good to provide corresponding citations of pior art or - if the others believe this is the first time such a claim was made - highlight the significance of their finding. \n\nl. 165-166: Please provide further arguments/proof, why the effect of Proposition 1 can be mitigated by smaller learning rate (even if such claim sounds plausible, it would be good to refer to previous work or more detailed reasoning). In other words: how does the learning rate affect the result of ||y - y'|| as a result of Propsotion 1 (it is likely related to epsilon in the formula but if so this relation could be mentioned more explicitly)?\n\nl. 167-168: Similar remark as before: this sentence remarks a general claim about MLPs which should either be backed up by reference citations or by stronger reasoning or explanations.\n\nl. 195: Given the typical level of stochasticity or dependency on random intial states, a trial number of 6 seems relatively low to incorporate statistical fluctuations in the return evaluation, in particular for the following ablation studies. Have the authors made sure that their reported results are not prone to larger statistical errors? What in detail is included in the choice of \"different random seeds\"? Does it only refer to the initial weight setup of the RNNs/MLPs or does it reflect varying intial conditions of the environemnts? Do all environments provide deterministic rollouts?\n\nFigure 3: Why are error bars only visible for the brown curves/datapoints?\n\nl. 218: How is \"action variation after one-step update\" as a function of rollout steps defined in the case of MLPs as they don't have an intrinsic time-convolution? I.e., what is the meaning of the i-th rollout time step in the case of MLP \"only\" (LR_CE = 0)? And why is that variation not close to zero for the 0-th rollout step, as indicated in Figure 3 (gray dashed curve) but starting from a value around 0.6?\n\nl. 223-229: At this point of studying the performance in various benchmarks as a function of LR, and also for some of the subsequent ablation studies of Sec. 5.3, it would be great to show a benchmark reference where the CE module is skipped or bypassed altogether, feeding the current and last obserivation only into the MLP networks but not using a CE representation of the latent state at all (while there is one case where LR_CE = 0 in Fig. 9a, this is still different from skipping the entire module and following the suggestion above). This would generally help to underline the significance of the CE for the chosen set of benchmarks and tasks.\n\nFig. 6: It appears that SAC does hardly profit from using a recurrent CE (SAC-GRU) if compared to the MLP variant (SAC-MLP); is there any explanation why? It could stress the importance of showing the RESeL performance with and without CE, as mentioned in my comment above\n\nMinor remarks:\n\nl. 126: Better don't use \" ' \" behind name of algorithm (\"RESeL's\") but speak of RESeL policy as a joint expression\n\nl. 171: Printing style (formatting) of scientific number notation \"3e - 4\" looks somewhat odd; check if correct Math formula style was chosen in the LaTex document or even better use decimal notation (\"$3 10^-4$\") or capital \"E\" instead of \"e\". This applies to other instances of number formatting of the main text as well.\n\nl. 240: SOTA (state-of-the-art) should be introduced as an abbreviation\n\nFig. 5 (and others): Quality of plots could generally be improved by avoiding overlapping of images and axes captions in some cases\n\nFig. 6: Colors/symbols of \"EPI\" and \"SAC-MLP\" are hardly distinguishable", "limitations": "The authors have addressed the main limitation well. As they state, what is missing is a study on the impact of using the RNN as a context encoder in general, which refers to my request for providing a reference baseline where the CE is bypassed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a sound, well-written and focused approach of studying the impact of using dedicated LR parameters for both the context encoding and policy/critic related NN representation as part a newly proposed RL algorithm (RESeL) to solve POMDP tasks with high performance. It is implicitly stated (by not mentioning previous work) that the authors' work represents the first time that such a clear separation and impact analysis of different LR between the major architecture components is made, which - if true - points to a relatively high degree of originality.\n\nThe authors underline their central proposition (of better using lower LR for the CE related network parts) both by theoretical arguments/proof and an extensive experimental study on a relatively large set of well-known POMDP tasks and benchmarks, including direct comparisons to a large variety of SOTA algorithms. This presentation, together with additional material in the appendix, is well-suited to allow reproducibility and comparability of their results.", "weaknesses": "Even though mentioned in the Limitiation section, it would have been very helpful to add some benchmark comparisons of the RESeL algorithm with its RNN-CE block entirely bypassed/skipped to clearly separate the impact of the latent space presentation from the rest of the RL algorithm. In particular for the classic POMDP tasks, where only position and velocity features are masked, respectively, it is important to know a baseline performance when simply feeding the current and last-step observation vector into the MLP networks (given that RESeL uses 2 x 256 neurons layers in their MLPs, adding a few more inputs show be more than tractable). Including such extension (not necessarily for all benchmarks but maybe a smaller subset) could significantly improve the scientific insight.\n\nHaving clear focus on the impact of varying LR the remainder of the RESeL algorithm is entirely built from existing RL building blocks, like the overarching SAC RL architecture or convential RNN/GRU components and, hence, provides no original contribution in itself (as correctly mentioned by the authors, using RNN for latent space representation is known for a long time, as is the SAC approach). Re-using existing building blocks per-se is not a bad idea (and I don't recommend addressing this issue in the rebuttal phase) but one has to accept that the algorithmic novelty in this work is hence limited. \n\nAs a minor remark: the authors spend a relatively large portion of their introduction on the mathematical formulation of Proposition 1, whose central statement (variation of RNN outputs grow as function of rollout-step and LR) appears not entirely original (if it really is a first-time, the authors should highlight this fact more strongly) since the result is quite intuitive for people working in the RNN/RL community. Besides that, the connection between lowering the LR as part of the update rule when training RNNs and the resulting formula of Proposition 1 is not obvious and could be highlighted more strongly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you have invested in reviewing our paper, as well as your insightful and constructive feedback. Your comments have greatly assisted us in improving the quality of our work.\n\n- **[Skipping CE blocks]** Thank you for pointing out the missing baseline. We implemented  this baseline by setting the CE outputs to zero while keeping everything else unchanged to isolate the effect of the latent space. The results are shown as the NO_CE curve in Fig. R1. The results indicate that historical information is crucial in our algorithm. Without it, achieving high returns on these POMDP tasks is hard.\n\n- **[Algorithmic novelty]** Our work primarily focuses on the impact of learning rates in recurrent RL. We believe the importance and ubiquity of this issue merit a dedicated discussion.\n\n- **[Originality of Proposition 1]** We referenced the proof from model imitation learning [1]. We consider the RNN as the transition function of an MDP and then, based on the derivation in [1], combined with the characteristics of RNNs, we derived the compounding error of RNNs. To the best of our knowledge, we are the first to derive the compounding error bound in RNNs. Thank you for your suggestion, we will highlight this point and include the relevant citations in the revised paper.\n\n- **[Relationship between Proposition 1 and reducing learning rate]** Lowering the learning rate reduces the single-step network output variation $\\epsilon$, thereby lowering the upper bound of Eq. (1).\n\n- **[Last-step observation]** Here we use an MLP-based pre-encoder, which is not the MLP policy, to map the last-step observation to a latent space. The MLP policy does not take the last-step observation as input. \n\n- **[MLP architecture]** We adopted the MLP network design from algorithms like SAC [2] and TD7 [3], which use a two-layer MLP for the policy network, with each layer consisting of 256 neurons. We previously tried smaller network structures, we find the impact was not significant.\n\n- **[Claims on MLP]** A smaller learning rate makes MLP training very slow, requiring many gradient updates to train the MLP adequately, leading to inefficiency. We will add this explanation to the revised paper. \n\n- **[Random seed]** \n\n  - We set a random seed at the start of each experiment to randomly initialize everything, including neural network weights and the environment. Every 1000 gradient updates, we perform a policy evaluation without resetting the environment's random state (forking the current random state). Using the current policy, we deterministically collect five trajectories for evaluation, where the policy outputs mean actions, but the initial state of the environment is re-sampled at the start of each trajectory. We repeated the experiment six times, each with a different random seed.\n  - To verify that our choice of seed count does not introduce larger statistical errors, we conducted an additional experiment with six more seeds on WalkerBLT-V-v0 and HopperBLT-V-v0. The results, shown in Fig. R5, demonstrate that the new curve (seed_7-12) closely matches the mean and shadow of the old curve (seed_1-6), indicating no significant statistical deviation. This shows that six seeds are sufficient for our experimental setup.\n\n- **[Figure 3]** Regarding Figure 3, we trained the policy using RESeL and collected trajectories with non-deterministic actions in the environment. We obtained the mean actions $\\{a_0,a_1,a_2,...\\}$ for each trajectory, where $1,2,3,...$ are rollout steps. We then performed a single gradient update on the policy and obtained the updated policy's mean action outputs at each state in the trajectory $\\{a_0', a_1', a_2', \\ldots\\}$. Figure 3 demonstrates $\\{||a_0-a_0'||_2,||a_1-a_1'||_2,||a_2-a_2'||_2,...\\}$ for different learning rates, with each experiment repeated 20 times. Although each curve has certain variance, the high repetition count results in a small standard error. The brown curves have a larger variance, hence the noticeable shadow (standard error), while other curves have smaller variance, making the shadow less apparent but present. In the gray dashed curve, the excessively high learning rate caused unpredictable behaviors, leading to significant initial variations in the policy. The gray dashed curve starts high and then decreases, possibly due to the larger action amplitude in the latter half of the trajectory, reaching the boundary values. As a result, many actions reach the boundary values and get compressed after substantial policy updates, leading to less variation compared to the first half.\n\n- **[SAC-GRU and SAC-MLP]** We believe the too large output variations of RNNs cause the value and policy learning process being instable, therefore preventing the policy from efficiently utilizing the historical information.\n\n- **[Minors]** Thank you for pointing these out. We will correct all these issues in the revised paper.\n\n  \n\n[1] Xu et al. \"Error bounds of imitating policies and environments.\" NeurIPS 2021.\n\n[2] Haarnoja et al. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\" ICML 2018.\n\n[3] Fujimoto et al. \"For sale: State-action representation learning for deep reinforcement learning.\" NeurIPS 2023.\n\n---\n\nThank you for guiding us towards making it a better work. We hope our responses have addressed your concerns effectively and enhanced the clarity of our main contributions. We look forward to your further comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j4od2jW68C", "reviewer": "Reviewer_dCQa", "summary": "The paper mitigates the training instability issue of the recurrent off-policy RL by using a smaller learning rate for the RNN block.", "questions": "See weakness above", "limitations": "The limitation is discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n- The paper proposes a simple solution with analysis.\n- The proposed solution is thoroughly verified in different experimental settings.", "weaknesses": "- My main concern is the novelty of the proposed solution. Although the paper gives a reason for using different learning rate for the RNN block and the MLPs, the proposed method is to choose a lower learning rate for the RNN block. In practice, it's somehow common to choose different learning rate for different components. For example, Dreamer [Hafner, Danijar, et al.], which also uses an RNN block in their model, chooses a smaller learning rate for the RNN-based world model. Also, when facing the training stability issue, tuning the learning rate is always on the checklist. \n\n    However, I would consider accepting the paper if it proposes an approach to automatically decide the learning rate of the RNN block by leveraging the analysis in section 4.2.\n- Some details need further explanation:\n    - In equation (1), it could be better to distinguish two $\\epsilon$.\n    - Will the amplification factor $\\beta = \\frac{K_y}{1-K_h}$ always larger than 1 (L160)?\n    - The average variation in network output $\\frac{1}{t}\\sum^{t-1}_{i=0} || y_i - y'_i||$ converges to $\\beta \\epsilon + \\epsilon$, which is not involved in $t$, but when this indicates with **longer** sequence lengths, the average variations in the RNN output induced by gradient deselect are amplified (L163-164)?\n    - In the caption of Fig 3, what does it mean by \"after one-step gradient-update\"? Does it mean one-gradient update per environment step?\n    - In L218, why does the learning rate of MLP must be increased **twentyfold**?\n    - In L221, \"The right panel shows the green and blue curve remain at similar levels until the final time step\". Where are the green and blue curves?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper, and for your insightful comments.\n\n- **[Dreamer]** It is not a common practice in reinforcement learning to use different learning rates for specific layers in a neural network. Typically, in reinforcement learning, the number of learning rates corresponds to the number of loss functions, with each loss function's associated network parameters sharing the same learning rate. For instance, in Dreamer, the world model is updated based on a prediction loss. Despite comprising various modules such as the non-RNN encoder, decoder, predictor, and RNN sequence model, these modules share a single learning rate in Dreamer families [1-3]. However, Figure 9(c) demonstrates that using the same learning rate for all layers during RL policy training could be suboptimal. Conversely, RESeL assigns a separate smaller learning rate specifically for the sequence model, while using a different learning rate for others. \n\n- **[Tuning the learning rate is always on the checklist]** We are not naively tuning the learning rate here. We have shown that uniformly tuning the overall learning rate does not achieve optimal performance. Instead, we propose RESeL to specifically set a smaller learning rate for the RNN, based on our theoretical analysis and intuitive understanding. Despite being straightforward, it is not trivial.\n\n- **[An approach to automatically decide the learning rate of the RNN block]**\n\n  - In all experiments, the learning rate for the RNN was set to 1/30 of the policy learning rate. We found this setting to be highly generalizable. From a practical standpoint, we recommend setting it to 1/30 of the policy learning rate or conducting a grid search centered around this ratio.\n\n  - On the other hand, for long trajectories, the upper bound of the average change in RNN outputs will eventually converge to $(1+\\beta)\\epsilon_{RNN}$, where $\\epsilon_{RNN}$ is the variation of the RNN’s first step output. Our goal is to find a scaling factor $\\alpha$ for the RNN learning rate so that the variation in RNN outputs matches $\\epsilon_{MLP}$: $\\alpha(1+\\beta)\\epsilon_{RNN}\\approx\\epsilon_{MLP}$. However, the values of $\\beta$, $\\epsilon_{RNN}$, and $\\epsilon_{MLP}$ are highly dependent on the neural network weights. Thus, we can periodically test $(1+\\beta)\\epsilon_{RNN}$ and $\\epsilon_{MLP}$ and use their ratio to determine $\\alpha$. Specifically, for every 500 gradient updates, we perform a pseudo-update to observe how much the outputs of the policy change after individually updating the RNN block or the MLP. We then use the ratio of these changes to scale the RNN learning rate. To avoid instability in training caused by frequent changes in the learning rate, we only adjust the learning rate during the initial 50 iterations (a warmup phase, corresponding to 50000 gradient updates) and keep it fixed thereafter. We set the initial RNN learning rate to $3\\times10^{-4}$. Our experimental results, shown in Fig. R6, indicate that the automated tuning method performs similarly to RESeL. We will add this discussion in our revised version. The final RNN learning rates are listed in the following table, which is close to our setting. \n\n    | Env. Name           | Auto-Tuned RNN Learning Rate            |\n    | ------------------- | --------------------------------------- |\n    | AntBLT-V-v0         | $6.0\\times 10^{-6}\\pm 3\\times 10^{-7}$  |\n    | HalfCheetahBLT-V-v0 | $1.1 \\times 10^{-5}\\pm 1\\times 10^{-7}$ |\n    | HopperBLT-V-v0      | $1.1 \\times 10^{-5}\\pm 4\\times 10^{-7}$ |\n    | WalkerBLT-V-v0      | $1.2 \\times 10^{-5}\\pm 1\\times 10^{-7}$ |\n\n    \n\n- **[Distinguish two $\\epsilon$]** Thank you for the suggestion. We will distinguish the two $\\epsilon$ in the revised version.\n\n- **[Is $\\beta$ always larger than 1]** It depends on the weights and gradient magnitude of the neural network. However, $\\beta$ is always larger than $0$, so the upper bound of $||y_i-y_i'||(i>0)$ is always greater than $\\epsilon$.\n\n- **[Average variation is not involved in $t$]** This needs to be understood in conjunction with Eq. (11) in Appendix B. Since the right-hand side of Eq. (11) is an increasing function of $t$, the mean value increases over time. We will include this explanation in the revised paper.\n\n- **[After one-step gradient-update in Fig. 3]** The action variation quantifies the difference in policy output after the gradient update compared to its output before the update, as considered in proposition 1. \"After a one-step gradient update\" indicates that we updated the policy only once, but we compare the differences in action outputs at various rollout steps before and after the policy update.\n\n- **[Learning rate of MLP must be increased twentyfold]** We found that increasing the MLP learning rate twentyfold to 0.006 resulted in action variation comparable to a CE learning rate of 0.0003. We will revise this statement in the updated paper to avoid confusion.\n\n- **[The green and blue curve]** This was a typo, it should actually refer to the orange and purple curves.\n\n[1] Hafner et al. \"Dream to control: Learning behaviors by latent imagination.\" arXiv preprint 2019.\n\n[2] Hafner et al. \"Mastering atari with discrete world models.\" arXiv preprint 2020.\n\n[3] Hafner et al. \"Mastering diverse domains through world models.\" arXiv preprint 2023.\n\n---\n\nWe hope that the above response can address your concerns adequately. We would greatly appreciate it if you could re-evaluate our paper based on the above responses.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wX6mFqkmeN", "reviewer": "Reviewer_vPkL", "summary": "This paper proposes RESeL which improves recurrent off-policy RL in POMDPs mainly by applying a lower learning rate to the context encoder. This is justified by their theoretical analysis that the recurrence amplifies the output difference in the long run. In practice, they also incorporate several techniques (Mamba architecture, critic ensembles, efficient data sampling, slow policy update). The main contribution is on strong empirical performance. Across several benchmarks (classic POMDP, meta-RL, credit assignment), RESeL attains SOTA performance.", "questions": "About architectural design. Is there a reason for using respective context encoders in actor-critic? \n\nAbout proposition 1. The connection between a smaller learning rate and the bound (1) is not very clear. Is the logic that a smaller learning rate leads to a smaller \\|\\theta - \\theta’\\| and thus a smaller \\epsilon, then a tighter upper bound in (1)? \n\nIs this a typo in the unusually large batch size described in Table 2? \n\nAs the main finding is on the learning rates, could the sensitivity analysis (Figure 9) be applied to more tasks? \n\nIndicated by Figure 10, is RESeL-GRU more sample-efficient than RESeL-Mamba? Is Mamba a crucial component of RESeL?", "limitations": "Yes, no major limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The problem setting of partial observability that this paper tackles is crucial and challenging. As indicated by the experiments, the partial observability is severe in some tasks and may require full context lengths (around 1000). \n\nThe paper is well written and easy to follow.\n\nThe performance gain averaged on a wide range of POMDP tasks is large enough to make RESeL a very strong empirical work. RESeL uses Mamba, which also accelerates training time a lot. \n\nThe ablation experiments on section 5.1 and 5.3 clearly shows that a smaller learning rate helps stability (through reduced action variation).", "weaknesses": "No major weaknesses. Please see the questions below.\n\nOne missing point is about the theoretical understanding -- why a smaller learning rate on context encoder can stabilize training? Reducing the output variation between consecutive updates, in my opinion, is a starting point, but not enough to explain the stability as a whole. This is connected with two-timescale update of feature vs value learning, e.g., https://openreview.net/forum?id=rJleN20qK7 Perhaps this related work is worth a discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time to review and provide positive feedback for our work.\n\n- **[Theoretical understanding]** Thank you very much for pointing out this relevant work. It has been very enlightening for us. RESeL and TTN are closely related, with the MLP value and the small learning rate RNN encoder corresponding to the value function and representation network, respectively. The convergence guarantee of TTN also explains why a small output variation between consecutive updates can enhance training stability.\n\n- **[Architectural design]** As noted in [1], sharing a context encoder could lead to a large gradient norm, causing instability. Therefore, we chose to use separate context encoders.\n\n- **[The connection between a smaller learning rate and the bound (1)]** You are right, reducing the learning rate can lower $\\epsilon$, thereby reducing the upper bound. We will include this discussion in the revised version of our paper.\n\n- **[Large batch size described in Table 2]** The batch size here refers to the number of transitions. For each update, we collect at least one complete trajectory to calculate the loss. In environments like HalfCheetah, one trajectory consists of 1000 steps. So, the 1000 and 2000 mentioned correspond to one or two complete trajectories in these environments. To maintain consistency in the number of valid transitions across different environments, especially those with varying trajectory lengths, we use the same batch size for them. For these environment, we will dynamically sample uncertain number of complete trajectories to ensure that the number of valid transitions is no less than this batch size.\n\n- **[Sensitivity analysis in more tasks]** Yes, we have conducted similar experiments in WalkerBLT-P-v0 and AntBLT-V-v0. The results are shown in Fig. R2. Limited by computational resources, we only included the setting of Figs. 9(a) and 9(c) to illustrate that the optimal $LR_{CE}$ is around $10^{-5}$ in these tasks, and to show that $LR_{CE} = LR_{MLP}$ does not achieve optimal performance.\n\n- **[Comparison between RESeL-Mamba and RESeL-GRU]** It's true that GRU is more sample-efficient than Mamba in some environments. Mamba loses some expressive power due to its parallelzability (the hidden state at time $t$ is not used as input to the nonlinear network at time $t+1$ but is accumulated through a linear transition matrix) and also is not crucial in RESeL. However, Mamba significantly reduces computation time during training and requires much less memory in environments with variable trajectory lengths.\n\n[1] Ni et al. \"Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs.\" ICML 2022.\n\n---\n\nThank you for taking the time to provide us with your feedback. We appreciate your valuable comments and suggestions, which will help us improve our work. We look forward to receiving your further feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "McCQvlLFJK", "reviewer": "Reviewer_B2hv", "summary": "The authors investigate the stability of training deep recurrent policies for POMDP tasks. They hypothesize that the recurrent part of the encoder faces less stable training compared to the fixed-length parts of the encoder (e.g. the MLP input layer) and propose that the former should use a lower learning rate. This is shown to be effective mostly through an empirical evaluation.", "questions": "1. How were hyperparameters tuned in the experiments?\n2. Have you considered the approaches mentioned in the weaknesses? (gradient clipping and truncated gradients)", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 1, "strengths": "The paper's ideas are communicated clearly and the authors present excellently summarize the state of the art for several partially observable settings, including the general setting, meta RL, and credit assignment. I appreciated the wide range of experimental domains and the sensitivity analysis demonstrating that the recurrent layers and the rest of the network should have different learning rates for optimal performance. The suggestion to introduce a new hyperparameter for the recurrent encoder could be broadly adopted.", "weaknesses": "Unfortunately, I believe the work is not sufficiently novel for acceptance at NeurIPS. The problem identified by the authors --- that a recurrent encoder is numerically less stable than a fixed-length encoder --- has been identified many times over the years with various techniques proposed to mitigate it. This includes classical works such as the original LSTM paper [1] aimed at tackling exponentially scaling gradients. More recently, techniques such as gradient clipping and truncating the number of recurrent backpropagation steps (which is not the same as shortening the context length) have been suggested to tackle this problem as well.\n\nExperimentally, I would have liked to see that these last two tricks do not already solve the stability issues, as there was no mention of them in the paper. It also seems that the hyperparameters were tuned by hand (since I could not find any mention of a hyperparameter selection scheme) which I think is insufficient for a work for which the sole contribution is an empirical result that we are meant broadly apply. I would have expected to see a very rigorous and objective hyperparameter selection method such as a grid search over reasonable hyperparameters. While I appreciated the sensitivity analysis in Fig 9, it was only conducted on a single domain.\n\nUltimately however, my opinion is that the idea is not substantial enough for publication in a top venue.\n \n[1]: Hochreiter, Sepp, and Jürgen Schmidhuber. \"Long short-term memory.\" Neural computation 9.8 (1997): 1735-1780.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and providing us with your valuable feedback.\n\n- **[LSTM]** Our paper discovered a totally different issue from gradient exploding (numerical instability) resolved in LSTM: even if the RNN gradients do not explode and are similar in magnitude to those of MLPs, the gradient updates can still cause significant variations in the RNN outputs. These large output variations lead to instability in RL training, requiring a slow down in RNN updates. This is evident in Figure 3, particularly when the rollout step is 0. The action variation of the brown curve is lower than that of the orange curve, indicating that the RNN parameter changes are small and its gradient is not large. However, as the rollout steps increase, the brown curve grows by several orders of magnitude, which can destabilize reinforcement learning training.\n\n- **[The gradient clipping & truncated gradient tricks]** \n  - We have tried gradient clipping (0.5 maximum gradient norm as did in Dreamer) and truncated gradient (gradient maximum horizon 32) tricks, but we found that these tricks are irrelevant  to the problem of RNNs in RL. Their purpose is to prevent gradient explosion in RNNs and cannot slow down the RNN updates. In Fig. R1, we show the effects of these tricks. It can be seen that gradient truncation may still lead to instability and infinite values in tasks like Ant and HalfCheetah. Gradient clipping can suppress extreme values but does not improve performance. We conducted an in-depth investigation and compared the policy gradient norms of RESeL and the gradient clipping approach. As shown in Fig. R3, the policy gradient norm of the gradient clipping method has many spikes, reflecting training instability. This instability does not stem from gradient value instability, as we have already employed gradient clipping. \n\n  - We find the gradient instability originates from the instability of value function learning. In Fig. R4, we observe that in the baseline with gradient clipping but unchanged learning rates, the value loss frequently encounters outliers. This is due to large variation in the outputs of the policy and value networks, which make value function learning, based on bootstrapping, more unstable. This instability results in large value loss, which in turn causes spikes in the policy gradients. This issue is distinct from the instability and explosion of RNN gradients.\n\n- **[Sensitivity analysis was only conducted on a single domain]**  We have conducted similar experiments in WalkerBLT-P-v0 and AntBLT-V-v0. The results are shown in Fig. R2. Limited by computational resources, we only included the setting of Figs. 9(a) and 9(c) to illustrate that the optimal $LR_{CE}$ is around $10^{-5}$ in these tasks, and to show that $LR_{CE} = LR_{MLP}$ does not achieve optimal performance. The conclusions drawn from Fig. R2 are consistent with those presented in Figs. 9(a) and 9(c).\n- **[Hyper-parameters]** Some hyper-parameters (discount factor, reward as input) were chosen based on the nature of the environments, while others (Context encoder LR, policy and value function LR, and batch size) are determined via grid search:\n  - **Context Encoder LR**: We fixed the learning rate of the context encoder to be 1/30 of the policy learning rate in all tasks. We aim to have the action variation caused by RNN updates to be comparable to that caused by MLP updates, thus fixing the learning rate ratio between the context encoder and MLP. After conducting parameter searches in several environments, as shown in Figure 9, we find the 1/30 ratio performed well (this conclusion is also evident in Figure R2), so we fixed this ratio across all environments.\n  - **Policy and Value Function LR**: We mainly followed the default learning rate settings in CleanRL [1], using a policy learning rate of $3 \\times 10^{-4}$ and an action-value function learning rate of $10^{-3}$. We conducted a grid search between the aforementioned learning rates and those reduced by a factor of $5$  (aiming to further stabilize training). We found that a smaller learning rate was more stable in tasks where the partial observability is not very significant (e.g., MDPs like classic MuJoCo).\n  - **Discount Factor ($\\gamma$)**: For general tasks, we used a common $\\gamma$ value of 0.99. For environments requiring long-term memory (e.g., Key-to-Door), we used a larger $\\gamma$ of 0.9999.\n  - **Reward as Input**: The Classic Meta-RL tasks require inferring the true reward function based on real-time rewards. Thus, we also included real-time reward feedback as an input to the policy and value in these tasks.\n  - **Batch Size**: We conducted a grid search for batch size, mainly exploring sizes that were 1x or 2x the maximum trajectory length. We found that in classic POMDP tasks, a larger batch size performed better. In other tasks, the advantage of a large batch size was not significant and could even reduce training speed.\n\n[1] Huang et al. \"CleanRL: High-quality single-file implementations of deep reinforcement learning algorithms.\" JMLR 2022.\n\n----\n\nWe sincerely appreciate your time and expertise in reviewing our paper. We would greatly appreciate it if you could re-evaluate our paper based on the above responses.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the time and effort all the reviewers have dedicated to our paper and their highly constructive comments. Here, we provide a general response to the common concerns raised.

- **[Novelty/Contribution]**
  - We would like to emphasize that our core contribution lies in presenting a universal principle rather than specifying the hyperparameters for each task. This principle is supported by theoretical foundations and intuitive understanding, making it more than just an empirical trick. Although it is simple, it is by no means trivial.
  - Specifically, our findings indicate that the instability in recurrent RL arises from the amplified action variation by rollout step, specifically the compounding error in RNN output. This results in larger overall output variation for the RNN compared to the MLP, even when their single-step changes are the same, leading to instability during training. This paper proposes to solely slow down updates of the RNN by giving it a lower learning rate, then $\epsilon$ in Eq. (1) is reduced, thereby decreasing the RNN's output variation and improving the stability. There have been various traditional approaches that can improve the training stability of RNNs in non-RL scenarios, such as reducing the overall learning rate, gradient clipping, and gradient truncation. However, these methods cannot solely slow down the update of RNN thus still sufferring training instability in reinforcement learning. 

- **[Relationship between Proposition 1 and reducing learning rate]**
  Reducing the learning rate can decrease the single-step network output variation, denoted as $\epsilon$, thereby reducing the upper bound of Eq. (1).

- **[Figure 3]**
  Here we elaborate on how we obtained Figure 3: We trained a policy using RESeL and collected a batch of trajectories in the environment with stochastic exploration. For each trajectory, we calculated the mean action output by the RESeL policy $\{a_0, a_1, a_2, \ldots\}$, where $1, 2, 3, \ldots$ represent rollout steps. We then performed a single gradient update on the policy and obtained the updated policy's mean action outputs at each state in the trajectory $\{a_0', a_1', a_2', \ldots\}$. In Figure 3, we plotted the action variation $\{||a_0 - a_0'||_2, ||a_1 - a_1'||_2, ||a_2 - a_2'||_2, \ldots\}$ under different learning rate settings. **The action variation means the difference in policy output after the gradient update compared to its output before the update.**


- **[RNN learning rate setting]**
  - We fixed the learning rate of the RNN to be 1/30 of the policy learning rate for all tasks. Fig. 9 and Fig. R2 demonstrate that this ratio achieves the highest policy performance. Additionally, Fig. 3 also shows that this ratio effectively prevents the RNN output variation from significantly exceeding that of the MLP. Our experiments indicate that this ratio has good generalizability and works well in all tasks. Therefore, we recommend setting the RNN learning rate to 1/30 of the policy MLP learning rate in practice or conducting a grid search centered around this ratio.
  - Based on the suggestion from Reviewer dCQa, we have also implemented an automatic RNN Learning Rate Tuning method. At the beginning of training, we introduce a warmup phase where the RNN’s learning rate is automatically adjusted. The tuning objective is ensuring that the action variation caused by updating the RNN alone is roughly consistent with that of the MLP. 

**We have uploaded the additional experiment results (Figs. R1-R6) in a separate PDF.**

Meta-review: While Reviewer B2hv votes for rejecting the paper due to insufficient depth, the three remaining reviewers vote in favor of acceptance provided extensive changes are made. 
Both views are well-founded and comprehensible.  In my view, these views are best summarized as “Borderline Accept” and as there are no major flaws in the paper, it depends on where the bar is set for NeurIPS 2024 whether this paper can be accepted.
Decision: Accept (poster)