Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate

Authors: fan-ming  luo
zuolin  tu
zefang  huang
yang  yu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:45:11

Abstract: Real-world decision-making tasks are usually partially observable Markov decision processes (POMDPs), where the state is not fully observable. Recent progress has demonstrated that recurrent reinforcement learning (RL), which consists of a context encoder based on recurrent neural networks (RNNs) for unobservable state prediction and a multilayer perceptron (MLP) policy for decision making, can mitigate partial observability and serve as a robust baseline for POMDP tasks. However, prior recurrent RL algorithms have faced issues with training instability. In this paper, we find that this instability stems from the autoregressive nature of RNNs, which causes even small changes in RNN parameters to produce large output variations over long trajectories. Therefore, we propose **R**ecurrent Off-policy RL with Context-**E**ncoder-**S**p**e**cific **L**earning Rate (RESeL) to tackle this issue. Specifically, RESeL uses a lower learning rate for context encoder than other MLP layers to ensure the stability of the former while maintaining the training efficiency of the latter. We integrate this technique into existing off-policy RL methods, resulting in the RESeL algorithm. We evaluated RESeL in 18 POMDP tasks, including classic, meta-RL, and credit assignment scenarios, as well as five MDP locomotion tasks. The experiments demonstrate significant improvements in training stability with RESeL. Comparative results show that RESeL achieves notable performance improvements over previous recurrent RL baselines in POMDP tasks, and is competitive with or even surpasses state-of-the-art methods in MDP tasks. Further ablation studies highlight the necessity of applying a distinct learning rate for the context encoder. Code is available at https://github.com/FanmingL/Recurrent-Offpolicy-RL.

Main paper: 
 introduction: 
 In many real-world reinforcement learning (RL) tasks [1] , complete state observations are often unavailable due to limitations such as sensor constraints, cost considerations, or task-specific requirements. For example, visibility can be obstructed by obstacles in autonomous driving [2, 3] or robotic manipulation [4] , and measuring ground friction may be infeasible when controlling quadruped robots on complex terrain [5] . These scenarios are common and typically conceptualized as Partially Observable Markov Decision Processes (POMDPs). Traditional RL struggles with POMDPs due to the lack of essential state information [6] .A mainstream class of POMDP RL algorithms infers unobservable states by leveraging historical observation information, either explicitly or implicitly [6] . This often requires memory-augmented Figure 1 : A simple recurrent policy architecture.Partially Observable Markov Decision Processes (POMDPs) [22, 6] enhance MDPs for scenarios with limited state visibility, addressing a number of real-world decision-making challenges with imperfect or uncertain data. A POMDP is defined by ⟨S, A, O, P, O, r, γ, ρ 0 ⟩, consisting of state space S, action space A, observation space O, state transition function P , observation function O, reward function r, discount factor γ, and initial state distribution ρ 0 . Unlike MDPs, agents in POMDPs receive observations o ∈ O instead of direct state information, requiring a belief state-a probability distribution over S-to inform decisions. The objective is to search for a policy π to maximize the expected rewards, while factoring in observation and transition uncertainties.Recurrent RL [23, 11] involves employing RNN-based policy or critic models within the framework of RL. Figure 1 provides a commonly utilized recurrent policy architecture. This architecture has two main components. First, a context encoder processes the current observation o, the last action â, the reward r2 , and the RNN hidden state. Then, an MLP policy uses the context embedding and o to generate actions, facilitating the extraction of non-observable states into a context embedding. We can train the recurrent policy by incorporating it into any existing RL algorithm. During training, recurrent RL requires batch sampling and loss computation on a trajectory-by-trajectory basis. 
related work: 
 In this work, we focus on recurrent RL methods [24, 25, 23] , which leverage RNN-based models for RL tasks. These methods have demonstrated robust performance in POMDP [10, 22] and meta-RL tasks [25, 26] , having shown notable success in practical applications such as robotic motion control [27, 28, 29] and MOBA games [30] .Recurrent RL utilizes RNNs such as GRUs [8] , LSTMs [7] , and more recently, SSMs [12] to construct policy and critic models. Typically, in recurrent actor-critic RL, both policy and critic adopt RNN architectures [23, 11] . When full state information is available during training, an MLP critic can also be used instead [29] , receiving the full state as its input. In this work, we do not assume that complete state information can be accessed, so policy and critic models are both RNN structures.The most direct way to train RNN models in recurrent RL is combining an RNN structure with existing RL algorithms, such as being integrated into on-policy methods like PPO [28, 30, 12, 31] and TRPO [25, 32] as well as off-policy methods like TD3 [11] and SAC [23] . This work follows the line of recurrent off-policy RL [11, 23] , as we believe that existing recurrent off-policy RL algorithmshave not yet achieved their maximum potential.Despite the simplicity of direct integration, however, training RNNs often suffers from the training instability issue [12, 15] , especially in long sequence length. Previous methods usually use truncated trajectory to train RNNs [11] . However, deploying recurrent models with full-trajectory lengths can result in a mismatch between training and deployment. ESCP [16] addressed this issue by using the same sequence lengths in both training and deployment by truncating the historical length in deployment scenarios, but this could impair policy performance due to the restricted memory length. On the other hand, many studies also employed auxiliary losses in addition to the RL loss for RNN training [19] . These losses aim to enable RNNs to stably extract some certain unobservable state either implicitly or explicitly. For instance, they might train the RNNs to predict transition parameters [33] , distinguish between different tasks [16, 34] , or accurately predict state transitions [17, 18, 19] . These methods require the RNN to learn specific information, which may limit the RNN's potential and may only be applicable to a subset of tasks, reducing the algorithmic generalizability.RESeL directly trains recurrent models using off-policy RL due to its simplicity, flexibility, and potential high sample efficiency. The most relevant study to RESeL is [11] , which focused on improving recurrent off-policy RL by analyzing the hyperparameters, network design, and algorithm choices. More in depth, RESeL studies the instability nature of previous methods and enhances the stability of recurrent off-policy RL through a specifically designed learning rate. 
method: 
 In this section, we address the training stability challenges of recurrent RL. We will first elaborate on our model architecture in Sec. 4.1 and introduce how we address the stability issue in Sec. 4.2. Finally, we will summarize the training procedure of RESeL in Sec. 4.3.The architectures of RESeL policy and critic models are depicted in Fig. 2 . Specifically, RESeL policy initially employs MLPs as pre-encoders to map the current and last-step observations (o and ô respectively), actions (â), and rewards (r) into hidden spaces with equal dimensions, ensuring a balanced representation of various information types. These pre-encoded inputs are then concatenated < l a t e x i t s h a 1 _ b a s e 6 4 = " g 1 o s P + / + 6 v W Q k u Z i N x c P i G Q Q e o 8 = " > A A A C x H i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V k U x G U L 9 g F a J J l O a 2 i a C Z O J U I r + g F v 9 N v E P 9 C + 8 M 0 5 B L a I T k p w 5 9 5 4 z c + 8 N 0 z j K l O e 9 F p y 5 + Y X F p e J y a W V 1 b X 2 j v L n V y k Q u G W 8 y E Q v Z C Y O M x 1 H C m y p S M e + k k g e j M O b t c H i m 4 + 0 7 L r N I J J d q n P L u K B g k U T 9 i g S K q I W 7 K F a / q m e X O A t + C C u y q i / I L r t G D A E O O E T g S K M I x A m T 0 X M G H h 5 S 4 L i b E S U K R i X P c o 0 T a n L I 4 Z Q T E D u k 7 o N 2 V Z R P a a 8 / M q B m d E t M r S e l i j z S C 8 i R h f Z p r 4 r l x 1 u x v 3 h P j q e 8 2 p n 9 o v U b E K t w S + 5 d u m v l f n a 5 F o Y 8 T U 0 N E N a W G 0 d U x 6 5 K b r u i b u 1 + q U u S Q E q d x j + K S M D P K a Z 9 d o 8 l M 7 b q 3 g Y m / m U z N 6 j 2 z u T n e 9 S 1 p w P 7 P c c 6 C 1 k H V P 6 r 6 j c N K 7 d S O u o g d 7 G K f 5 n m M G i 5 Q R 9 N 4 P + I J z 8 6 5 E z u Z k 3 + m O g W r 2 c a 3 5 T x 8 A F e 0 j 3 o = < / l a t e x i t > o < l a t e x i t s h a 1 _ b a s e 6 4 = " Y M f 9 3 L d Z f c l I x 4 K k s A H p 4 3 D i d M c = " > A A A C y n i c j V H L S s N A F D 2 N r / q u u n Q T L I K r k o i o y 6 I b F y 4 q 2 A e 0 R S b T a R u a J m E y E U r o z h 9 w q x 8 m / o H + h X f G F N Q i O i H J m X P P u T P 3 X i 8 O / E Q 5 z m v B W l h c W l 4 p r q 6 t b 2 x u b Z d 2 d h t J l E o u 6 j w K I t n y W C I C P x R 1 5 a t A t G I p 2 N g L R N M b X e p 4 8 1 7 I x I / C W z W J R X f M B q H f 9 z l T R D U 7 Q 6 Y y N r 0 r l Z 2 K Y 5 Y 9 D 9 w c l J G v W l R 6 Q Q c 9 R O B I M Y Z A C E U 4 A E N C T x s u H M T E d Z E R J w n 5 J i 4 w x R p 5 U 1 I J U j B i R / Q d 0 K 6 d s y H t d c 7 E u D m d E t A r y W n j k D w R 6 S R h f Z p t 4 q n J r N n f c m c m p 7 7 b h P 5 e n m t M r M K Q 2 L 9 8 M + V / f b o W h T 7 O T Q 0 + 1 R Q b R l f H 8 y y p 6 Y q + u f 2 l K k U Z Y u I 0 7 l F c E u b G O e u z b T y J q V 3 3 l p n 4 m 1 F q V u 9 5 r k 3 x r m 9 J A 3 Z / j n M e N I 4 r 7 m n F v T k p V y / y U R e x j w M c 0 T z P U M U V a q i b K h / x h G f r 2 p L W x M o + p V Y h 9 + z h 2 7 I e P g A 6 Q p I 5 < / l a t e x i t > â < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 F h K d r H / o o I W O n / H 7 C X 1 C 2 6 9 G R Y = " > A A A C y n i c j V H L S s N A F D 2 N r / q u u n Q T L I K r k o i o y 6 I b F y 4 q 2 A e 0 R Z L p t B 2 a J m E y E U r o z h 9 w q x 8 m / o H + h X f G F N Q i O i H J m X P P u T P 3 X j 8 O R K I c 5 7 V g L S w u L a 8 U V 9 f W N z a 3 t k s 7 u 4 0 k S i X j d R Y F k W z 5 X s I D E f K 6 E i r g r V h y b + w H v O m P L n W 8 e c 9 l I q L w V k 1 i 3 h 1 7 g 1 D 0 B f M U U c 3 O 0 F O Z n N 6 V y k 7 F M c u e B 2 4 O y s h X L S q 9 o I M e I j C k G I M j h C I c w E N C T x s u H M T E d Z E R J w k J E + e Y Y o 2 8 K a k 4 K T x i R / Q d 0 K 6 d s y H t d c 7 E u B m d E t A r y W n j k D w R 6 S R h f Z p t 4 q n J r N n f c m c m p 7 7 b h P 5 + n m t M r M K Q 2 L 9 8 M + V / f b o W h T 7 O T Q 2 C a o o N o 6 t j e Z b U d E X f 3 P 5 S l a I M M X E a 9 y g u C T P j n P X Z N p 7 E 1 K 5 7 6 5 n 4 m 1 F q V u 9 Z r k 3 x r m 9 J A 3 Z / j n M e N I 4 r 7 m n F v T k p V y / y U R e x j w M c 0 T z P U M U V a q i b K h / x h G f r 2 p L W x M o + p V Y h 9 + z h 2 7 I e P g B i s 5 J K < / l a t e x i t > r Pre-Encoder < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 j r x d V I J 8 S q G / g y nV i 9 h f U s o 6 3 o = " > A A A C y n i c j V H L S s N A F D 2 N r / q u u n Q T L I K r k o i o y 6 I b F y 4 q 2 A e 0 R Z J 0 2 g 5 N k z C Z C C V 0 5 w + 4 1 Q 8 T / 0 D / w j v j F N Q i O i H J m X P P u T P 3 X j 8 J e S o d 5 7 V g L S w u L a 8 U V 9 f W N z a 3 t k s 7 u 4 0 0 z k T A 6 k E c x q L l e y k L e c T q k s u Q t R L B v L E f s q Y / u l T x 5 j 0 T K Y + j W z l J W H f s D S L e 5 4 E n i W p 2 h p 7 M 4 + l d q e x U H L 3 s e e A a U I Z Z t b j 0 g g 5 6 i B E g w x g M E S T h E B 5 S e t p w 4 S A h r o u c O E G I 6 z j D F G v k z U j F S O E R O 6 L v g H Z t w 0 a 0 V z l T 7 Q 7 o l J B e Q U 4 b h + S J S S c I q 9 N s H c 9 0 Z s X + l j v X O d X d J v T 3 T a 4 x s R J D Y v / y z Z T / 9 a l a J P o 4 1 z V w q i n R j K o u M F k y 3 R V 1 c / t L V Z I y J M Q p 3 K O 4 I B x o 5 6 z P t v a k u n b V W 0 / H 3 7 R S s W o f G G 2 G d 3 V L G r D 7 c 5 z z o H F c c U 8 r 7 s 1 J u X p h R l 3 E P g 5 w R P M 8 Q x V X q K G u q 3 z E E 5 6 t a 0 t Y E y v / l F o F 4 9 n D t 2 U 9 f A B b k J J H < / l a t e x i t > ô x x: MLP x : concatenation< l a t e x i t s h a 1 _ b a s e 6 4 = " + 3 T e C I E N 7 v o V 8 C x k m D 0 N 3 + Y 6 D a E = " > A A A C x H i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V k U x G U L 9 g F a J J l O a + g 0 C Z O J U I r + g F v 9 N v E P 9 C + 8 M 0 5 B L a I T k p w 5 9 5 4 z c + 8 N U x F l y v N e C 8 7 c / M L i U n G 5 t L K 6 t r 5 R 3 t x q Z U k u G W + y R C S y E w Y Z F 1 H M m y p S g n d S y Y N R K H g 7 H J 7 p e P u O y y x K 4 k s 1 T n l 3 F A z i q B + x Q B H V C G 7 K F a / q m e X O A t + C C u y q J + U X X K O H B A w 5 R u C I o Q g L B M j o u Y I P D y l x X U y I k 4 Q i E + e 4 R 4 m 0 O W V x y g i I H d J 3 Q L s r y 8 a 0 1 5 6 Z U T M 6 R d A r S e l i j z Q J 5 U n C + j T X x H P j r N n f v C f G U 9 9 t T P / Q e o 2 I V b g l 9 i / d N P O / O l 2 L Q h 8 n p o a I a k o N o 6 t j 1 i U 3 X d E 3 d 7 9 U p c g h J U 7 j H s U l Y W a U 0 z 6 7 R p O Z 2 n V v A x N / M 5 m a 1 X t m c 3 O 8 6 1 v S g P 2 f 4 5 w F r Y O q f 1 T 1 G 4 e V 2 q k d d R E 7 2 M U + z f M Y N V y g j q b x f s Q T n p 1 z R z i Z k 3 + m O g W r 2 c a 3 5 T x 8 A D Z 0 j 2 w = < / l a t e x i t > a < l a t e x i t s h a 1 _ b a s e 6 4 = " N D R w g / h t F Z 3 G Z J G V f D t M 2 5 3 s G V Y = " > A A A C y 3 i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R 7 b L o x o 1 Q w T 6 g F k m m 0 z q Y F 5 m J U K t L f 8 C t / p f 4 B / o X 3 h l T U I v o h C R n z j 3 n z t x 7 / S Q Q U j n O a 8 G a m Z 2 b X y g u l p a W V 1 b X y u s b L R l n K e N N F g d x 2 v E 9 y Q M R 8 a Y S K u C d J O V e 6 A e 8 7 V 8 f 6 3 j 7 h q d S x N G 5 G i W 8 F 3 r D S A w E 8 x R R n Q s l Q i 7 t 2 m W 5 4 l Q d s + x p 4 O a g g n w 1 4 v I L L t B H D I Y M I T g i K M I B P E h 6 u n D h I C G u h z F x K S F h 4 h z 3 K J E 3 I x U n h U f s N X 2 H t O v m b E R 7 n V M a N 6 N T A n p T c t r Y I U 9 M u p S w P s 0 2 8 c x k 1 u x v u c c m p 7 7 b i P 5 + n i s k V u G K 2 L 9 8 E + V / f b o W h Q F q p g Z B N S W G 0 d W x P E t m u q J v b n + p S l G G h D i N + x R P C T P j n P T Z N h 5 p a t e 9 9 U z 8 z S g 1 q / c s 1 2 Z 4 1 7 e k A b s / x z k N W n t V 9 6 D q n u 1 X 6 k f 5 q I v Y w j Z 2 a Z 6 H q O M E D T T N H B / x h G f r 1 J L W r X X 3 K b U K u W c T 3 5 b 1 8 A E g W Z I n < / l a t e x i t > ⇥8 < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 Q t e D 9 i 0 D f m z V x I P W 5 9 H g Q Y E b 8 Y = " > A A A C / X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V l 0 4 1 L B a q G R k q R j H c y L y U Q s p f g n 7 t y J W 3 / A r W 7 F P 9 C / 8 M 4 0 F b W I T k j m 3 n P P O Z k 7 1 0 s C n k r L e i 0 Y Y + M T k 1 P F 6 d L M 7 N z 8 Q n l x 6 T i N M + G z u h 8 H s W h 4 b s o C H r G 6 5 D J g j U Q w N / Q C d u J d 7 K n 6 y S U T K Y + j I 9 l N 2 G n o d i J + x n 1 X E t Q q 7 z g e 6 / C o 5 4 W u F P y q X z p s W a b j 0 G a r z W n H M h 3 k 2 y W H R e 1 P Y q t c s a q W X u Z o Y O d B B f k 6 i M s v c N B G D B 8 Z Q j B E k B Q H c J H S 0 4 Q N C w l h p + g R J i j i u s 7 Q R 4 m 0 G b E Y M V x C L + j b o a y Z o x H l y j P V a p / + E t A r S G l i j T Q x 8 Q T F 6 m + m r m f a W a G /u v 2 m m Q l X u 5 9 w M 7 + q U N G D 7 5 z h H g + O N q r 1 V t Q 8 3 K 7 X d f N R F r G A V 6 z T P b d S w j w P U y f s G j 3 j C s 3 F t 3 B p 3 x v 2 A a h R y z T K + L e P h A / M Y p K w = < / l a t e x i t > 2 6 6 4 Q0 Q1 . . . Q7< l a t e x i t s h a 1 _ b a s e 6 4 = " g 1 o s P + / + 6 v W Q k u Z i N x c P i G Q Q e o 8 = " > A A A C x H i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V k U x G U L 9 g F a J J l O a 2 i a C Z O J U I r + g F v 9 N v E P 9 C + 8 M 0 5 B L a I T k p w 5 9 5 4 z c + 8 N 0 z j K l O e 9 F p y 5 + Y X F p e J y a W V 1 b X 2 j v L n V y k Q u G W 8 y E Q v Z C Y O M x 1 H C m y p S M e + k k g e j M O b t c H i m 4 + 0 7 L r N I J J d q n P L u K B g k U T 9 i g S K q I W 7 K F a / q m e X O A t + C C u y q i / I L r t G D A E O O E T g S K M I x A m T 0 X M G H h 5 S 4 L i b E S U K R i X P c o 0 T a n L I 4 Z Q T E D u k 7 o N 2 V Z R P a a 8 / M q B m d E t M r S e l i j z S C 8 i R h f Z p r 4 r l x 1 u x v 3 h P j q e 8 2 p n 9 o v U b E K t w S + 5 d u m v l f n a 5 F o Y 8 T U 0 N E N a W G 0 d U x 6 5 K b r u i b u 1 + q U u S Q E q d x j + K S M D P K a ZY M f 9 3 L d Z f c l I x 4 K k s A H p 4 3 D i d M c = " > A A A C y n i c j V H L S s N A F D 2 N r / q u u n Q T L I K r k o i o y 6 I b F y 4 q 2 A e 0 R S b T a R u a J m E y E U r o z h 9 w q x 8 m / o H + h X f G F N Q i O i H J m X P P u T P 3 X i 8 O / E Q 5 z m v B W l h c W l 4 p r q 6 t b 2 x u b Z d 2 d h t J l E o u 6 j w K I t n y W C I C P x R 1 5 a t A t G I p 2 N g L R N M b X e p 4 8 1 7 I x I / C W z W J R X f M B q H f 9 z l T R D U 7 Q 6 Y y N r 0 r l Z 2 K Y 5 Y 9 D 9 w c l J G v W l R 6 Q Q c 9 R O B I M Y Z A C E U 4 A E N C T x s u H M T E d Z E R J w n 5 J i 4 w x R p 5 U 1 I J U j B i R / Q d 0 K 6 d s y H t d c 7 E u D m d E t A r y W n j k D w R 6 S R h f Z p t 4 q n J r N n f c m c m p 7 7 b h P 5 e n m t M r M K Q 2 L 9 8 M + V / f b o W h T 7 O T Q 0 + 1 R Q b R l f H 8 y y p 6 Y q + u f 2 l K k U Z Y u I 0 7 l F c E u b G O e u z b T y J q V 3 3 l p n 4 m 1 F q V u 9 5 r k 3 x r m 9 J A 3 Z / j n M e N I 4 r 7 m n F v T k p V y / y U R e x j w M c 0 T z P U M U V a q i b K h / x h G f r 2 p L W x M o + p V Y h 9 + z h 2 7 I e P g A 6 Q p I 5 < / l a t e x i t > â < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 F h K d r H / o o I W O n / H 7 C X 1 C 2 6 9 G R Y = " > A A A C y n i c j V H L S s N A F D 2 N r / q u u n Q T L I K r k o i o y 6 I b F y 4 q 2 A e 0 R Z L p t B 2 a J m E y E U r o z h 9 w q x 8 m / o H + h X f G F N Q i O i H J m X P P u T P 3 X j 8 O R K I c 5 7 V g L S w u L a 8 U V 9 f W N z a 3 t k s 7 u 4 0 k S i X j d R Y F k W z 5 X s I D E f K 6 E i r g r V h y b + w H v O m P L n W 8 e c 9 l I q L w V k 1 i 3 h 1 7 g 1 D 0 B f M U U c 3 O 0 F O Z n N 6 V y k 7 F M c u e B 2 4 O y s h X L S q 9 o I M e I j C k G I M j h C I c w E N C T x s u H M T E d Z E R J w k J E + e Y Y o 2 8 K a k 4 K T x i R / Q d 0 K 6 d s y H t d c 7 E u B m d E t A r y W n j k D w R 6 S R h f Z p t 4 q n J r N n f c m c m p 7 7 b h P 5 + n m t M r M K Q 2 L 9 8 M + V / f b o W h T 7 O T Q 2 C a o o N o 6 t j e Z b U d E X f 3 P 5 Sl a I M M X E a 9 y g u C T P j n P X Z N p 7 E 1 K 5 7 6 5 n 4 m 1 F q V u 9 Z r k 3 x r m 9 J A 3 Z / j n M e N I 4 r 7 m n F v T k p V y / y U R e x j w M c 0 T z P U M U V a q i b K h / x h G f r 2 p L W x M o + p V Y h 9 + z h 2 7 I e P g B i s 5 J K < / l a t e x i t > r Pre-Encoder < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 j r x d V I J 8 S q G / g y nV i 9 h f U s o 6 3 o = " > A A A C y n i c j V H L S s N A F D 2 N r / q u u n Q T L I K r k o i o y 6 I b F y 4 q 2 A e 0 R Z J 0 2 g 5 N k z C Z C C V 0 5 w + 4 1 Q 8 T / 0 D / w j v j F N Q i O i H Jm X P P u T P 3 X j 8 J e S o d 5 7 V g L S w u L a 8 U V 9 f W N z a 3 t k s 7 u 4 0 0 z k T A 6 k E c x q L l e y k L e c T q k s u Q t R L B v L E f s q Y / u l T x 5 j 0 T K Y + j W z l J W H f s D S L e 5 4 E n i W p 2 h p 7 M 4 + l d q e x U H L 3 s e e A a U I Z Z t b j 0 g g 5 into a vector and inputted into a context encoder. For the context encoder, RESeL utilizes RNN such as GRU [8] and Mamba [14] , to mix sequential information and extract non-observable state information from the observable context. The MLP policy comprises a two-layer architecture, each layer consisting of 256 neurons. The architecture of the pre-encoder and context encoder for the RESeL critic mirrors that of the policy. To improve training stability, RESeL adopts the ensemble-Q technique from REDQ [21] , employing 8 MLP critic models, and each one of them comprises two hidden layers with 256 neurons.To locate the source of instability, we first formalize the RNN and its sequence generation process. An RNN network f θ (x t , h t ) : X × H → Y × H is defined as: y t , h t+1 = f θ (x t , h t ), where θ represents the network parameters, x t , y t , and h t are the RNN input, output, and hidden state, respectively. For simplicity, we denote y t = f θ y (x t , h t ) and h t+1 = f θ h (x t , h t ). Given an input sequence {x 0 , x 1 , . . . , x T } of length T + 1 and an initial hidden state h 0 , the output sequence generated by f θ is denoted as {y 0 , y 1 , . . . , y T }. Let θ ′ be the parameter neighboring θ after a onestep gradient update. The output sequence produced by f θ ′ is denoted as {y ′ 0 , y ′ 1 , . . . , y ′ T }. With certain assumptions, we can derive Proposition 1, which bounds the difference between y t and y ′ t at any time step t. Proposition 1. Assuming f θ and f θ ′ both satisfy Lipschitz continuity, i.e., for all θ ∈ {θ, θ ′ }, x ∈ X , h, h ′ ∈ H, there exist constants K h ∈ [0, 1) and K y ∈ R such that:f θ h (x, h) -f θ h (x, h ′ ) ≤ K h ∥h -h ′ ∥ , f θ y (x, h) -f θ y (x, h ′ ) ≤ K y ∥h -h ′ ∥ ,and for all x ∈ X , h ∈ H, the output differences between the RNN parameterized by θ and θ ′ are bounded by a constant ϵ ∈ R:max f θ h (x, h) -f θ ′ h (x, h) , f θ y (x, h) -f θ ′ y (x, h) ≤ ϵ,the RNN output difference at t-th step is bounded byEQUATIONRefer to Appendix A for a detailed proof. Proposition 1 focuses on the case where K h ∈ [0, 1), as most RNN models adhere to this condition. GRU [8] and LSTM [7] meet this requirement through the use of sigmoid activation functions, while Mamba [14] achieves it by constraining the values in the hidden state's transition matrix.In Eq. (1), ϵ represents the maximum variation of the model output when the model parameters are modified and both the model input and hidden state remain constant, i.e., the first-step output variation. However, from Eq. ( 1), we observe that due to the variability in the hidden state, the final variation in the model output is amplified. This amplification factor is minimal at t = 0, being 0. As t increases, this amplification factor gradually increases and ultimately converges to β := Ky 1-K h . Additionally, it can be verified that as t increases, the upper bound of the average variation in network output . Applying different learning rates to different modules is similar to Two-Timescale Network [35] , which has been shown to improve the training convergence.Based on the SAC framework [20] , combining context-encoder-specific learning rate and the ensemble-Q mechanism from REDQ [21] , we developed the practical RESeL algorithm. Specifically, we initially configure context-encoder-specific optimizers for both the policy and value models.After each interaction with the environment, RESeL samples a batch of data from the replay buffer, containing several full-length trajectories. This batch is used to compute the critic loss according to REDQ, and the critic network is optimized using its context-encoder-specific optimizer. Notably, unlike REDQ, we did not adopt a update-to-data ratio greater than 1. Instead of that, we update the network model once per interaction. The policy network is updated every two critic updates. During the policy update, the policy loss from REDQ is used, and the policy network is optimized with its context-encoder-specific optimizer. Here, we delay the policy update by one step to allow more thoroughly training of the critic before updating the policy, which is inspired by TD3 [36] to improve the training stability. The detailed algorithmic procedure is provided in Appendix C and Algorithm 1. 
experiments: 
 To validate our argument and compare RESeL with other algorithms, we conducted a series of validation and comparison experiments across various POMDP environments. We primarily considered three types of environments: classical partially observable environments, meta-RL environments, and credit assignment environments. To assess the generalizablity of RESeL to MDP tasks, we also test RESeL in five MuJoCo [37] locomotion tasks with full observation provided. We implement the policy and critic models with a parallelizable RNN, i.e., Mamba [14] to accelerate the training process. A detailed introduction of the network architecture can be found in Appendix D.3.2.In all experiments, the RESeL algorithm was repeated six times using different random seeds, i.e. 1-6. All experiments were conducted on a workstation equipped with an Intel Xeon Gold 5218R CPU, four NVIDIA RTX 4090 GPUs, and 250GB of RAM, running Ubuntu 20.04. For more detailed experimental settings, please refer to Appendix D.The updates of the RNN lead to large variations in the model output. In Sec. 4.2, we observe that the variations in model output induced by a model update are amplified. In this part, we quantify the amplification factor, assess its impact on RL training, and discover how a context-encoder-specific learning rate addresses this issue. We use a POMDP task, specifically WalkerBLT-V-v0, to validate our argument. We loaded a policy model trained by RESeL and updated it with various LR CE and Figure 3 : Action variations as the rollout step increases after a single gradient update with different values of LR CE and LR other . Action variation refers to the change in policy output after the gradient update compared to its output before the update, using the same input sequences.LR other for a one-step gradient update. The output differences for all rollout steps between the preand post-update models are presented in Fig. 3 . The right panel shows the variations in model output as the rollout step increases, while the left panel zooms in on the rollout steps from 0 to 75.In the zoom-in panel, comparing the brown-circle and yellow-triangle curves reveals that at the first rollout step, the two curves are very close. This indicates that with the same learning rate, identical input, and hidden states, the updates of RNN and MLP have almost the same impact on the model's output. However, as the rollout steps increase, the brown-circle curve gradually rises, increasing by 400% at the 20-th step. This observation aligns with Proposition 1, demonstrating that changes in the hidden state lead to increasing variations in the model's output. With further increases in rollout length, the right panel shows that the brown-circle curve converges at around 0.4 after 250 steps, indicating that the amplification of the output variations eventually converges. This also meets Proposition 1. Ultimately, the increase in action variations is approximately tenfold.This action variation magnitude is equivalent to an MLP policy network (the gray-dashed curve) with a learning rate increased twentyfold. At this point, the learning rate reaches 0.006, which is typically avoided in RL scenarios due to the risk of training instability. Conversely, reducing LR CE to 10 -5 (the purple-diamond curve) significantly suppresses the amplification of variations. The right panel shows that the orange and purple curves remain at similar levels until the final time step.Large output variations lead to instability in RL training. To investigate how the large variations influence the RL process, we visualized the gradient norm and value function loss during training for two POMDP tasks, as shown in Figs. 4 and 5 . We fixed LR other = 3 × 10 -4 and compared the training processes of LR CE = 10 -5 (red line) and 3 × 10 -4 (orange line). For the latter, we applied gradient clipping; otherwise, training would diverge and stop early. The orange line in Fig. 4 shows the norm of the policy gradient before gradient rescaling, where significant oscillations appear in the gradient in the later stages of training. The orange gradient norm is eventually scaled to 0.5, consistently smaller than the red line, which does not show oscillations. This suggests that the late-stage instability of the orange line does not stem from large gradient magnitudes.On the other hand, as seen in Fig. 5 , the value loss for the orange line remains consistently high, with extremely large values appearing in later stages. This is due to large variations in the RNN output, which lead to an unstable bootstrapped update target for the value function. Consequently, the value loss diverges, causing the abnormal gradient norm observed in Fig. 4 . The results in Figs. 4 and 5 reveal the unique challenges of using RNN structures in RL, where traditional RNN stabilization techniques may no longer be effective.Using a context-encoder-specific learning rate improves training stability, while traditional RNN stabilization techniques fall short. To further examine the impact of a lower learning rate on the stability of RL training, we conducted experiments across several POMDP tasks. Building on the setup in Fig. 4 , we further evaluated a variant without the gradient clipping technique and another that replaces gradient clipping with a truncation of recurrent backpropagation steps to 32 (green line). For reference, we also included a variant where the CE output is entirely masked to zero (purple line).Comparing the vanilla RNN-based RL (blue line) with the purple line shows that introducing an RNN can partially address partial observability issues and improve policy performance. However, the blue line still exhibits instability, even triggering early stopping due to outliers. Setting a specific learning rate for the CE (red line) significantly enhances training stability, indicating that reducing the RNN learning rate indeed improves overall RL training stability. However, we also found that traditional RNN stabilization techniques do not reliably improve RL training stability. For example, truncating the number of recurrent backpropagation steps (green line) can still lead to early stopping, while clipping the gradient norm (orange line) prevents early stopping but does not improve the policy performance. This is because these two stabilization techniques can only suppress gradient explosions in the RNN, keeping the RNN gradients within a normal range. However, the output of the RNN can still vary significantly, which continues to cause instability in RL. WalkerBLT-V-v0 . Some learning curves in AntBLT-V and HalfCheetahBLT-V are incomplete as some runs encountered infinite or NaN outputs.In this part, we compare RESeL with previous methods in various tasks. The detailed introduction of the baselines and tasks can be found in Appendix D.2 and Appendix D.1. More comparative results can be found in Appendix E.3.Classic POMDP Tasks. We first compare RESeL with previous baselines in four PyBullet locomotion environments: AntBLT, HalfCheetahBLT, HopperBLT, and WalkerBLT. To create partially observable tasks, we obscure part of the state information as done in previous studies [18, 11, 15] , preserving only position (-P tasks) or velocity (-V tasks) information from the original robot observations. We benchmark RESeL against prior model-free recurrent RL (MF-RNN) [11] , VRM [18] , and GPIDE [38] . GPIDE, which extracts historical features inspired by the principle of PID [39] controller, is the state-of-the-art (SOTA) method for these tasks.The comparative results are shown in Fig. 7 . RESeL demonstrates significant improvements over previous recurrent RL methods (MF-RNN, PPO-GRU, and A2C-GRU) in almost all tasks, except for HalfCheetahBLT-P where its performance is close to that of MF-RNN. These results highlight the advantage of RESeL in classic POMDP tasks over previous recurrent RL methods. Furthermore, RESeL outperforms GPIDE in most tasks, establishing it as the new SOTA method. The learning curves of RESeL are more stable than that of GPIDE, suggesting that fine-grained feature design could also introduce training instability, while a stably trained RNN can even achieve superior performance.Dynamics-Randomized Tasks. Instead of directly obscuring part of the immediate state, meta-RL considers a different type of POMDP. In meta-RL scenarios [26] , the agent learns across various tasks with different dynamics or reward functions [16, 19] , where the parameters of the dynamics and reward functions are not observable. We first compare RESeL with previous methods in the dynamics randomization meta-RL tasks. Following previous meta-RL work [16] , we randomized the gravity in MuJoCo environments [37] . We created 60 dynamics functions with different gravities, using the first 40 for training and the remaining for testing. The gravity is unobservable, requiring the agents to infer it from historical experience. We compare RESeL to various meta-RL methods in these tasks, including ProMP [40] , ESCP [16] , OSI [33] , EPI [41] , and PEARL [42] . Notably, ESCP, EPI, and OSI use recurrent policies with different auxiliary losses for RNN training. ESCP and PEARL are previous SOTA methods for these tasks. The comparative results are shown in Fig. 8 .We find that in Ant, Humanoid, and Walker2d, RESeL demonstrates significant improvement over all other methods. In Hopper, RESeL performs on par with ESCP, while surpassing other methods. Specifically, RESeL outperforms SAC-RNN by a large margin, further highlighting its advantages. Moreover, ESCP, EPI, and OSI use RNNs to extract dynamics-related information, with ESCP having inferred embeddings highly related to the true environmental gravity. The advantages of RESeL over these methods suggest that the context encoder of RESeL may extract not only gravity (see Appendix E.1) but also other factors that help the agent achieve higher returns. Classic meta-RL Tasks. We further compare RESeL with baselines in other meta-RL tasks, particularly the tasks possessing varying reward functions, sourced from [19, 11] . In AntDir, CheetahDir, and HalfCheetahVel, the robots' target moving direction or target velocity are changeable and unobservable. Agents must infer the desired moving direction or target velocity from their historical observations and rewards. Wind is a non-locomotion meta-RL task with altered dynamics. We compare RESeL to previous meta-RL methods, including RL 2 [25] , VariBad [19] , and MF-RNN [11] .The learning curves of these methods are shown in Fig. 9 . In the AntDir and HalfCheetahDir tasks, we found that RESeL can converge within 5M steps. Our results show that RESeL demonstrates high sample efficiency and superior asymptotic performance in AntDir, CheetahDir, and HalfCheetahVel. In Wind, RESeL performs comparably to MF-RNN, as this task is less complex. These findings suggest that RESeL effectively generalizes to POMDP tasks with hidden reward functions. Credit Assignment Tasks. A notable application of recurrent RL is solving credit assignment problem [11, 15] . We also compare RESeL with MF-GPT [15] and MF-RNN [11] in a credit assignment task, namely Key-to-Door. In this task, it is required to assign a reward obtained at the last-step to an early action. We compared RESeL with algorithms MF-GPT and MF-RNN. As did in [15] Classic MuJoCo Locomotion Tasks. Finally, we would like to discover how RESeL performs in MDP tasks. We adopt five classic MuJoCo tasks and compare RESeL with previous RL methods, e.g., SAC [20] , TD3 [36] , and TD7 [43] . As the current SOTA method for these environment, TD7 introduces several enhancements based on TD3, e.g., a representation learning method and a checkpoint trick, which are not existing in RESeL. The results, listed in magnitude. Additionally, it can also be found from Fig. 11c that setting LR CE equally significantly degrades policy performance. This emphasizes the importance of using different learning rates for context encoder and other layers. We extended the experiments from Figs. 11a and 11c to additional seven POMDP tasks, with results presented in Fig. 20 in Appendix E.6. The conclusions drawn from Fig. 20 are largely consistent with those from Figs. 11a and 11c , indicating that the impact of the context-encoder-specific learning rate on training is relatively generalizable.Ablation Studies. We then explore whether the context-encoder-specific learning rate applies to other RNN or Transformer architectures [9] . We integrated a GRU [8] or Transformer as the context encoder while keeping all other settings constant. Due to the high computational cost of training GRUs and Transformers on full-length trajectories, we tested the RESeL variant with these architectures on four classic POMDP tasks for 0.75M time steps. Figure 12 displays the learning curves for various variants. MF-RNN [11] and SAC-Transformer [38] serve as baselines based on GRUs and Transformers from prior studies.We observe that the context-encoder-specific learning rate also improves and stabilizes the performance of the GRU and Transformer variants. Our findings show that both RESeL-GRU and RESeL-Transformer perform comparably to RESeL-Mamba on three out of four tasks. These results suggest that the performance gains of RESeL are not solely attributed to an advanced RNN architecture, and that different arhitectures may be suited to different tasks. Additional sensitivity experiments and ablation studies on context length can be found in Appendices E.4 to E.6. 
i b e g w x g m e s t h e b 5 s e t p w 4 s a h r o u c o e: 
 G I 6 z j D F G v k z U j F S O E R O 6 L v g H Z t w 0 a 0 V z l T 7 Q 7 o l J B e Q U 4 b h + S J S S c I q 9 N s H c 9 0 Z s X + l j v X O d X d J v T 3 T a 4 x s R J D Y v / y z Z T / 9 a l a J P o 4 1 z V w q i n R j K o u M F k y 3 R V 1 c / t L V Z I y J M Q p 3 K O 4 I B x o 5 6 z P t v a k u n b V W 0 / H 3 7 R S s W o f G G 2 G d 3 V L G r D 7 c 5 z z o H F c c U 8 r 7 s 1 J u X p h R l 3 E P g 5 w R P M 8 Q x V X q K G u q 3 z E E 5 6 t a 0 t Y E y v / l F o F 4 9 n D t 2 U 9 f A B b k J J H < / l a t e x i t > ô x Context Encoder (CE) x MLP Policy < l a t e x i t s h a 1 _ b a s e 6 4 = " + 3 T e C I E N 7 v o V 8 C x k m D 0 N 3 + Y 6 D a E = " > A A A C x H i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V k U x G U L 9 g F a J J l O a + g 0 C Z O J U I r + g F v 9 N v E P 9 C + 8 M 0 5 B L a I T k p w 5 9 5 4 z c + 8 N U x F l y v N e C 8 7 c / M L i U n G 5 t L K 6 t r 5 R 3 t x q Z U k u G W + y R C S y E w Y Z F 1 H M m y p S g n d S y Y N R K H g 7 H J 7 p e P u O y y x K 4 k s 1 T n l 3 F A z i q B + x Q B H V C G 7 K F a / q m e X O A t + C C u y q J + U X X K O H B A w 5 R u C I o Q g L B M j o u Y I P D y l x X U y I k 4 Q i E + e 4 R 4 m 0 O W V x y g i I H d J 3 Q L s r y 8 a 0 1 5 6 Z U T M 6 R d A r S e l i j z Q J 5 U n C + j T X x H P j r N n f v C f G U 9 9 t T P / Q e o 2 I V b g l 9 i / d N P O / O l 2 L Q h 8 n p o a I a k o N o 6 t j 1 i U 3 X d E 3 d 7 9 U p c g h J U 7 j H s U l Y W a U 0 z 6 7 R p O Z 2 n V v A x N / M 5 m a 1 X t m c 3 O 8 6 1 v S g P 2 f 4 5 w F r Y O q f 1 T 1 G 4 e V 2 q k d d R E 7 2 M U + z f M Y N V y g j q b x f s Q T n p 1 z R z i Z k 3 + m O g WIn this paper, we proposed RESeL, an efficient recurrent off-policy RL algorithm. RESeL tackles the training stability issue existing in previous recurrent RL methods. RESeL uses difference learning rates for the RNN context encoder and other fully-connected layers to improve the training stability of RNN and ensure the performance of MLP. Experiments in various POMDP benchmarks showed that RESeL can achieve or surpass previous SOTA across a wider variety of tasks, including classic POMDP tasks, meta-RL tasks, and credit assignment tasks.Limitations. We also notice that there are several noticeable limitations concerning this work: (1) We have shown that RESeL can even surpass SOTA methods in some MDP tasks, but it is unclear how the RNN helps the policy improve the performance. (2) RESeL introduces one more hyper-parameter, i.e., LR CE . It is would be nice to explore the relationship between optimal LR CE and LR other or automatically parameter-tuning strategy for LR CE .The following proof process is primarily based on the proof approach in [44, 45] . To the best of our knowledge, this is the first time the compounding error analysis method has been applied to study the properties of RNNs.Let an RNN network f θ (x t , h t ) : X × H → Y × H be formalized as:y t , h t+1 = f θ (x t , h t ),where θ is the network parameter, x t is the RNN input, y t is the RNN output, h t and h t+1 are the RNN hidden states. For simplicity, we denotey t = f θ y (x t , h t ), h t+1 = f θ h (x t , h t ).We consider another RNN parameter θ ′ , which is obtained by updating θ by one gradient step. We make the following assumptions:1. A1 (Lipschitz continuity): f θ and f θ ′ both satisfy the Lipschitz continuity. There exist K h ∈ [0, 1)3 and K y ∈ R satisfying:f θ h (x, h) -f θ h (x, h ′ ) ≤ K h ∥h -h ′ ∥ , ∀x ∈ X , h, h ′ ∈ H, (A1.1)EQUATIONf θ ′ y (x, h) -f θ ′ y (x, h ′ ) ≤ K y ∥h -h ′ ∥ , ∀x ∈ X , h, h ′ ∈ H. (A1.4)2. A2 (one-step output difference constraint): The output differences of the RNN parameterized by θ and θ ′ are bounded by a constant ϵ ∈ R:f θ h (x, h) -f θ ′ h (x, h) ≤ ϵ, ∀x ∈ X , h ∈ H, (A2.1)f θ y (x, h) -f θ ′ y (x, h) ≤ ϵ, ∀x ∈ X , h ∈ H. (A2.2)With the above assumptions, we now prove Proposition 1 in following.Proof. ∀x ∈ X , h, h ′ ∈ H, we haveEQUATIONSimilarly, ∀x ∈ X , h, h ′ ∈ H, we also haveEQUATIONGiven an input sequence {x 0 , x 1 , . . . , x T } with sequence length T + 1 and an initial hidden state h 0 , let the output and hidden state sequences produced by f θ be {y 0 , y 1 , . . . , y T } and {h 1 , . . . , h T +1 }.Similarly, let the output and hidden state sequences produced by f θ ′ be {y ′ 0 , y ′ 1 , . . . , y ′ T } and {h ′ 1 , . . . , h ′ T +1 }. We can measure the upper bound of the hidden state differences between f θ and f θ ′ .EQUATION)∥h 2 -h ′ 2 ∥ = f θ h (x 1 , h 1 ) -f θ ′ h (x 1 , h ′ 1 )Eq. ( 2)≤ K h ∥h 1 -h ′ 1 ∥ + ϵEq. ( 4)EQUATIONSimilarly, we haveEQUATION)EQUATIONThen we can derive the output difference at the t-th step (t > 0):∥y t -y ′ t ∥ = f θ y (x t , h t ) -f θ ′ y (x t , h ′ t )Eq. ( 3)≤ K y ∥h t -h ′ t ∥ + ϵEq. ( 7)EQUATIONAs 0 ≤ K h < 1, Eq. ( 8) can be further simplified toEQUATIONEq. ( 9) also holds for t = 0, i.e., ∥y 0y ′ 0 ∥ ≤ ϵ. Thus, we finally haveEQUATIONB Average Output Differences Bound Proposition 2. Following the setting in Proposition 1, the average output difference satisfiesEQUATIONProof. From Eq. ( 10), we have ∥y t -y ′ t ∥ ≤ K y 1 -K h ϵ + ϵ -K y ϵ 1 -K h K t h . As 1 t + 1 t i=0 K i h = 1 -K t+1 h (1 -K h )(t +τ ← ∅; Reset context â ← 0, ô ← 0, r ← 0; Reset hidden state h ← 0; Reset E, obtain new observation o t+1 from E; Sample data batch D = {τ i | i = 1, 2, . . . } with data count i |τ i | ≥ BS from B;Obtain REDQ [21] critic loss with Q ψ , Q ψ ′ , α, and D following Eq. ( 19); Update ψ with the critic optimizer via one-step gradient descent; Softly assign ψ to ψ ′ ; if t%2 = 0 then Obtain REDQ [21] policy loss with π ϕ , α, and D following Eq. ( 20); Update ϕ with the policy optimizer via one-step gradient descent;Automatically tune α to control the policy entropy to T E;t ← t + 1;we can get1 t + 1 t i=0 ∥y i -y ′ i ∥ ≤ K y 1 -K h ϵ + ϵ -K y ϵ 1 -K h 1 t + 1 t i=0 K t h = K y 1 -K h ϵ + ϵ -K y 1 -K t+1 h (1 -K h ) 2 (t + 1) ϵ.The training procedure of RESeL is summarized in Algorithm 1. In the following part, we elaborate on the form of REDQ [21] losses in the setting of recurrent RL. Let the critic Q ψ (o, a, ô, â, r, h Q ) : O × A × O × A × R → Ri = {o i 0 , a i 0 , r i 0 , o i 1 , a i 1 , r i 1 , o i 2 , . . . , r i Li-1 , o iLi } is the i-th trajectory in the batch, L i is the trajectory length. We then re-formalize τ i to tensorized data:o i ∈ O Li , a i ∈ A Li , ôi ∈ O Li , âi ∈ A Li , r i ∈ R Li , ri ∈ R Li , where, for example, o t i ∈ O = o i tis the t-th row of o i . ô0 i = 0, â0 i = 0, r0 i = 0 as the there is no last step at the first time step. The policy receives the tensorized data and outputs action distributions. For simplicity, we formalize the outputs of the policy as the action sampled from the action distribution and its logarithm sampling probability:EQUATIONwhere ãi ∈ A Li , log pi ∈ R Li , h Li π ∈ H are obtained autoregressively:EQUATIONSimilarly, the critic model receives the tensorized data and outputs 8-dim Q-values as follows:EQUATIONwhere Qi ∈ R Li×8 , h Li π ∈ H are obtained autoregressively:EQUATIONNote that, in Mamba, the process demonstrated in Eq. ( 13) and Eq. ( 15) can be parallelized.In order to compute the target value, we first extend the tensorized dataEQUATIONwhere o Li i is the next observation of the last step, o Li-1 i , a Li-1 i , and r Li-1 i are the observation, action, and reward at the last step, respectively.The target value can be obtained:EQUATIONThen, we sample a set M of 2 distinct indices from {0, 1, ..., 7}. Let Q+,j target,i be the j-th column of Q+ target,i . We choose the minimum value from { Q+,j target,i |j ∈ M}:EQUATIONwhere Q + target,i ∈ R Li+1 . We then let y i be the last L i rows of Q + target,i :y i = Q + target,i [1 : L i + 1]. The REDQ critic loss can be written asEQUATION)where Qi is obtained following Eq. ( 14), ∥ • ∥ 2 denotes L2-norm. As for the policy loss, we first obtain ãi and log pi via Eq. ( 12). We can get the critic value Q⋆ i corresponding to the new action ãi following Eq. ( 14):Q⋆ i , h Li Q ← Q ψ (o i , ãi, ôi , âi , ri , 0). We can get the policy loss:EQUATIONwhere denominator of 8 results from our ensemble size being 8, requiring the averaging of the outputs of the eight critics to compute the policy loss. J V f 4 K t F a S 2 e Z k s h i f a d 4 = " > A A A C z X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V 0 U U o i o i 6 L b t x Z w T 6 w F k n S a Q 3 N i 5 m J U K p u / Q G 3 + l v i H + h f e G d M Q S 2 i E 5 K c O fx x J 1 E V b l J 0 y L 4 v t z l W x Z F U s v c x J Y G e g h G z V 4 u I L L t F F D A 8 p Q j B E k I Q D O B D 0 t G H D Q k J c B y P i O C F f x x n u U C B t S l m M M h x i B / T t 0 6 6 d s R H t l a f Q a o 9 O C e j l p D S x R Z q Y 8 j h h d Z q p 4 6 l 2 V u x v 3 i P t= " > A A A C y 3 i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V l 0 4 0 a o Y B 9 Q S 0 m m 0 x q a J m F m I t T a p T / g V v 9 L / A P 9 C + + M K a h F d E K S M + e e c 2 f u v X 4 S B l I 5 z m v O m p t f W F z K L x d W V t f W N 4 q b W 3 U Z p 4 L x G o v D W D R 9 T / I w i H h N B S r k z U R w b + i H v O E P z n S 8 c c u F D O L o S o 0 S 3 h 5 6 / S j o B c x T R D W v l Z d 2 x s 6 k U y w 5 Z c c s e x a 4 G S g h W 9 W 4 + I J r d B G D I c U Q H B E U 4 R A e J D 0 t u H C Q E N f G m D h B K D B x j g k K 5 E 1 J x U n h E T u g b 5 9 2 r Y y N a K 9 z S u N m d E p I r y C n j T 3 y x K Q T h P V p t o m n J r N m f 8 s 9 N j n 1 3 U b 0 9 7 N c Q 2 I V b o j 9 y z d V / t e n a 1 H o 4 c T U E F B N i W F 0 d S z L k p q u 6 J v b X 6 p S l C E h T u M u x Q V h Z pv + N Y d D I = " > A A A C y i c j V H L S s N A F D N r p f V Z d u g k V w V R I R d V l a o Y B Q S n S a R a J i E z E W r t h w q / l / o H + h X f G K a h F d E K S M + e e c f u v X S c i E d z V n z c v L C l l w s r q v r G X N r b q I s z R g t S A O T p e K F P G I y W X I m k n K v K E f s o Y / O F P x x i L B Y + j K z l K W H v o S P e E n i W p e S y / r j N J p h y y o e i x w D S j B r G p c f M E u o g R I M M Q D B E k R A e B D t u H C Q E N f G m L i U E N d x h g k K M I x U j h E T u g b L c N G t F c h X Y H d E p I b p O G v k i U m X E l a n T q e c y K / S W O d U d x v R z e h s R K B D l + q / K P S L R w m u g V N N i W Z U d Y H J k u m u q J v b X q S l C E h T u E u x V P C g X Z O + x r j C q O v m l Y p V + B o M y r W K A Z / j n A X g J V H Y v D u V U z P q P H a w i a z E q O E c V N T H R z z h b q w h H V n X K r Z z x b O P b s h + A P F g k n = < / l a t e x i t > ⌧ 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " c n O u e u U G 1 w k k M n 7 D E p N l V z 1 q / p 4 = " > A A A C y 3 i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V Z I i 6 r L o x o 1 Q w T 6 g L S V J p 3 V o X m Q m Q q 1 d + g N uj P Q 1 a X X P q s F S f M C V y f N d 3 R u Y o 3 b 1 k i e B R e y 3 H M u o E z D P m A e 4 4 k q t W R T t q b V K a 9 Y s k q W 3 q Z 8 8 D O Q A n Z q k X F F 3 T Q R w Q P K Q I w h J C E f T g Q 9 L R h w 0 J M X B c T 4 h J C X M c Z p i i Q N y U V I 4 V D 7 I i + Q 9 q 1 M z a k v c o p t N u j U 3 x 6 E 3 K a O C B P R L q E s D r N 1 P F U Z 1 b s b 7 k n O q e 6 2 5 j + b p Y r I F b i h t i / f D P l f 3 2 q F o k B T n U N n G q K Na O q 8 7 I s q e 6 K u r n 5 p S p J G W L i F O 5 T P C H s a e e s z 6 b 2 C F 2 7 6 q 2 j 4 2 9 a q V i 1 9 z J t i n d 1 S x q w / X O c 8 6 B R K d v H Z f v q q F Q 9 y 0 a d x x 7 2 c U j z P E E V F 6 i h r u f 4 i C c 8 G 5 e G M O 6 M + 0 + p k c s 8 u / i 2 j I c P 8 8 G S g A = = < / l a t e x i t > ⌧ 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " Q E 8 q 6 Q U 5 x M S H 1 C L l 0 X Q 2 9 1 N b R z 4 = " > A A A C y 3 i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I V d V l 0 4 0 a o Y B / Q l j J J p z U 0 L y Y T o d Y u / Q G 3 + l / i H + h f e G d M Q S 2 i E 5 K c O f e c O 3 P v d W L f S 6 R l v e a M u f m F x a X 8 c m F l d W 1 9 o 7 i 5 V U + i V L i 8 5 k Z + J J o O S 7 j v h b w m P e n z Z i w 4 C x y f N 5 z h u Y o 3 b r l I v C i 8 l q O Y d w I 2 C L 2 + 5 z J J V L M t W d o d H 0 6 6 x Z J V t v Q y Z 4 G d g R K y V Y 2 K L 2 i j h w g u U g T g C C E J + 2 B I 6 G n B h o W Y u A 7 G x A l C n o 5 z T F A g b 0 o q T g p G 7 J C + A 9 q 1 M j a k v c q Z a L d L p / j 0 C n K a 2 C N P R D p B W J 1 m 6 n i q M y v 2 t 9 x j n V P d b U R / J 8 s V E C t x Q + x f v q n y v z 5 V i 0 Q f p 7 o G j 2 q K N a O q c 7 M s q e 6 K u r n 5 p S p J G W L i F O 5 R X B B 2 t X P a Z 1 N 7 E l 2 7 6 i 3 T 8 T e t V K z a u 5 k 2 x b u 6 J Q 3 Y / j n O W V A / K N v H Z f v q q F Q 5 y 0 a d x w 5 2 s U / z P E E F F 6 i i p u f 4 i C c 8 G 5 d G Y t w Z 9 5 9 S I 5 d 5 t v F t G Q 8 f 9 i K S g Q = = < / l a t e x i t > ⌧ 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " j 8 t d A w u 9 T V Z Z g E u 1 b G F y k s P x T P Q = " > A A A C y 3 i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I p 6 r L o x o 1 Q w T 6 g L S V J p 3 V o X i Q T o d Y u / Q G 3 + l / i H + h f e G e c g l p E J y Q 5 c + 4 5 d + b e 6 8 Y + T 4 V l v e a M h c W l 5 Z X 8 a m F t f W N z q 7 i 9 0 0 i j L P F Y 3 Y v 8 K G m 5 T s p 8 H r K 6 4 M J n r T h h T u D 6 r O m O z m W 8 e c u S l E f h t R j H r B s 4 w 5 A P u O c I o l o d 4 W S 9 S W X a K 5 a s s q W W O Q 9 s D U r Q q x Y V X 9 B B H x E 8 Z A j A E E I Q 9 u E g p a c N G x Z i 4 r q Y E J c Q 4 i r O M E W B v B m p G C k c Y k f 0 H d K u r d m Q 9 j J n q t w e n e L T m 5 D T x A F 5 I t I l h O V p p o p n K r N k f 8 s 9 U T n l 3 c b 0 d 3 W u g F i B G 2 L / 8 s 2 U / / X J W g Q G O F U 1 c K o p V o y s z t N Z M t U VC 9 x H H 3 v D w m Q a E O B O r u X R + A t 1 B p o = " > A A A C y 3 i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R L x t S y 6 c S N U s A 9 o S 5 m k 0 x q a F 5 O J U G u X / o B b / S / x D / Q v v D O m o B b R C U n O n H v O n b n 3 O r H v J d K y X n P G 3 P z C 4 l J + u b C y u r a + U d z c q i d R K l x e c y M / E k 2 H J d z 3 Q l 6 T n v R 5 M x a c B Y 7 P G 8 7 w X M U b t 1 w k X h R e y 1 H M O w E b h F 7 f c 5 k k q t m W L O 2 O j y b d Y s k q W 3 q Z s 8 D O Q A n Z q k b F F 7 T R Q w Q X K Q J w h J C E f T A k 9 L R g w 0 J M X A d j 4 g Q h T 8 c 5 J i i Q N y U V J w U j d k j f A e 1 a G R v S X u V M t N u l U 3 x 6 B T l N 7 J E n I p 0 g r E 4 z d T z V m R X 7 W + 6 x z q n u N q K / k + U K i J W 4 I f Y v 3 1 T 5 X 5 + q R a K P U 1 2 D R z X F m l H V u V m W V H d F 3 d z 8 U p W k D D F x C v c o L g i 7 2 j n t s 6 k 9 i a 5 d 9 Z b p + J t W K l b t 3 U y b 4 l 3 d k g Z s / x z n L K g f l O 3 j s n 1 1 W K q c Z a P O Y w e 7 2 K d 5 n q C C C 1 R R 0 3 N 8 xB O e j U s j M e 6 M + 0 + p k c s 8 2 / i 2 j I c P + u S S g w = = < / l a t e x i t > ⌧ 5< l a t e x i t s h a 1 _ b a s e 6 4 = " B j p o a T i H m j n In RESeL, we collect several full-length trajectories from the replay buffer as a batch for policy training. However, in many environments, trajectory lengths could vary a lot, making it impossible to directly concatenate them into a single tensor for subsequent computations. Typically, we pad trajectories to the same length with zeros, resulting in a tensor of shape [B, L, C], where B is the batch size, L is the sequence length, and C is the tensor dimension. Additionally, we introduce a mask tensor of shape [B, L, 1], where elements equal to 1 indicate valid data, and 0s indicate padding.X b i d 6 / f S O j A i G 0 o g = " > A A A C y 3 i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R K R 6 r L o x o 1 Q w T 6 g L S W Z T m t o X i Q T o d Y u / Q G 3 + l / i H + h f e G e cE N v 4 D F H E N X q C C f r T S r T X r F k l S 2 1 z H l g a 1 C C X r W o + I I O + o j A k C E A R w h B 2 I e D l J 4 2 b F i I i e t i Q l x C y F N x j i k K 5 M 1 I x U n h E D u i 7 5 B 2 b c 2 G t J c 5 U + V m d I p P b 0 J O E w f k i U i X E J a n m S q e q c y S / S 3 3 R O W U d x v T 3 9 W 5 A m I F b o j 9 y z d T / t c n a x E Y 4 F T V 4 F F N s W J k d U x n y V R X 5 M 3 N L 1 U J y h A T J 3 G fK P 4 J u o B J u s = " > A A A C y 3 i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V l 0 4 0 a o Y B 9 Q S 0 m m 0 x q a J m F m I t T a p T / g V v 9 L / A P 9 C + + M K a h F d E K S M + e e c 2 f u v X 4 S B l I 5 z m v O m p t f W F z K L x d W V t f W N 4 q b W 3 U Z p 4 L x G o v D W D R 9 T / I w i H h N B S r k z U R w b + i H v O E P z n S 8 c c u F D O L o S o 0 S 3 h 5 6 / S j o B c x T R D W v l Z d 2 x s 6 k U y w 5 Z c c s e x a 4 G S g h W 9 W 4 + I J r d B G D I c U Q H B E U 4 R A e J D 0 t u H C Q E N f G m D h B K D B x j g k K 5 E 1 J x U n h E T u g b 5 9 2 r Y y N a K 9 z S u N m d E p I r y C n j T 3 y x K Q T h P V p t o m n J r N m f 8 s 9 N j n 1 3 U b 0 9 7 N c Q 2 I V b o j 9 y z d V / t e n a 1 H o 4 c T U E F B N i W F 0 d S z L k p q u 6 J v b X 6 p S l C E h T u M u x Q V h Z p z T P t v G I 0 3 t u r e e i b 8 Z p W b 1 n m X a F O / 6 l j R g 9 + c 4 Z 0 H 9 o O w e l d 3 L w 1 L l N B t 1 H j v Y x T 7 N 8 x g V n K O K m p n j I 5 7 w b F 1 Y 0 r q z 7 j + l V i 7 z b O P b s h 4 + A O 7 / k n 4 = < / l a t e x i t > ⌧ 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " U P 9 z g n r X H O c 7 I 5 f 5 r q v + N Y 8 d D 8 I = " > A A A C y 3 i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V l 0 4 0 a o Y B 9 Q S 0 n S a R 2 a J i E z E W r t 0 h 9 w q / 8 l / o H + h X f G K a h F d E K S M + e e c 2 f u v X 4 S c i E d 5 z V n z c 0 v L C 7 l l w s r q 2 v r G 8 X N r b q I s z R g t S A O 4 7 T p e 4 K F P G I 1 y W X I m k n K v K E f s o Y / O F P x x i 1 L B Y + j K z l K W H v o 9 S P e 4 4 E n i W p e S y / r j N 1 J p 1 h y y o 5 e 9 i x w D S j B r G p c f M E 1 u o g R I M M Q D B E k 4 R A e B D 0 t u H C Q E N f G m L i U E N d x h g k K 5 M 1 I x U j h E T u g b 5 9 2 L c N G t F c 5 h X Y H d E p I b 0 p O G 3 v k i U m X E l a n 2 T q e 6 c y K / S 3 3 W O d U d x v R 3 z e 5 h s R K 3 B D 7 l 2 + q / K 9 P 1 S L R w 4 m u g V N N i W Z U d Y H J k u m u q J v b X 6 q S l C E h T u E u x V P C g X Z O + 2 x r j 9 C 1 q 9 5 6 O v 6 m l Y p V + 8 B o M 7 y r W 9 K A 3 Z / j n A X 1 g 7 J 7 V H Y v D 0 u V U z P q P H a w i 3 2 a 5 z E q O E c V N T 3 H R z z h 2 b q w h H V n 3 X 9 K r Z z x b O P b s h 4 + A P F g k n 8 = < / l a t e x i t > ⌧ 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " B j p o a T i H m j n X b i d 6 / f S O j A i G 0 o g = " > A A A C y 3 i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R K R 6 r L o x o 1 Q w T 6 g L S W Z T m t o X i Q T o d Y u / Q G 3 + l / i H + h f e G e cWe then use a modified mean operation:mask_mean(x, mask) = (x × mask)/sum(mask),to prevent padding from biasing the mean. This approach allows for GPU parallelization of all trajectories in the batch, though calculations for padded 0s are redundant. If trajectory lengths vary greatly, the proportion of padding can become significant, severely reducing computational efficiency, even resulting in out-of-GPU-memory. For instance, in a batch with one trajectory of length 1000 and ten trajectories of length 50, there would be 1500 valid data points and 9500 padded points, substantially lowering efficiency.To address this issue, we adopt the batch sampling method illustrated in Fig. 13 , stacking shorter trajectories along the time dimension. This requires the RNN to accept a hidden reset flag, which we insert at the beginning of each trajectory to reset the hidden state. Additionally, we insert K steps of zero data between trajectories to prevent convolution1d from mixing adjacent trajectory data, as shown in Fig. 13 . To implement this sampling strategy, we modify the original Mamba implementation 4 , adding an input denoting hidden reset flag. When occurring the hidden reset flag with value of 1, the hidden state will be reset to zero. The introduction of hidden reset flag will not influence the parallelizability of Mamba but can avoid potential redundant computation. These classic POMDP tasks5 are developed using PyBullet locomotion environments. The environments include AntBLT, HopperBLT, WalkerBLT, and HalfCheetahBLT. Partial observability is introduced by obscuring parts of the observation. For each environment, two types of tasks are created: one where only joint velocity is observable and another where only joint position is observable. We denote these as "-V" for joint velocity information and "-P" for joint position information. In such settings, the agent must reconstruct the missing information, whether velocity or position, based on historical observations.The Dynamics-Randomized Tasks6 environments are based on the MuJoCo environment [37] , following the training settings in [16] . In these tasks, the environment's gravity is altered using an open-source tool 7 . The gravity is sampled as following procedure: We first sample a value a ∈ R uniformly from [-3, 3] . Let the original gravity in MuJoCo be g origin , the new gravity is obtained as:EQUATIONBefore training, we sample 60 values of g new . Of these, 40 are used for training and 20 for testing the learned policies. In these tasks, the agent must infer the gravity based on historical interactions with the environment and adjust its policy accordingly.These meta-RL tasks 8 have been utilized in various meta-RL algorithms [42, 19] . Specifically, in these tasks, the reward function is modified in the Ant and HalfCheetah MuJoCo environments. In the 'Vel' tasks, the agent is rewarded based on its proximity to a desired velocity. In the 'Dir' tasks, the agent must move in a desired direction. The desired velocity and direction information are reflected only in the reward function. In the Wind environment, an external, unobservable disturbance factor (wind speed) influences state transitions. The agent must navigate to a goal under varying wind conditions. In these tasks, the agent uses past interaction data, including rewards, to infer the underlying reward functions or transition disturbance factors.The Key-to-Door environment9 consists of three phases. In the first phase, the agent spawns at a random location on the map, and a key will appear at another location on the map, the agent can touch the key. In the second phase, the agent will have the task of picking up apples. In the third phase, if the agent has touched the key in the first phase, it can receive a reward; otherwise, it cannot.The action space is four-dimensional, encompassing the actions of moving forward, backward, left and right. The reward structure is dual-faceted: it includes rewards garnered from the collection of apples during the second phase and the ultimate reward obtained upon door opening in the final phase. The transition function is defined by the positional changes of the agent subsequent to the execution of any of the four actions, with stage transitions dictated by an intrinsic timer, which facilitates a shift to the subsequent stage upon the expiration of the countdown. The optimal policy should locate and touch the key in the initial phase, which is served for the final reward acquisition upon door opening in the third phase, while maximizing apple collection during the second phase. The credit assignment length in Fig. 10 denotes the time period staying at the second stage.In the section, we give brief introductions of the baseline algorithms we comparing in our work. We first introduce the baselines in the classic POMDP tasks Fig. 7 .• Variational Recurrent Model (VRM) [18] . VRM utilizes a variational RNN [46] to learn a transition model, and feeding the RNN hidden state into an MLP policy to obtain the action.• GPIDE (GPIDE-ESS) [38] . Inspired by the PID controller's success, which relies on summing and derivative to accumulate information over time, GPIDE propose two history encoding architectures: one using direct PID features and another extending these principles for general control tasks by consisting of a number of "heads" for accumulating information about the history.• Recurrent Model-free RL (MF-RNN) [11] . MF-RNN proposes several design considerations in recurrent off-policy RL methods. MF-RNN analyzed the best choice concerning the context length, RL algorithm, whether sharing the RNN between critic and policy networks, etc. We used the best results reported in [11] in Fig. 7 .The baseline results in Fig. 7 come from two sources: (1) The evaluation results provided in [11] 10 , including MF-RNN, PPO-GRU, SAC-MLP, TD3-MLP, VRM, and A2C-GRU; (2) The results provided in [38] 11 , consisting GPIDE-ESS and SAC-Transformer.The baselines compared in Fig. 8 are elaborated as follows:• Environment Probing Interaction Policy (EPI) [17] . This baseline aims to extracting features from the transition. A context encoder is obtained to enable a transition model to predict the future state through minimizing the following loss:EQUATIONwhere T (s t , a t , z t i ; θ T ) is a transition model parameterized by θ T , z t i is the output of the context encoder. In addition, the transition model is also optimized as described in Eq. ( 22).• Online system identification (OSI) [33] . OSI is structured by a context encoder which is used to predict the parameters of the environment dynamics. The architecture of OSI is the same as Fig. 1 . OSI adopts a supervised learning loss function to train the context encoder:L OSI CE = E z t i ∥z t i -c i ∥ 2 2, where c i is the dynamics parameter such as gravity, z t i is the embedded context.• Environment sensitive contextual policy learning (ESCP) [16] 12 . This approach is design to improve both the the robustness and sensitivity context encoder, based on a contrastive loss.In order to improve the policy robustness in environment where dynamics sudden changes could occurs, ESCP truncate the memory length in both training and deployment phase.• Proximal Meta-Policy Search (ProMP) [40] 13 . This method learns a pretrained policy model in training phase and finetune it in deployment phase with few gradient steps. The reported performance of ProMP is obtained by the policies after finetuning.• Soft actor-critic with MLP network (SAC-MLP) [20] . An implementation of the soft actor-critic algorithm with MLP policy and critic models.• Probabilistic embeddings for actor-critic RL (PEARL) [42] . The context encoder of PEARL is trained to make a precise prediction of the action-value function while maximize entropy. The loss of the context encoder can be written as:L PEARL CE = L critic + D KL [p(z | τ ) ∥ N (0, I)]where L critic is the critic loss in SAC. D KL [p(z|τ ) ∥ N (0, I)] denotes the KL divergence between distribution of the embedded context and a standard Gaussian distribution. In p(z|τ ), z denotes the output of the context encoder based on a given trajectory. PEARL is implemented with the officially provided codes 14 . • SAC-GRU. This is an upgrade version of SAC with GRU models. It use standard SAC algorithm to optimize an RNN policy.The baselines compared in Fig. 9 :• RL 2 [25]. RL 2 is a meta-RL algorithm that trains a meta-learner to efficiently adapt to new tasks by leveraging past experiences. It involves two nested loops: an inner loop where the meta-learner interacts with the environment to learn task-specific policies, and an outer loop where the meta-learner updates its parameters based on the performance across multiple tasks. The inner loop is accomplished by an RNN. • VariBad [19] . VariBAD (Variational Bayes for Adaptive Deep RL) is also a meta-RL method that uses variational inference to enable fast adaptation to new tasks. It employs a Bayesian approach to meta-learning, inferring task-specific posterior distributions over the parameters of the policy and value function. Specifically, it first learns an RNN variational auto-encoder to extract the sequential features of the dynamics or tasks. The policy then receives the current state and the output of the context encoder to make adaptive decisions.The results of the baselines in Fig. 9 are sourced from [11] .MF-GPT [15] , i.e. the baseline in Fig. 10 , denotes to train a Transformer network, i.e. GPT-2, to solve the memory-based tasks. The baseline result of the baselines in Fig. 10 are from [15] 15 .In our implementation, we use Mamba to form the context encoder, as it supports processing sequences in parallel. Therefore, we will first introduce state space models and Mamba before discussing the network architecture.State Space Model (SSM) is a sequential model [47, 14, 48] commonly used to model dynamic systems. The discrete version of an SSM can be expressed as:EQUATIONwhere ∆, A, B, and C are parameters; x t , y t , and h t represent the input, output, and hidden state at time step t, respectively. A and B are derived from ∆, A, and B.Mamba [14] is an RNN variant that integrates SSM layers to capture temporal information. In Mamba, the parameters ∆, B, and C are dependent on the input x t , forming an input-dependent selection mechanism. Unlike traditional RNNs which repeatedly apply Eq. ( 23) in a for-loop to generate the output sequence, Mamba employs a parallel associative scan, which enables the computation of the output sequence in parallel. This method significantly enhances computational efficiency.Pre-encoder. The pre-encoders consist of one-layer MLPs with a consistent output dimension of 128.Context Encoder. The context encoder is comprised of one or more Mamba blocks, each flanked by two MLPs. Typically, the MLP before the Mamba block consists of a single layer with 256 neurons, except for the Key-to-Door task, where a three-layer MLP with 256 neurons per layer is employed.Except in Key-to-Door task, a single Mamba block is utilized, with a state dimension of 64. The kernel size of the convolution1d layer is fixed at 8. Additionally, for these tasks, a position-wise feedforward network, used by Transformer [9] , is appended to the end of the Mamba block to slightly enhance stability during training. In Key-to-Door tasks, the convolution1d layer in Mamba is omitted, as the focus shifts to long-term credit assignment, rendering the convolution1d unnecessary for capturing short-term sequence features. Conversely, in Key-to-Door tasks with a credit assignment length of 500, four Mamba blocks are employed to improve the long-term memory capacity.The MLP after the Mamba block also comprises a single layer with an output dimension of 128.MLP Policy/Critic. As detailed in Sec. 4.1, both the MLP policy and critic consist of MLPs with two hidden layers, each comprising 256 neurons.Activations. Linear activation functions are employed for the output of the pre-encoder, context encoder, and the MLP policy/critic. For other layers, the ELU activation function [49] is used.Optimizer. We use AdamW [50] for RESeL training.The hyperparameters used for RESeL is listed in Table 2 . We mainly tuned the learning rates, batch size for each tasks. γ and last reward as input are determined according to the characteristic of the tasks. The batch size in Table 2 (1000 or 2000) is much larger than that used in previous MLP-based SAC algorithms (128 or 256) [20] . This is because we train on at least one complete trajectory at a time, and in most environments, the maximum trajectory length is 1000. Thus, batch sizes of 1000 or 2000 correspond to one or two trajectories of maximum length. Hyperparameter Determination Some hyper-parameters (discount factor, reward as input) were chosen based on the nature of the environments, while others (Context encoder LR, policy and value function LR, and batch size) are determined via grid search:• Context Encoder LR: We fixed the learning rate of the context encoder to be 1/30 of the policy learning rate in all tasks. We aim to have the action variation caused by RNN updates to be comparable to that caused by MLP updates, thus fixing the learning rate ratio between the context encoder and MLP. After conducting parameter searches in several environments, as shown in Fig. 20 , we find the 1/30 ratio performed well, so we fixed this ratio across all environments.• Policy and Value Function LR: We mainly referred to the default learning rate settings in CleanRL [51] , using a policy learning rate of 3 × 10 -4 and an action-value function learning rate of 10 -3 . We conducted a grid search between the aforementioned learning rates and those reduced by a factor of 5 (aiming to further stabilize training). We found that a smaller learning rate was more stable in tasks where the partial observability is not very significant (e.g., MDPs like classic MuJoCo). • Discount Factor (γ): For general tasks, we used a common γ value of 0.99. For environments requiring long-term memory (e.g., Key-to-Door), we used a larger γ of 0.9999. • Reward as Input: The Classic Meta-RL tasks require inferring the true reward function based on real-time rewards. Thus, we also included real-time reward feedback as an input to the policy and value in these tasks. • Batch Size: We conducted a grid search for batch size, mainly exploring sizes that were 1x or 2x the maximum trajectory length. We found that in classic POMDP tasks, a larger batch size performed better. In other tasks, the advantage of a large batch size was not significant and could even reduce training speed.E.1 Visualization In order to explore what the context encoder has learned, we attempt to conduct in-depth analysis on the embedded context data. We conducted experiments in the environment of Halfcheetah with varying gravity. We uniformly sample gravity acceleration within the range of [2.90, 33.10]m/s 2 , resulting in 40 environments with different gravities. For each environment, we used a policy learned by RESeL to collect one trajectory, with a length of 1000 steps. Ultimately, we obtained data of size 40×1000, where each step's output of the encoder, a 128-dimensional vector, was saved. Subsequently, we performed t-SNE [52] processing on the 40×1000×128 data to map the high-dimensional 128-dimensional data into 2-dimensional data which can assist in our visualization analysis.Finally, we presented the results in Fig. 14 . In this figure, the colorbar represents the gravity acceleration magnitude. The x and y axes of the figure represent the two-dimensional data results after t-SNE mapping. From the figure, we can distinctly observe the non-random distribution characteristics of colors from left to right. The different data represented by the cool and warm colors are not randomly intermingled but rather exhibit certain clustering characteristics. We can find that the embeddings are strongly correlated with gravity acceleration, indicating that RESeL has recovered the hidden factor of the POMDP environment from the historical interaction data.We find the computational efficiency is also a common issue among previous recurrent RL methods, especially in the context of recurrent off-policy RL. This issue mainly comes from the non-parallelizable nature of traditional RNNs. In our work, we tackle this problem by using the SSM structure, i.e. Mamba to improve the computational efficiency. Mamba is able to process sequential data in parallel based on the technique of parallel associative scan. In the following parts, we will analyze the time cost of Mamba compared with that of GRU. Baseline results are primarily sourced from [43] . In environments such as Hopper, Walker2d, and Humanoid, RESeL achieves or surpasses the previous SOTA performance. Notably, RESeL may underperform compared to previous baselines at the 300k time step. However, its asymptotic performance eventually matches or exceeds the SOTA. This suggests that incorporating RNN structures might slightly reduce sample efficiency initially, as the observation space expands to include full historical data, requiring more optimization to find optimal policies. Nevertheless, RESeL ultimately leverages the enlarged observation space to achieve superior asymptotic performance. Since trajectory length (or called context length) in training data is often a critical parameter in previous studies [11] , our work has emphasized the importance of using complete trajectories.Training with trajectory segments but deploying with full-trajectory length can lead to distribution shift issues. To investigate this, we conducted experiments using full trajectory data versus trajectory segments (64 steps). The results are illustrated in Fig. 17 .In general, models trained with full trajectory data perform better than those trained with trajectory segments. However, in the AntBLT-V and HalfCheetah-V environments, the performance gap is smaller. The key difference in these tasks is the longer trajectory lengths, especially in HalfCheetahBLT-V, where the trajectory length is fixed at 1000 steps. The data in these tasks are usually cyclic, while in other tasks, agents may drop and terminate trajectories before achieving stable performance. We suspect that strong cyclic data can mitigate distribution shift issues, as trajectory segments containing a complete cycle can effectively represent the properties of the full trajectory.AntBLT-V-v0 0 Answer: [Yes] Justification: The codes and corresponding hyper-parameters can be found in https: //github.com/FanmingL/Recurrent-Offpolicy-RL. Guidelines:• The answer NA means that paper does not include experiments requiring code. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: This paper presents work whose goal is to advance the field of recurrent reinforcement learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.The inclusion of reward (r) in policy inputs is optional and depends on the RL task throughout the paper.We only consider K h ∈ [0, 1) as most RNN models satisfy this condition. GRU/LSTM and SSMs achieve this condition by adopting a sigmoid activation function and directly limits the value of the transition matrix of the hidden state, respectively. The RNN with a K h > 1 could diverge during long-sequence forwarding.https://github.com/state-spaces/mambahttps://github.com/oist-cnru/Variational-Recurrent-Modelshttps://github.com/FanmingL/ESCPhttps://github.com/dennisl88/rand_param_envshttps://github.com/lmzintgraf/varibadhttps://github.com/twni2016/Memory-RLhttps://github.com/twni2016/pomdp-baselineshttps://github.com/ianchar/gpidehttps://github.com/FanmingL/ESCPhttps://github.com/jonasrothfuss/ProMPhttps://github.com/katerakelly/oysterhttps://github.com/twni2016/Memory-RL 
 
 Appendix:  


Decision: While Reviewer B2hv votes for rejecting the paper due to insufficient depth, the three remaining reviewers vote in favor of acceptance provided extensive changes are made. 
Both views are well-founded and comprehensible.  In my view, these views are best summarized as “Borderline Accept” and as there are no major flaws in the paper, it depends on where the bar is set for NeurIPS 2024 whether this paper can be accepted.

Reviews: review: paper_summary: paper_summary: The paper contributes to the important and long-standing issue of representing latent state information for successfully finding optimal RL control policies in the context of partially observable Markov decision processes (POMDP). The major contribution is a newly composed RL algorithm (RESeL), which, by design, enables a thorough study of the impact and importance of the actual learning rate (LR) used for the recurrent (latent) context encoder part of the solution architecture. The author highlight the fact that current state-of-the-art (SOTA) methods commonly use an equal LR for training both the latent representation and the policy/critic networks. 

Their idea of separating the LR between recurrent CE and policy networks, as well as the overall performance benefits w.r.t current SOTA algorithms, is demonstrated by an extensive survey of selected POMDP tasks and benchmarks. These experiments together with theoretical arguments let the authors conclude that using a single  (and often too large) LR parameter approach of previous methods leads to sub-optimal performance results in solving POMDP tasks.
strength_weakness: strength_weakness: Even though mentioned in the Limitiation section, it would have been very helpful to add some benchmark comparisons of the RESeL algorithm with its RNN-CE block entirely bypassed/skipped to clearly separate the impact of the latent space presentation from the rest of the RL algorithm. In particular for the classic POMDP tasks, where only position and velocity features are masked, respectively, it is important to know a baseline performance when simply feeding the current and last-step observation vector into the MLP networks (given that RESeL uses 2 x 256 neurons layers in their MLPs, adding a few more inputs show be more than tractable). Including such extension (not necessarily for all benchmarks but maybe a smaller subset) could significantly improve the scientific insight.

Having clear focus on the impact of varying LR the remainder of the RESeL algorithm is entirely built from existing RL building blocks, like the overarching SAC RL architecture or convential RNN/GRU components and, hence, provides no original contribution in itself (as correctly mentioned by the authors, using RNN for latent space representation is known for a long time, as is the SAC approach). Re-using existing building blocks per-se is not a bad idea (and I don't recommend addressing this issue in the rebuttal phase) but one has to accept that the algorithmic novelty in this work is hence limited. 

As a minor remark: the authors spend a relatively large portion of their introduction on the mathematical formulation of Proposition 1, whose central statement (variation of RNN outputs grow as function of rollout-step and LR) appears not entirely original (if it really is a first-time, the authors should highlight this fact more strongly) since the result is quite intuitive for people working in the RNN/RL community. Besides that, the connection between lowering the LR as part of the update rule when training RNNs and the resulting formula of Proposition 1 is not obvious and could be highlighted more strongly.
questions: questions: l. 127: Why is the last-step observation also needed as input to the MLP if both current obs and context-encoded obs are already fed into the  NN? According to Fig. 2, the last-step obs is fed into the CE only (but not into the MLP)

l. 132: Is there any justification/reasoning behind the choice of the MLP architecture, in particular why a relatively large capacity (2 x 256 neurons) was chosen? Is there an estimate of how much this choice impacts the overall performance results of the policy training?

l. 163-164: It seems to me that the claim of Proposition 1 and the subsequential statements ("average variations in the RNN output ... are amplified") are not a novel discovery but have been studied and found by others in similar or different contexts of RNN before. In that case, it would be good to provide corresponding citations of pior art or - if the others believe this is the first time such a claim was made - highlight the significance of their finding. 

l. 165-166: Please provide further arguments/proof, why the effect of Proposition 1 can be mitigated by smaller learning rate (even if such claim sounds plausible, it would be good to refer to previous work or more detailed reasoning). In other words: how does the learning rate affect the result of ||y - y'|| as a result of Propsotion 1 (it is likely related to epsilon in the formula but if so this relation could be mentioned more explicitly)?

l. 167-168: Similar remark as before: this sentence remarks a general claim about MLPs which should either be backed up by reference citations or by stronger reasoning or explanations.

l. 195: Given the typical level of stochasticity or dependency on random intial states, a trial number of 6 seems relatively low to incorporate statistical fluctuations in the return evaluation, in particular for the following ablation studies. Have the authors made sure that their reported results are not prone to larger statistical errors? What in detail is included in the choice of "different random seeds"? Does it only refer to the initial weight setup of the RNNs/MLPs or does it reflect varying intial conditions of the environemnts? Do all environments provide deterministic rollouts?

Figure 3: Why are error bars only visible for the brown curves/datapoints?

l. 218: How is "action variation after one-step update" as a function of rollout steps defined in the case of MLPs as they don't have an intrinsic time-convolution? I.e., what is the meaning of the i-th rollout time step in the case of MLP "only" (LR_CE = 0)? And why is that variation not close to zero for the 0-th rollout step, as indicated in Figure 3 (gray dashed curve) but starting from a value around 0.6?

l. 223-229: At this point of studying the performance in various benchmarks as a function of LR, and also for some of the subsequent ablation studies of Sec. 5.3, it would be great to show a benchmark reference where the CE module is skipped or bypassed altogether, feeding the current and last obserivation only into the MLP networks but not using a CE representation of the latent state at all (while there is one case where LR_CE = 0 in Fig. 9a, this is still different from skipping the entire module and following the suggestion above). This would generally help to underline the significance of the CE for the chosen set of benchmarks and tasks.

Fig. 6: It appears that SAC does hardly profit from using a recurrent CE (SAC-GRU) if compared to the MLP variant (SAC-MLP); is there any explanation why? It could stress the importance of showing the RESeL performance with and without CE, as mentioned in my comment above

Minor remarks:

l. 126: Better don't use " ' " behind name of algorithm ("RESeL's") but speak of RESeL policy as a joint expression

l. 171: Printing style (formatting) of scientific number notation "3e - 4" looks somewhat odd; check if correct Math formula style was chosen in the LaTex document or even better use decimal notation ("$3 10^-4$") or capital "E" instead of "e". This applies to other instances of number formatting of the main text as well.

l. 240: SOTA (state-of-the-art) should be introduced as an abbreviation

Fig. 5 (and others): Quality of plots could generally be improved by avoiding overlapping of images and axes captions in some cases

Fig. 6: Colors/symbols of "EPI" and "SAC-MLP" are hardly distinguishable
limitations: limitations: The authors have addressed the main limitation well. As they state, what is missing is a study on the impact of using the RNN as a context encoder in general, which refers to my request for providing a reference baseline where the CE is bypassed.
paper_summary: paper_summary: The paper mitigates the training instability issue of the recurrent off-policy RL by using a smaller learning rate for the RNN block.
strength_weakness: strength_weakness: - My main concern is the novelty of the proposed solution. Although the paper gives a reason for using different learning rate for the RNN block and the MLPs, the proposed method is to choose a lower learning rate for the RNN block. In practice, it's somehow common to choose different learning rate for different components. For example, Dreamer [Hafner, Danijar, et al.], which also uses an RNN block in their model, chooses a smaller learning rate for the RNN-based world model. Also, when facing the training stability issue, tuning the learning rate is always on the checklist. 

    However, I would consider accepting the paper if it proposes an approach to automatically decide the learning rate of the RNN block by leveraging the analysis in section 4.2.
- Some details need further explanation:
    - In equation (1), it could be better to distinguish two $\epsilon$.
    - Will the amplification factor $\beta = \frac{K_y}{1-K_h}$ always larger than 1 (L160)?
    - The average variation in network output $\frac{1}{t}\sum^{t-1}_{i=0} || y_i - y'_i||$ converges to $\beta \epsilon + \epsilon$, which is not involved in $t$, but when this indicates with **longer** sequence lengths, the average variations in the RNN output induced by gradient deselect are amplified (L163-164)?
    - In the caption of Fig 3, what does it mean by "after one-step gradient-update"? Does it mean one-gradient update per environment step?
    - In L218, why does the learning rate of MLP must be increased **twentyfold**?
    - In L221, "The right panel shows the green and blue curve remain at similar levels until the final time step". Where are the green and blue curves?
questions: questions: See weakness above
limitations: limitations: The limitation is discussed.
paper_summary: paper_summary: This paper proposes RESeL which improves recurrent off-policy RL in POMDPs mainly by applying a lower learning rate to the context encoder. This is justified by their theoretical analysis that the recurrence amplifies the output difference in the long run. In practice, they also incorporate several techniques (Mamba architecture, critic ensembles, efficient data sampling, slow policy update). The main contribution is on strong empirical performance. Across several benchmarks (classic POMDP, meta-RL, credit assignment), RESeL attains SOTA performance.
strength_weakness: strength_weakness: No major weaknesses. Please see the questions below.

One missing point is about the theoretical understanding -- why a smaller learning rate on context encoder can stabilize training? Reducing the output variation between consecutive updates, in my opinion, is a starting point, but not enough to explain the stability as a whole. This is connected with two-timescale update of feature vs value learning, e.g., https://openreview.net/forum?id=rJleN20qK7 Perhaps this related work is worth a discussion.
questions: questions: About architectural design. Is there a reason for using respective context encoders in actor-critic? 

About proposition 1. The connection between a smaller learning rate and the bound (1) is not very clear. Is the logic that a smaller learning rate leads to a smaller \|\theta - \theta’\| and thus a smaller \epsilon, then a tighter upper bound in (1)? 

Is this a typo in the unusually large batch size described in Table 2? 

As the main finding is on the learning rates, could the sensitivity analysis (Figure 9) be applied to more tasks? 

Indicated by Figure 10, is RESeL-GRU more sample-efficient than RESeL-Mamba? Is Mamba a crucial component of RESeL?
limitations: limitations: Yes, no major limitations.
paper_summary: paper_summary: The authors investigate the stability of training deep recurrent policies for POMDP tasks. They hypothesize that the recurrent part of the encoder faces less stable training compared to the fixed-length parts of the encoder (e.g. the MLP input layer) and propose that the former should use a lower learning rate. This is shown to be effective mostly through an empirical evaluation.
strength_weakness: strength_weakness: Unfortunately, I believe the work is not sufficiently novel for acceptance at NeurIPS. The problem identified by the authors --- that a recurrent encoder is numerically less stable than a fixed-length encoder --- has been identified many times over the years with various techniques proposed to mitigate it. This includes classical works such as the original LSTM paper [1] aimed at tackling exponentially scaling gradients. More recently, techniques such as gradient clipping and truncating the number of recurrent backpropagation steps (which is not the same as shortening the context length) have been suggested to tackle this problem as well.

Experimentally, I would have liked to see that these last two tricks do not already solve the stability issues, as there was no mention of them in the paper. It also seems that the hyperparameters were tuned by hand (since I could not find any mention of a hyperparameter selection scheme) which I think is insufficient for a work for which the sole contribution is an empirical result that we are meant broadly apply. I would have expected to see a very rigorous and objective hyperparameter selection method such as a grid search over reasonable hyperparameters. While I appreciated the sensitivity analysis in Fig 9, it was only conducted on a single domain.

Ultimately however, my opinion is that the idea is not substantial enough for publication in a top venue.
 
[1]: Hochreiter, Sepp, and Jürgen Schmidhuber. "Long short-term memory." Neural computation 9.8 (1997): 1735-1780.
questions: questions: 1. How were hyperparameters tuned in the experiments?
2. Have you considered the approaches mentioned in the weaknesses? (gradient clipping and truncated gradients)
limitations: limitations: Yes.

Comments: title: Thank you
Re: Additional Comment
Re: Rebuttal
comment: Thank you for your response. Due to time and computational resource limitation, we were only able to add two additional environments during the rebuttal phase. We will continue with sensitivity studies and include the results and discussion of the remaining tasks in the next version of our paper. We appreciate your constructive suggestions, which have helped improve our paper.
We are glad that our rebuttal was able to address your concerns. Your suggestions have been very helpful to us. On the other hand, should you have any additional questions or lingering concerns, please do not hesitate to contact us.
Thank you for the rebuttal and the additional experiments, although what I expected was a thorough sensitive analysis across all the tasks (currently only 3 tasks are shown).

Overall the paper and the rebuttal looks good to me. I keep my rating.
Thanks for the detailed reply and the efforts in preparing the rebuttal. After reading your rebuttal, I agree that although only a separate learning rate is used for RNN, the paper explains the motivation behind this and supports the conclusion with good experiments. I would improve my score to borderline accept.
Thank you very much for your prompt response. We will address each of your concerns in detail.

1. **[The scope of the results]** We observed that significant RNN output variations can introduce instability in value learning when using bootstrap-based methods. This is because the target for value learning, $r(s,a) + \gamma \hat{Q}(s',a')$, experiences substantial changes. Off-policy RL algorithms such as TD3, SAC, TD7, and DQN, which heavily rely on value estimation, are especially prone to this instability. Additionally, in actor-critic off-policy RL methods, the optimization objective of the policy, $\max_{\pi} Q(s,a)$, also experiences large fluctuations, leading to even greater instability. On the other hand, on-policy algorithms, especially PPO and TRPO, have trust region constraints that prevent significant changes in policy outputs, which inherently mitigates the impact of RNN output variations. However the value function in these methods can still exhibit instability. We believe that incorporating the RESeL framework into on-policy algorithms could further enhance training stability. In supervised learning, the impact of large RNN output variations is less significant. This is mainly due to the use of fixed labels, which contributes to more stable training. Additionally, certain loss functions in supervised learning, such as MSE loss in the case of continuous labels, naturally suppress the issue of large RNN output variations as the gradient decreases rapidly with the loss. We will include this discussion in the revised version of the paper.

2. **[This finding]** Yes, we are the first to identify this fundamental cause of instability in recurrent off-policy RL.

Please feel free to reach out if you need further clarification on any point. We are more than happy to answer any additional questions you may have.
Thank you for clarifying the difference between gradient instability vs large output variations and also for gathering further results for truncated BPTT. This is much appreciated.

I now realize that I misinterpreted the equation in Proposition 1 to mean the output variation exponentially increases with $t$ when in fact it is bounded by a constant factor. I will reassess the work with this in mind. It would help if you could answer the following questions:

- Could you give insight on the scope of the results with respect to off-policy RL vs RL in general vs supervised learning? The theoretical foundations don't seem particularly tied to off-policy RL, and the other paradigms also use neural architectures with recurrent and MLP layers. Is there any reason this work specifically targets off-policy RL? 

> To the best of our knowledge, this finding has not been reported in previous work. 

- When you mentioned the above statement, which finding are you specifically referring to? Is it that "the fundamental cause of instability in Recurrent off-policy RL is not gradient instability, but rather excessively large output variations between consecutive updates of the RNN"?
Thank you for your reply. It appears that there may be a misunderstanding. The reviewer seems to believe that the instability in Recurrent off-policy RL is due to numerical explosion or instability of RNN gradients, and that addressing the gradient issues alone would resolve the overall training stability. This understanding aligns with the traditional issues of RNNs and corresponding solutions. However, there will be different issues in off-policy RL.

Our experimental results (as shown by the red line in Fig. R3) indicate that the gradient explosion problem is not significant, as advanced RNNs (such as GRU, Mamba, etc.) have already addressed gradient instability quite effectively. If the gradients were indeed growing exponentially with the number of steps, merely reducing the RNN learning rate by 30 times would not be sufficient to counteract this exponential divergence, whereas gradient clipping would effectively mitigate such issues. Contrary to this, the experimental results show that our method is significantly more stable than gradient clipping. 

This is because the fundamental cause of instability in Recurrent off-policy RL is not gradient instability, but rather excessively large output variations between consecutive updates of the RNN. Even if the gradients are not large (on the same order of magnitude as those in MLPs), the outputs of the recurrent policy/value function can vary significantly between updates. These large output changes introduce instability into RL. For instance, in value function training, the (simplified) optimization target is $||Q(s,a)-r(s,a)-\gamma\hat{Q}(s',a')||_2^2$, where $\hat{Q}$ and $a'$ are the target value function network and the outputs of the current policy on $s'$, respectively. If the outputs of the value function network and policy network vary significantly, $a'$ will change greatly, and $\hat{Q}(s',a')$ will also fluctuate substantially. This results in large shifts in the optimization target of the value function after each update, leading to training instability. Similar instabilities occur in policy training, contributing to overall RL training instability. This is why, as shown in Fig. R4, even after clipping the gradients, the value loss remains unstable. **To the best of our knowledge, this finding has not been reported in previous work.** We will include these discussions in the revised version of our paper.

Our theoretical results demonstrate that output variation does not increase exponentially with the number of steps but converges to a certain value. Therefore, we can balance the output variations, amplified by time steps, with a small RNN-specific learning rate. In contrast, if gradient clipping or truncation is used to keep the gradients within a normal range (similar to MLPs), the RNN output variations will still be large, and RL training will remain unstable.

The results of Gradient truncation with 4 or 8 truncation steps are shown in the table below. In both variants, the algorithm produced infinite outputs before 250 iterations, leading to early stopping. 



|                     |           $LR_{CE}$=$10^{-5}$           | $LR_{CE}$=$3\times10^{-4}$, grad-step-truncation-4 | $LR_{CE}$=$3\times10^{-4}$, grad-step-truncation-8 | $LR_{CE}$=$3\times10^{-4}$, grad-step-truncation-32 |
| :-----------------: | :-------------------------------------: | :------------------------------------------------: | :------------------------------------------------: | :-------------------------------------------------: |
|     AntBLT-V-v0     | $ \mathbf{1986} \pm\mathbf{73} ^\star$  |                    $408\pm 74$                     |                    $269\pm 88$                     |                      $499±55$                       |
| HalfCheetahBLT-V-v0 | $ \mathbf{2679} \pm\mathbf{176} ^\star$ |                   $-107\pm 320$                    |                   $-703\pm 330$                    |                     $−458±244$                      |
Thank you to the authors for the effort put into the rebuttal. I appreciate the inclusion of an investigation on gradient clipping and truncating the number of gradient steps, although I would have liked to see truncations with a smaller number of steps as well. Many implementations consider 4 or 8 steps (e.g. [1,2]) which may do well given that the output variation is exponential in the number of steps -- truncating to 32 steps doesn't provide much insight into whether it can tackle the exponential variation.

I'm still not convinced about the novelty of the work after reading the authors' rebuttal and the other reviews. The authors claim that the novelty is not so much in their implementation trick but in the theoretical and intuitive understanding gained from this work. However, the idea that RNNs tend to be more unstable than MLPs is not a particularly new or surprising finding. Or is the novel understanding something different from what I just described?

In reply to Reviewer 89BQ's argument for the novelty of this work, I can agree that the trick of using separate learning rates is not considered common practice. With a more rigorous investigation to ensure that current methods don't already solve this, such as gradient truncation with a smaller number of steps, I can agree that this knowledge could be useful to many practitioners (though perhaps it doesn't require a 9-page NeurIPS paper to disseminate). However, this paper doesn't seem to offer much beyond that. Most papers that propose a new algorithm also come with new and surprising insights that future works can build off of. Unfortunately, with this work, I don't think the insights are sufficiently novel. I don't see them pushing a research area forward, inspiring future works, or significantly improving our scientific understanding. 

I've increased by score since the experimentation was improved, but unfortunately my main complaint regarding lack of novelty still stands.

[1] https://github.com/MarcoMeter/recurrent-ppo-truncated-bptt?tab=readme-ov-file 

[2] https://github.com/lcswillems/torch-ac?tab=readme-ov-file
Thank you very much for your prompt response and kind comments. We are delighted that our reply has addressed your concerns. Your valuable suggestions have greatly improved our work.
I'd like to thank the authors for their detailed response to my remarks and concerns, and in particular for their additional study of bypassing the CE to provide another baseline for the performance evaluation. Together with other explanations and clarifications done in the revised version my remaining questions and suggestions also have been sufficiently addressed. Hence, I continue to vote for "Accept (7)".

Towards the main concern regarding lack of novelty, as raised in my own review and reflected by some of the other reviewers' remarks: It is true that tuning the LR in any context of machine learning is on the checklist of every good practioner, so I was tempted to come to the same conclusion in the beginning that this work doesn't add enough novelty regarding this aspect. What changed my final perspective is related to the answer given by the authors in their rebuttal to some of the other reviews: In the particular context of multi-component POMDP-RL solution architectures it is not common practice to use separate LR for modules which otherwise contribute to the same overall loss function. And even if that idea was previously applied in this particular context, it would highly likely not entail a comprehensive study as provided in the present work.