Recurrent Model-Free RL is a Strong Baseline for Many POMDPs

1 INTRODUCTION While reinforcement learning ( RL ) is often cast as the problem of learning a single fully observable task , also known as MDP , training and testing on that same task , most real-world applications of RL demand some degree of transfer and partial observability . For example , visual navigation ( Zhu et al. , 2017 ) requires adaptation to unseen scenes with occlusion in observations , and human-robot collaboration requires that robots infer the intentions of human collaborators . ( Chen et al. , 2018 ) . Many subareas in RL study problems that are special cases of POMDPs , and we summarize them in Table 1 . For example , meta RL ( Duan et al. , 2016 ; Schmidhuber , 1987 ; Thrun & Pratt , 2012 ; Wang et al. , 2017 ) is a POMDP where certain aspects of the reward function or ( less commonly ) dynamics function are unobserved but held constant through one episode . The robust RL problem ( Bagnell et al. , 2001 ; Pattanaik et al. , 2018 ; Pinto et al. , 2017 ; Rajeswaran et al. , 2017a ) assumes that certain aspects of the dynamics or reward function are unknown , aiming at finding optimal policies that perform against adversarially-chosen perturbations . Generalization in RL ( Cobbe et al. , 2019 ; Packer et al. , 2018 ; Whiteson et al. , 2011 ; Zhang et al. , 2018a ) focuses on unobserved aspects of the dynamics or reward function that are novel during testing , using an average-case objective instead of a worst-case objective like robust RL . Recent work has proposed efficient and performant algorithms for solving these specialized problem settings . However , these algorithms often make assumptions that preclude their application to other classes of POMDPs . For example , methods for robust RL are rarely used for the meta RL setting due to objective mismatch ; methods for meta RL are rarely used for general POMDPs due to the stationarity assumption in meta RL . 1Code : https : //drive.google.com/drive/folders/1I5mLlKPf2Gmdpm0nzy9OkR494nCJll1g ? usp=sharing Nonetheless , many prior works have used a simple baseline that is applicable to all POMDPs : model-free RL equipped with a recurrent policy and ( sometimes ) value function ( Duan et al. , 2016 ; Fakoor et al. , 2020 ; Igl et al. , 2018 ; Packer et al. , 2018 ; Rakelly et al. , 2019 ; Wang et al. , 2017 ; Yu et al. , 2019 ) . We will refer to this approach as recurrent model-free RL . This baseline is simultaneously simple ( requiring changing only a few lines of code from a model-free RL algorithm ) and general . However , prior work has consistently found that recurrent model-free RL performs poorly across a wide range of problem settings , including meta RL ( Rakelly et al. , 2019 ; Zintgraf et al. , 2020 ) , general POMDPs ( Han et al. , 2020 ; Igl et al. , 2018 ) , robust RL ( Zhang et al. , 2021 ) , and generalization in RL ( Packer et al. , 2018 ) . One common explanation is that specialized algorithms that are tailored to specific types of POMDPs are very likely to outperform recurrent model-free RL because they ( implicitly ) encode inductive biases for solving these specific tasks . For example , algorithms for meta RL may leverage the assumption that the underlying dynamics ( while unknown ) are fixed , and the underlying goals are fixed within one episode ( Rakelly et al. , 2019 ; Zintgraf et al. , 2020 ) ; algorithms for robust RL may assume that the dynamics parameters are known ( Rajeswaran et al. , 2017a ) and dynamics is Lipschitz continuous ( Jiang et al. , 2021 ) . This paper challenges this explanation . We argue that , contrary to popular belief , recurrent modelfree RL is competitive with recent state-of-the-art algorithms across a range of different POMDP settings . Similar to prior work in Markovian on-policy RL methods ( Andrychowicz et al. , 2021 ; Engstrom et al. , 2020 ) , our experiments show that implementation in recurrent model-free RL matters . Fig . 1 shows a typical scenario in PyBullet occlusion environments ( Coumans & Bai , 2016 ) to support this argument . Through extensive experiments , we show that the careful design and implementation of recurrent model-free RL is critical to its performance . Design decisions , such as the actor-critic architecture , conditioning on previous actions and/or rewards , the underlying model-free RL algorithms , and context length in RNNs , are especially crucial . The main contribution of this paper is a performant implementation of recurrent-model free RL . We demonstrate that simple yet important design decisions , such as the underlying RL algorithm and the context length , yield a recurrent model-free RL algorithm that performs on par with prior specialized POMDP algorithms on the environments those algorithms were designed to solve . Ablation experiments identify the importance of these design decisions . We also open-sourced our code that is easy to use and memory-efficient . 2 BACKGROUND . MDP . A Markov decision process ( MDP ) ( Bellman , 1957 ) is a tuple ( S , A , T , T0 , R , H , γ ) , where S is the set of states , A is the set of actions , T : S × A × S → [ 0 , 1 ] is the transition function ( dynamics ) , T0 : S → [ 0 , 1 ] is the initial state distribution , R : S × A × S → R is the reward function , H ∈ N is the time horizon , and γ ∈ [ 0 , 1 ) is the discount factor . Solving an MDP requires learning a memoryless policy π : S × A → [ 0 , 1 ] that maximizes the expected discounted return : π∗ = argmaxπ Est , at , rt∼T , π [ ∑H−1 t=0 γ trt+1 | s0 ] . For any MDP , there exists an optimal policy that is both memoryless and deterministic ( Puterman , 2014 ) . MaxEnt RL algorithms ( Ziebart , 2010 ) , such as SAC ( Haarnoja et al. , 2018a ) , add an entropy bonus to the RL objective . POMDP . A partially observable Markov decision process ( POMDP ) ( Åström , 1965 ) is a tuple ( S , A , O , T , T0 , O , O0 , R , H , γ ) , where the underlying process is an MDP ( S , A , T , T0 , R , H , γ ) . Let O be the set of observations and let O : S × A ×O → [ 0 , 1 ] be the emission function . Let the observable trajectory up to time-step t be τ0 : t = ( o0 , a0 , o1 , r1 , . . . , at−1 , ot , rt ) , the memory-based policy in the most general form is defined as π ( at | τ0 : t ) , conditioning on the whole history . At the first time step t = 0 , an initial state s0 ∼ T0 ( · ) and initial observation o0 ∼ O0 ( · | s0 ) are sampled . At any time-step t ∈ { 0 , . . . , H − 1 } , the policy emits the action at ∈ A to the system , the system updates the state following the dynamics , st+1 ∼ T ( · | st , at ) , then the next observation is sampled ot+1 ∼ O ( · | st+1 , at ) and the reward is computed as rt+1 = R ( st , at , st+1 ) . We refer to the part of the state st at current time-step t that can be directly unveiled from current observation ot as the observable state sot , and the rest part of the state as the hidden state sht . We call the hidden state s h t stationary if it does not change within an episode . In this scenario , the policy objective can be rewritten as π∗ = argmaxπ Esh∼T0 [ Est , at , rt∼T , O , O0 , π [ ∑H−1 t=0 γ trt+1 | sh ] ] for the average-case POMDP objec- 3 RELATED WORK . In this section , we discuss several subareas of RL that both explicitly and implicitly solve POMDPs , as well as algorithms proposed for these specialized settings . Table 1 summarizes these subareas . RL for “ Standard ” POMDPs . We use the term “ standard ” to refer to prior work that explicitly labels the problems studied as POMDPs . Common tasks include scenarios where the states are partially occluded ( Heess et al. , 2015 ) , different states correspond to the same observation ( perceptual aliasing ( Whitehead & Ballard , 1990 ) ) , random frames are dropped ( Hausknecht & Stone , 2015 ) , observations use egocentric images ( Zhu et al. , 2017 ) , or the observations are perturbed with random noise ( Meng et al. , 2021 ) . These POMDPs often have hidden states that are non-stationary and affect both the rewards and the dynamics . POMDPs are hard to solve ( Littman , 1996 ; Papadimitriou & Tsitsiklis , 1987 ) because of the curse of dimensionality : the size of the history grows linearly with the horizon length . Many prior POMDP algorithms ( Cassandra et al. , 1994 ; Kaelbling et al. , 1998 ) attempt to infer the state from the past sequence of observations , and then apply standard RL techniques to that inferred state . However , the exact inference requires the knowledge of the dynamics , emission , and reward functions , and is intractable in all except the most simple settings . A common strategy for solving these general POMDPs is to use recurrent policies , which take the entire history of past observations as inputs ( Bakker , 2001 ; Schmidhuber , 1991 ; Wierstra et al. , 2007 ) . This strategy is very simple and general , and can be applied to arbitrary tasks without knowledge of the task structure ( e.g. , whether the hidden states change within an episode ) across long time horizons ( Duan et al. , 2016 ) . These recurrent strategies can be further subdivided into model-free methods ( Hausknecht & Stone , 2015 ; Heess et al. , 2015 ; Meng et al. , 2021 ) , where the single objective is to maximize the return , and model-based methods ( Freeman et al. , 2019 ; Han et al. , 2020 ; Igl et al. , 2018 ; Watter et al. , 2015 ) that have explicit objectives on modeling the belief states and use them as the inputs of memoryless policies . The recurrent model-free RL that we focus on belongs to the class of model-free off-policy memory-based algorithms . Meta RL . Meta RL , also called “ learning to learn ” ( Schmidhuber , 1987 ; Thrun & Pratt , 2012 ) , focuses on POMDPs where some parameters in the rewards or ( less commonly ) dynamics are varied from episode to episode , but remain fixed within a single episode , which represent different tasks with different values ( Humplik et al. , 2019 ) . The meta RL setting is almost the same as multitask RL ( Wilson et al. , 2007 ; Yu et al. , 2019 ) , but differs in that multi-task RL can observe the task parameters , making it an MDP instead of a POMDP . Algorithms for meta RL can be roughly categorized based on how the adaptation step is performed . Gradient-based algorithms ( Fakoor et al. , 2020 ; Finn et al. , 2017 ; Hochreiter et al. , 2001 ) run a few gradient steps on the pre-trained models to adapt . Memory or context-based algorithms use RNNs to implicitly adapt , which can be further subdivided into implicit and explicit task inference methods . Implicit task inference methods ( Duan et al. , 2016 ; Wang et al. , 2017 ) use RL objective only to learn recurrent policies . Explicit task inference methods ( Rakelly et al. , 2019 ; Zintgraf et al. , 2020 ) train an extra inference model to explicitly estimate task embeddings ( i.e. , a representation of the unobserved parameters ) by variational inference . Task embeddings are then used as additional inputs to memoryless policies . Robust RL . The goal of robust RL is to find a policy that maximizes returns in the worst-case environments . Early work in the control and operations research community ( Khalil et al. , 1996 ; Nilim & Ghaoui , 2005 ) and RL community ( Bagnell et al. , 2001 ; Morimoto & Doya , 2005 ) focused on linear or finite systems . Prior work designs deep RL algorithms that are robust against a variety of adversarial attacks , including attacks on the dynamics ( Jiang et al. , 2021 ; Rajeswaran et al. , 2017a ) , observations ( Huang et al. , 2017 ; Pattanaik et al. , 2018 ; Zhang et al. , 2021 ) , and actions ( Gleave et al. , 2020 ; Pinto et al. , 2017 ; Tessler et al. , 2019 ) . Treating the robust RL problem as a POMDP , rather than an MDP ( as done in most prior work ) , unlocks a key capability for RL agents , because agents can use their memory to identify the hidden states of the current adversarial environment , although previous work ( Jiang et al. , 2021 ; Rajeswaran et al. , 2017a ) only train Markovian policies on POMDPs . While some work find memory-based policies are more robust to the adversarial attacks than Markovian policies ( Russo & Proutière , 2021 ; Zhang et al. , 2021 ) , they train these baselines in a single MDP without adversaries , which differs from our training setting where the recurrent model-free RL can have access to a set of MDPs . Generalization in RL . The goal of generalization in RL is to make RL algorithms perform well in test domains that are unseen during training , which emphasizes the average case on the novel test domains instead of the worse case in the possibly seen test domains as in robust RL . Prior work have studied generalization to initial states in the same MDP ( Rajeswaran et al. , 2017b ; Whiteson et al. , 2011 ; Zhang et al. , 2018b ) , random disturbance in dynamics ( Rajeswaran et al. , 2017b ) , states ( Stulp et al. , 2011 ) , observations ( Song et al. , 2020 ; Zhang et al. , 2018a ) , and actions ( Srouji et al. , 2018 ) , and different modes in procedurally generated games ( Cobbe et al. , 2019 ; Farebrother et al. , 2018 ; Justesen et al. , 2018 ) . Among them , Packer et al . ( 2018 ) provides a benchmark on both in-distribution ( ID ) and out-of-distribution ( OOD ) generalization to different dynamics parameters , and Zhao et al . ( 2019 ) extends the benchmark by introducing random noise in states , observations , and actions . Algorithms for improving generalization in RL can be roughly divided into classic regularization methods such as weight decay , dropout , batch normalization , and entropy regularization ( Cobbe et al. , 2020 ; Farebrother et al. , 2018 ; Igl et al. , 2019 ) , model architectures ( Raileanu & Fergus , 2021 ; Srouji et al. , 2018 ) , data augmentation through randomization ( Lee et al. , 2020 ; Tobin et al. , 2017 ) , Although introducing observational noise and the change in dynamics parameters will transform MDPs to POMDPs , few work study memory-based policies such as model-free recurrent RL with mixed results . Same algorithm RL2 ( Duan et al. , 2016 ) was found to perform badly in Packer et al . ( 2018 ) but relatively well in Yu et al . ( 2019 ) .

Summary: This paper is concerned with understanding when RNNs are useful in POMDPs. They should that, for a class of POMDPS (e.g., meta RL, robust RL), standard RNNs can be competitive with solutions that are tailored to the given POMDP structure. The authors discuss four design considerations that they claim are essential for performance: (1) decoupling actor and critic networks, (2) using off-policy instead of on-policy algorithms, (3) context length of RNN, and (4) using rewards as historical input. They should on 4 mujoco baselines that their results are competitive with sota methods whenever these design choices are taken into account.