Recurrent Model-Free RL is a Strong Baseline for Many POMDPs
Authors: Tianwei Ni
Benjamin Eysenbach
Sergey Levine
Ruslan Salakhutdinov
Venue: ICLR 2022 Submitted (2022)
Keywords: POMDP
RNN
recurrent model-free RL
baseline
meta RL
robust RL
generalization in RL

Abstract: Many problems in RL, such as meta RL, robust RL, and generalization in RL can be cast as POMDPs. In theory, simply augmenting model-free RL with memory, such as recurrent neural networks, provides a general approach to solving all types of POMDPs. However, prior work has found that such recurrent model-free RL methods tend to perform worse than more specialized algorithms that are designed for specific types of POMDPs. This paper revisits this claim. We find that a careful architecture and hyperparameter decisions yield a recurrent model-free implementation that performs on par with (and occasionally substantially better than) more sophisticated recent techniques in their respective domains. We also release a simple and efficient implementation of recurrent model-free RL for future work to use as a baseline for POMDPs.

Under review as a conference paper at ICLR 2022
RECURRENT MODEL-FREE RL IS A STRONG BASELINE FOR MANY POMDPS
Anonymous authors Paper under double-blind review
ABSTRACT
Many problems in RL, such as meta RL, robust RL, and generalization in RL, can be cast as POMDPs. In theory, simply augmenting model-free RL with mem- ory, such as recurrent neural networks, provides general approach to solving all types of POMDPs. However, prior work has found that such recurrent model-free RL. methods tend to perform worse than more specialized algorithms that are de- signed for specific types of POMDPs. This paper revisits this claim. We find that a careful architecture and hyperparameter decisions yield a recurrent model-free implementation that performs par with (and occasionally substantially better than) more sophisticated recent techniques in their respective domains. We also release simple and efficient implementation of recurrent model-free RL. for fu- ture work to use as a baseline for POMDPs."
INTRODUCTION
While reinforcement learning (RL) is often cast the problem of learning single fully observ- able task, also known as MDP, training and test- ing on that same task, most real-world appli- cations of RL demand some degree of transfer and partial observability. For example, visual navigation (Zhu et al., 2017) requires adapta- tion to unseen scenes with occlusion in observa- tions, and human-robot collaboration requires that robots infer the intentions of human col- laborators. (Chen et al., 2018).
Curs recument mocal-hwe ft.
peo, gni
Many subareas RL study problems that are special cases of POMDPs, and summa- rize them Table For example, meta RL (Duan al., 2016; Schmidhuber, 1987: Thrun & Pratt, 2012; Wang et al., 2017) is POMDP where certain aspects of the reward function or (less commonly) dynamics function are unob- served but held constant through one episode. The robust RI. problem (Bagnell et al., 2001; Pattanaik et al., 2018; Pinto al., 2017: Ra- jeswaran et al., 2017a) assumes that certain as- pects of the dynamics or reward function are unknown, aiming at finding optimal policies that perform against adversarially-hhh chosen perturba- tions. Generalization in RL (Cobbe et al., 2019; Packer et al., 2018; Whiteson et al., 2011; Zhang et al., 2018a) focuses unobserved aspects of the dynamics ing testing, using an average-case objective instead of a worst-case objective like robust RI. Recent work has proposed efficient and performant algorithms for solving these specialized problem set- tings. However, these algorithms often make assumptions that preclude their application to other classes of POMDPs. For example, methods for robust RL rarely used for the meta RL setting due to objective mismatch; methods for meta RL are rarely used for general POMDPs due to the stationarity assumption in meta RL.
APB
Oheeca-7 Creatabyy Hospen?
Hosperry
Walkaed
Willery
Figure Implementation Matters for Recurrent Model-Free RL. This paper identifies critical design decisions for recurrent model-free RL that outperforms only prior implementations (e.g. PPO-GRU and A2C-GRU from Kostrikov (2018)), hut also purpose- designed methods (c.g. VRM from Han et al. (2020). We also show Markovian policies as lower bounds for reference. The y-axis is normalized return given the return oracle policy (Raffin et al., 2021).
reward function that novel dur-
Code: httpa: / //r wee. wow..ooolle...oooooooo www.e/e..oo.ooooommooomoooo dars// 3m..1

---

Under review as a conference paper at ICLR 2022
Nonetheless, many prior works have used simple baseline that is applicable to all POMDPs: model-free RL equipped with a recurrent policy and (sometimes) value function (Duan et al., 2016; Fakoor al., 2020; Igl et al., 2018; Packer et al., 2018; Rakelly et al., 2019; Wang et al., 2017; Yu et al., 2019). We will refer to this approach as recurrent model-free RI. This baseline is simulta- neously simple (requiring changing only few lines of code from model-free RI. algorithm) and general. However, prior work has consistently found that recurrent model-free RL performs poorly across wide range of problem settings, including meta RL (Rakelly et al., 2019; Zintgraf et al., 2020), general POMDPs (Han et al., 2020; Igl et al., 2018), robust RL (Zhang et al., 2021), and generalization in RL (Packer et al.. 2018). One common explanation is that specialized algorithms that are tailored to specific types of POMDPs are very likely to outperform recurrent model-free RL because they (implicitly) encode inductive biases for solving these specific tasks. For example, algorithms for meta RI. may leverage the assumption that the underlying dynamics (while unknown) are fixed, and the underlying goals are fixed within one episode (Rakelly et al., 2019; Zintgraf et al., 2020); algorithms for robust RL may assume that the dynamics parameters are known (Rajeswaran et al., 201 7a) and dynamics is Lipschitz continuous (Jiang et al., 2021).
This paper challenges this explanation. We argue that, contrary to popular belief, recurrent model- free RL is competitive with recent state-of-theear algorithms across a range of different POMDP settings. Similar to prior work in Markovian on-policy RL methods (Andrychowicz et al., 2021; Engstrom et al., 2020), our experiments show that implementation in recurrent model-free RI. mat- ters. Fig. shows a typical scenario in PyBullet occlusion environments (Coumans & Bai. 2016) to support this argument. Through extensive experiments, we show that the careful design and imple- mentation of recurrent model-free RL is critical to its performance. Design decisions, such as the actor-critic architecture, conditioning on previous actions and/or rewards, the underlying model-free RL algorithms, and context length in RNNs, are especially crucial.
The main contribution of this paper is performant implementation of ceuurrent-mode free RL. We demonstrate that simple yet important design decisions, such as the underlying RI. algorithm and the context length, yield recurrent model-free RI. algorithm that performs on par with prior spe- cialized POMDP algorithms on the environments those algorithms were designed to solve. Ablation experiments identify the importance of these design decisions. We also open-sourced our code that is easy to use and memory-effiiient
2 BACKGROUND
where S is the set of states, A is the set of actions, T : S x A x S + [0 is the transition function (dynamics). To : S + [0, 1] is the initial state distribution, R : S x A x S + R is the reward function, H E N is the time horizon, and 7 E [0,1) is the discount factor. Solving an MDP requires learning a memoryless policy : S
MDP. A Markov decision process (MDP) (Bellman. 1957) is tuple (S, To,
x A - [0, 1) that maximizes the expected discounted
return: πt = arg max, Eeg
7're1 80 For any MDP, there exists an optimal policy that is both memoryless and deterministic (Puterman, 2014). MaxEnt RL algorithms (Ziebart, 2010), such as SAC (Haarnoja et al., 2018a), add an entropy bonus to the RL objective.
POMDP. A partially observable Markov decision process (POMDP) (Aström, 1965) is a tuple (S. A. O, Tu, O, Ou, R, H)) where the underlying process is MDP AS To, R, H, 7)) Let O be the set of observations and let S x A x O - [0, 1] observable trajectory up time-step / be To. (oo, an 01 rl,..., at- 1, or, r2). the memory-based policy in the most general form is defined as π7 (aa Th=t), conditioning on the whole history. At the first time step 0, an initial state 86 To(-) and initial observation og Oo( 80) are sampled. At any time-step E [O.... ......, H 1}. the policy emits the action a E A to the system, the system updates the state following the dynamics, st+1 T(- st, az). then the next observation is sampled 0041 O(- st41,4;) 4z) and the reward is computed as rt+1 = R(st, at 8(41).
be the emission function. Let the
We refer to the part of the state St at current time-step t that can be directly unveiled from current observation or as the observable state si. and the rest part of the state
as the hidden state sh. We call the hidden state of stationary if it does not change within an episode. In this scenario, the policy objective can be rewritten as n² for the average-case POMDP objec-
arg max, Egrnih [T, Ett-1 7/r411 [b]]
2

---

Under review as a conference paper at ICLR 2022
Table I: The summary of selected POMDP subareas. For each subarea, we list the information the hid- den state s including its appearance in dynamics and reward function and its stationarity during one trajectory. We also list the policy input space that are connected with the hidden states, where o, a, I, and d refer to the sequence of observations, actions, rewards, and done signals, respectively. Finally, we list the RL objective in terms of average-case or worst-case, and whether there is a domain shift between training and testing eavi- ronments. We append the check (V) or cross mark (X) with * if applies to some but not all the work in that subarea. The notation in this table will be covered in Sec. 2.
Subarea
8 in dynam- ics?
s" in re-
Is 8 sta- tionary?
Policy in- put space
RL objec-
Domain shift?
ward?
tive
"Standard" POMDP
x oar
Avg
x
Meta RL
x*
oard
Avg
x
Robust RL
/
x
/*
oa
Worst
x
Generalization
/
x
oa
Avg
/
in RL
tive, '² arg max, Essppp))))( F,
2i=0
Mr41
sh)
for the worst-case
POMDP objective.
3 RELATED WORK
In this section, we discuss several subareas of RL that both explicitly and implicitly solve POMDPs, as well as algorithms proposed for these specialized settings. Table summarizes these subareas.
RL for "Standard" POMDPs. We use the term "standard" to refer to prior work that explicitly labels the problems studied as POMDPs. Common tasks include scenarios where the states are par- tially occluded (Heess et al., 2015), different states correspond to the same observation (perceptual aliasing (Whitehead & Ballard, 1990)), random frames are dropped (Hausknecht & Stone, 2015), observations use egocentric images (Zhu et al., 2017). or the observations are perturbed with ran- dom noise (Meng et al., 2021). These POMDPs often have hidden states that are non-stationary and affect both the rewards and the dynamics. POMDPs are hard to solve (Littman, 1996; Papadimitriou & Tsitsiklis, 1987) because of the curse of dimensionality: the size of the history grows linearly with the horizon length. Many prior POMDP algorithms (Cassandra et al., 1994; Kaelbling al., 1998) attempt to infer the state from the past sequence of observations, and then apply standard RL techniques that inferred state. However, the exact inference requires the knowledge of the dy- namics, emission, and reward functions, and is intractable in all except the most simple settings. A common strategy for solving these general POMDPs is use recurrent policies, which take the en- tire history of past observations as inputs (Bakker, 2001; Schmidhuber, 1991; Wierstra et al., 2007). This strategy is very simple and general, and can be applied to arbitrary tasks without knowledge of the task structure (e.g., whether the hidden states change within an episode) across long time horizons (Duan et al., 2016). These recurrent strategies can be further subdivided into model-free methods (Hausknecht & Stone, 2015; Heess et al., 2015; Meng et al., 2021). where the single objec- tive is to maximize the return, and model-basee methods (Freeman et al., 2019; Han et al., 2020; Igl et al., 2018; Watter al., 2015) that have explicit objectives modeling the belief states and use them as the inputs of memoryless policies. The recurrent model-free RL that we focus on belongs to the class of model-free off-policy memory- bease a algorithms.
Meta RL. Meta RL, also called "learning to learn" (Schmidhuber, 1987; Thrun & Pratt, 2012), focuses on POMDPs where some parameters in the rewards or (less commonly) dynamics are varied from episode to episode, but remain fixed within a single episode, which represent different tasks with different values (Humplik et al., 2019). The meta RL setting is almost the same as multi- task RL (Wilson et al., 2007; Yu al., 2019), but differs in that multi-task RL can observe the task parameters, making it an MDP instead of a POMDP. Algorithms for meta RL can be roughly categorized based on how the adaptation step performed. Gradient-based algorithms (Fakoor et al., 2020; Finn et al., 2017: Hochreiter et al., 2001) run few gradient steps on the pre-trained models to adapt. Memory or context- based algorithms use RNNs to implicitly adapt, which can be further subdivided into implicit and explicit task inference methods. Implicit task inference methods (Duan et al., 2016; Wang et al., 2017) use RL objective only to learn recurrent policies. Explicit task inference methods (Rakelly et al., 2019; Zintgraf et al., 2020) train extra inference model to explicitly estimate task embeddings (i.e., a representation of the unobserved parameters) by variational inference. Task embeddings are then used as additional inputs to memoryless policies.
3

---

Under review as a conference paper at ICLR 2022
Robust RL. The goal of robust RL is to find policy that maximizes returns in the worst-case environments. Early work in the control and operations research community (Khalil et al.. 1996; Nilim & Ghaoui, 2005) and RL community (Bagnell et al., 2001; Morimoto & Doya, 2005) focused on linear or finite systems. Prior work designs deep RI. algorithms that are robust against a variety of adversarial attacks, including attacks on the dynamics (Jiang et al., 2021; Rajeswaran et al., 201 7a), observations (Huang et al.. 2017; Pattanaik et al.. 2018: Zhang et al., 2021), and actions (Gleave al., 2020; Pinto et al., 2017; Tessler et al., 2019). Treating the robust RL problem as POMDP, rather than an MDP (as done in most prior work), unlocks a key capability for RL agents, because agents can use their memory to identify the hidden states of the current adversarial environment, although previous work (Jiang et al., 2021; Rajeswaran et al., 2017a) only train Markovian policies on POMDPs. While some work find memory-based policies are more robust to the adversarial attacks than Markovian policies (Russo & Proutière, 2021; Zhang et al., 2021), they train these baselines in a single MDP without adversarics, which differs from our training setting where the recurrent model-free RL can have access to set of MDPs.
Generalization in RL. The goal of generalization in RL is to make RL algorithms perform well in test domains that are unseen during training, which emphasizes the average case on the novel test domains instead of the worse case in the possibly seen test domains as in robust RL. Prior work have studied generalization to initial states in the same MDP (Rajeswaran et al., 2017b; White- son et al., 2011; 1; Zhang et al., 201 8b), random disturbance in dynamics (Rajeswaran al., 2017h), states (Stulp et al., 2011), observations (Song et al., 2020; Zhang et al., 2018a), and actions (Srouji et al., 2018), and different modes in procedurally generated games (Cobbe et al., 2019; Farebrother et al., 2018; Justesen et al., 2018). Among them, Packer et al. (2018) provides a benchmark on both in-distribution (ID) and out-of-distribution (OOD) generalization to different dynamics parameters, and Zhao et al. (2019) extends the benchmark by introducing random noise in states, observations, and actions. Algorithms for improving generalization in RL can be roughly divided into classic regularization methods such as weight decay, dropout, batch normalization, and entropy regulariza- tion (Cobbe et al., 2020; Farebrother et al., 2018; Igl et al., 2019), model architectures (Raileanu & Fergus, 2021: Srouji et al.. 2018), data augmentation through randomization (Lee et al., 2020; Tobin et al., 2017), Although introducing observational noise and the change in dynamics parame- ters will transform MDPs to POMDPs, few work study memory- recurrent RL with mixed results. Same algorithm RL2 (Duan et al., 2016) was found to perform badly in Packer et al. (2018) but relatively well in Yu al. (2019).
policies such as model-free
4 DESIGN CONSIDERATIONS FOR RECURRENT MODEL-FREE RL
Implementing a recurrent model-free RL algorithm requires making a number of design decisions. This section describes the decisions that we found most important to make recurrent model-free RL competitive with more complex, recent algorithms. We will focus on continuous control problems with state-based inputs (i.c., not image-based inputs). Importantly, we assume that the policy can observe the reward and done signals (the end of one episode during one trial (Duan et al., 2016)) from the environment during evaluation. This assumption is common in prior work (Han et al., 2020; Zintgraf et al., 2020), but many recurrent model-free implementations do not provide the agent with information. In the following paragraphs, we will describe the important decision factors in recurrent model-free -feee RL. Table 2 summarizes how prior work and our method makes these design decisions when implementing recurrent model-free RL.
The first important design decision is whether the recurrent policy (actor) and the recurrent Q-value function (critic) use shared RNN encoder (and embedders) or use separate ones. In the experiment section (Sec. 5.2) we will show that a shared encoder would cause large gradient norm in the (off-policy) recurrent actor-critic and thus hinder learning, while separate encoders can greatly mitigate this issue and learn efficiently. This echoes prior work (Fakoor et al., 2020; Meng al., 2021; Sun et al., 2021; Wang al., 2020) that also use separate encoders in their (off-policy) recurrent actor-critic. To avoid running an inordinate number of experiments, we will use the separate architecture in the rest of the paper.
Recurrent Off-Policy Actor-Critic Architecture.
Policy Input Space. The next consideration is the input space of the model-free policy. The maximal input space of policy to emit an action a, at time t, should be the history of all quantities that the policy has observed, namely the past observations Op the past actions d0 the past
4

---

Under review as a conference paper at ICLR 2022
How the prior work and our method implement the recurrent model-free RL as their own method or baseline. We can see that none of the prior work share the same set of decision variables, some of which have bad choices that may lead to the poor performance reported in the prior work. Our method covers a range of choices in these decision factors and finds the combinations in the last rows that lead to the best performance in terms of the average performance across the experimented environments in each subarea.
Table 2:
Algorithm
Domain
Arch
Encoder
Inputs
Len
RL
Duan et al. (2016)
Meta RL
separate
GRU
oard
1000
TRPO, PPO
Wang et al. (2017)
Meta RL
shared
LSTM oart
5-150
A2C
Baseline from Rakelly et al. (2019)
Meta RL
separate
GRU oard
100
PPO
Baseline from Zintgraf et al. (2020)
Meta RL
separate
GRU
oard
Max
A2C, PPO
Baseline from Fakoor et al. (2020)
Meta RL
separate
GRU
oar
10-25
TD3
Baseline from Yu et al. (2019)
Mcta RL
separate
GRU
oard
500
PPO
Kostrikov (2018)
POMDP
shared
GRU
5-2048
PPO, A2C
Wang al. (2020)
POMDP
separate
LSTM
Oa 150
TD3, SAC
Meng al. (2021)
POMDP
separate
LSTM
Oa
1-5
TD3
Yang& & Nguyen (2021)
POMDP
separate
both oa
Max
SAC
Baseline from Igl ct al. (2018)
POMDP
shared
GRU
oa
25
A2C
Baseline from Han et al. (2020)
POMDP
shared
LSTM
64
SAC
Baseline from Zhang al. (2021)
Robust RL
separate
LSTM
100
PPO
Baselinel from Packer et al. (2018)
Generalization
shared
LSTM
128-512
PPO, A2C
Baseline2 from Packer et al. (2018)
Generalization
separate
LSTM
oard
128-512
PPO, A2C
Our method
Meta RL
separate
LSTM
oard
64
TD3
Our method
POMDP
separate
GRU
oa
64
TD3
Our method
Robust RI.
separate
LSTM
64
TD3
Our method
Generalization
separate
LSTM
64
TD3
rewards ro:s, and the past done signals da which was already employed in the early work (Duan et al., 2016). Generally, the input space of optimal policy should only depend on the quantities that have connections with hidden states (defined in Sec. 2) (Izadi Precup, 2005; Poupart & Boutilier, 2002). We show the policy input spaces that are connected with the hidden states for the discussed subareas in the "Inputs" column of Table 1. While prior work often only conditions the recurrent RL baseline on previous observations (and actions) (Han et al., 2020; Igl et al., 2018; Kostrikov, 2018; Meng et al., 2021; Wang et al., 2020; Yang & Nguyen, 2021), our experiments in Sec. 5.2 find that additionally conditioning on other previous information, such as previous rewards, can increase reward by up to 30%.
Model-free RL Algorithms. Recurrent model-free RL can be understood applying an off-the- shelf model-free RL algorithm with an actor and a Q function parametrized to take sequences of inputs. As such, the choice of the underlying model-free RL algorithm is paramount. Most prior work on continuous control POMDP problems used on-policy algorithms, such as A2C (Mnih et al., 2016), TRPO (Schulman et al., 2015) PPO (Schulman et al., 2017). While off-policy algorithms such as TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 201 8a;b) greatly improve the per- formance in continuous control MDP problems in terms of sample efficiency and asymptotic per- formance, these methods are rarely used in recurrent model-free RL baselines (Rakelly et al., 2019; Zhang et al., 2020; Zintgraf et al., 2020). In the experiment section (Sec. 5.1), we will show that using these off-policy algorithms for recurrent model-free RL provides results that are better than using on- polooic algorithms and are comparable their specialized methods in POMDP. This echoes the finding that model-free off-policy TD3-Context (Fakoor et al., 2020) can be better than the spe- cialized method PEARL (Rakelly et al., 2019) in meta RL.
RNN Variants and Context Length. RNN training is known to be unstable, especially with long sequences input (Bengio al., 1994). The RNN variants like LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Chung et al., 2014) mitigate the training issues, but still may fail to learn long- tergteerm dependencies (Trinh al., 20. In POMDP problems, these dependencies reflect the memory that an agent must have to solve a task. For example, a POMDP that hides velocities from observations theoretically requires a short memory length to infer velocities through consecutive positions (Meng et al., 2021). Prior work in POMDPs choose variety set of context lengths for RNNs from I to 2048 (see the "Len" column of Table 2), and we select three representatives of short (5), medium (64), and long length (larger than 100) in the experiments (Sec. 5) for comparison. We also try both LSTM and GRU as RNN variants to compare their performance. We find that the optimal context length and RNN variant are task-speeific (see Sec. 5.2).
5

---

Under review as a conference paper at ICLR 2022
5 EXPERIMENTS
Our experiments aim to answer two questions. First, how does well-tuned implementation of recurrent model-free RL compare to specialized POMDP methods, such as drepsedd meta RL and robust RL algorithms? To give these prior methods the strongest possible footing, will compare prior methods on the specific problem types for which they were developed (i.e., meta RI. algorithms were tested meta RI tasks). Our second question studies which design decisions essential for recurrent model-free RL. We put the environment details in Appendix D.
Code Implementation. We release modular and rent (off-polln) coff-) model-free RL: url. Our implementation is efficient in terms of computer memory compared to previous off-policy RL methods for POMDPs (200x less RAM than Han et al. (2020) and 9x less GPU memory than Dorfman et al. (2020)). Please see the appendix A for details, includ- ing an explanation of why our implementation is more memory-efficiet than prior work.
implementation of reeur-
5.1
RECURRENT MODEL-FREE RL COMPARABLE WITH PRIOR SPECIALIZED METHODS
While prior work has studied a range of different POMDP settings (e.g., meta RL, occluded observa- tions), recurrent model-free RL is ubiquitous baseline (Han al., 2020; Humplik et al., 2019; Igl et al., 2018; Rakelly et al., 2019; Zintgraf et al., 2020). However, prior work consistently report that this baseline is reported to be unperformed to more specialized methods. This section casts doubt on that claim, showing that a well-tuned implementation of recurrent model-free RL can perform at least as well as more specialized methods.
We study four subareas of POMDPs: the "standard" POMDP, meta RL, robust RI, and generaliza- tion in RL. We tune a wide range of decision factors shown in Sec. 4 in our implemented recurrent model-free RL. Appendix A.3 shows the details of the tuning options. For each subarea, we show the performance of a single variant that works best across the environments in that subarea, com- pared with the prior specialized methods in this subsection. In other words, the following plots of each subarea report the same model-free recurrent RL algorithm with the same hyperparame- ters. The exact configurations of each subarea can be found in the last four rows of Table 2. Under this restricted setting, we find that our implementation can actually outperform prior (specialized) methods by a wide margin across the four subareas.
For each plot of learning curves, we show three approaches as reference. First, an Oracle policy has access to the POMDP hidden states, turning the POMDP into an MDP; this policy should therefore be treated as an upper bound on the performance that any POMDP method should receive. Sec- ond, as lower bound, we use a Markovian policy to solve the POMDP. Both Oracle policy and Markovian policy are trained with the same hyperparameters our recurrent model-free RL imple- mentation. Third, we add Random policy, which represents a trivial lower bound. We show the full learning curves in Appendix E. due to the space limit. See Appendix B for details about the implementation of these comparisons.
"Standard" POMDP. Our first experiments look at the "standard" POMDPs that typically oc- clude some part of states in the environment. We will compare against VRM (Han et al., 2020), recent state-of-the-art model-based POMDP algorithm. We directly apply the environment design of VRM paper that occludes either positions&angles or velocities of the simulated robot in PyBul- let (Coumans Bai, 2016). There are environments (Hopper, Ant, Walker, Cheetah)-(P.V) where "-P" stands for observing positions&angles only, and "-V" stands for observing velocities only. Fig. and Fig. 18 in appendix show that the best single variant of our model-free recurrent RL implementation outperform VRM in 6 out of 8 environments, especially in Schbetah,Hooper--PEP (over 80% of the Oracles). Our results suggest that, while the variational dynamics model used by VRM may be useful for some tasks, simple recurrent model-free RL baseline can outperform VRM properly tuned. While we are primarily interested in sample complexity, but not compute, worth noting that our recurrent model-free RL implementation is substantially more efficient than the open-source VRM implementation, training 5x faster and requiring at most 200 less RAM usage (see Appendix A).
Meta RL. We next compare the recurrent model-free RL the meta RL setting, where some indicator of the task is unobserved. We compare our implementation of recurrent model-free RL to specialized state-of-the-art method, VariBAD (Zintgraf et al., 2020) that explicitly learns the

---

Under review as a conference paper at ICLR 2022
Ours: Setreenttttt---eeeeettt
Ours: recurnent model free RL
-200
Oracle/
Oracle
Markovian
Markpdian
Rancoo
return
Rarddm
20
-600
-000
00002000100040005500000000000000
sre_steps
steps
1e6
Wind
Cheetah-Vel
Figure 2: Learning curves on meta RL environments. The single best variant of our implementation on recurrent model-free free RL can surpass the specialized meta RL method off-policy VariBAD (Dorfman et al., 2020), and match the performance of an "Oracle" policy that gets to observe the hidden state.
task embeddings by variational model-based objectives. As suggested by Dorfman al. (2020), we modified VariBAD to use SAC instead of PPO. This change also allows us for fair comparison with our implementation of recurrent model-free RI., which uses SAC and TD33. We adopt the three environments used in Dorfman et al. (2020) for experiments, including Semi-Circle and Cheetah- Vel, and we also adapt Wind to make harder to solve. Figure shows that our best single variant outperforms VariBAD and even reaches Oracles in the two meta RL environments, Cheetah-Vo and Wind, leaving Semi-Cireh in the appendix. Prior work (Rakelly et al., 2019; Zintgraf et al., 2020) show that disentangling task inference and control can stabilize training. However, experiments suggest that joint training of task inference and control could also have comparable performance well implemented. Additionally, because recurrent model-free RL is trained end-to-end, without ing pre- tre-trained task representations saved in the replay buffer like the off-policy VariBAD (Dorfman al., 2020), our implementation does not have con-stationarity issue task representations.
2000
2500
recurrent model-free
Ours: recurrent model
MRPO
MRPC
2000
Orpcle
retumn W90rSt
1500
Oraple
1500
Random
1000
Bandomm
500
500
0.8
1.0
1.2
1.4
0.4
0.5
0.8
1.0
3.2
env steps
env steps
Figure 3: Learning curves on one robust RL environment, Cheetah-Robuse We show the average returns (left figure) and worst returns (right figure) of cach method. The single best variant of our implementation on recurrent model-free RL can greatly outperform the specialized robust RL method MRPO (Jiang et al., 2021), and surpass the Oracle that are trained with same simulation and gradient steps.
Robust RL. Thirdly, we focus on the robust RL that aims to maximize the worst returns over the tasks. We choose the recent specialized algorithm MRPO (Jiang et al., 2021) the compared method, and adopt their used environments based on SunBlaze benchmark (Packer et al., 2018). These environments have hidden states that are fixed during one episode, including the density and the friction coefficients of the simulated robots, namely (Cheetah, Hopper, Walker)-Robus, Fig. shows both the average return and worst return of our single best variant and MRPO on the three environments, where the worst return is measured by the average return in the worst 10% testing tasks following the practice in Jiang et al. (2021). The results are quite surprising: although our implementation, using average-case RL objective and without access to the hidden states, is not expected to surpass MRPO and Oracle with access to hidden states worst return, we found that best variant vastly outperforms the specialized MRPO and Oracle in both average return and worst return, with over 80% fewer simulation steps. Our implementation benefits from its memory and off- off-policy algorithms, while MRPO might suffer from its Markovian on-policy algorithm and a bit ideal Lipschitz assumption in dynamics. Nevertheless, our implementation is around 17.5x slower than MRPO given the same simulation steps (see Appendix A), so we only run with 3M steps with a limited time budget.
Generalization in RL. Finally, we focus on the SunBlaze benchmark from Packer et al. (2018) for investigating generalization in RL, including (Hopper, specialized method the tables of final performance in Packer et al. (2018), Markovian on on-policy robust RL method EPOpt-PPO-FF (Rajeswaran al., 2017a). Fig. 4 show the interpolation and
We pick the best

---

Under review as a conference paper at ICLR 2022
0.8
Curs: recurtent modill-rae Bi
Ours recumennt modekfree Rt
A2C-RC
A2C-RC
Interpolato
0.6
Oracle
EPOPt-PPO-FF
Extrapoletion
0.2
Gracie
EEOPL-PROSEE
Markgwian
0.4
Random
Raydom
0.1
0.5
2.5
www_steps
1e6
stv_steps
1e5
Figure 4: Learning curves on generalization in one RL environment, Geneer--en, We show the interpolation success rates (left figures) and extrapolation success rates (right figures) of cach method. The single best variant of our implementation on recurrent model-free RL can par with the specialized method EPOpt-PPO-FF (Rajeswaran et al., 2017a) in interpolation and outperform in extrapolation. The data of EPOPt-PPO-FF and A2C-RC CCC (a recurrent model-free on on-oolicy RL method) are copied from the Table 7 & 8 in Packer et al. (2018).
extrapolation success rates in one environment, where in the interpolation the testing tasks have same distribution of hidden states as that of the training tasks, while the extrapolation the testing distribution is disjoint from that of training. We see that model-free method is par PPO-FF requires access to
with the EPOpt-PPO-EF in the interpolation benchmark, while the dynamics parameters but ours does not. In the extrapolation benchmark, method greatly outperforms the previous method, although our objective does not consider extrapolation.
see that with careful tuning recurrent model-free RL., it can least perform as well as the specialized or more complicated methods, in various kinds of POMDPs.
Overall,
5.2 WHAT MATTERS IN RECURRENT MODEL-FREE RL. ALGORITHMS?
In the previous subsection, showed that recurrent model-free RL perform par with the specialized (state-of-the-are methods, then a natural question comes: Why our implementation of recurrent model-fres RL outperforms the implementation used in prior work?
Our analysis will focus on ablating the five important design decisions introduced in Sec. 4: the actor-critic architecture (Arch), the policy imput space (Inputs), the underlying model-free RL algorithm (RL), the RNN encoder (En (Encoder) and the RNN context length (Len). See Table. 2 for summary of how prior work made these design decisions. Due the space limit, show the ablation results in some but not all the environments to compare the performance between the best single variant and the other variant that only differs in one decision factor. We also provide "single factor analysis" plots for each decision factor by averaging the performance over the other factors in Appendix E.2.
Recurrent Off-Policy Actor-Critic Ar- chitecture. First, ments with both shared and separate ar- chitectures on toy POMDP environ- ments. Fig. 5 show the results in one of them (see Appendix E.3 for the other). We can see that the shared architecture failed learn, compared to the separate archi- tecture. The large RNN gradient norm in the shared architecture suggests that the actor and critic losses may cause conflicts in gradient update. Our results echo prior work (Fakoor et al., 2020; Meng al., 2021; Sun et al., 2021) that only consider separate RNN encoders that can achieve high asymp- totic rewards, and also echo that (Han et al., 2020) shows poor results in the shared architecture of SAC-LSTM.
M44A
some experi-
30
mum
son- seporata
16
Jaace
0.00
0.5
0.50
0.75
1.00
1.25
1.55
0.00
0.2
0.50
0.75
3.00
3.25
3.50
Figure 5: Comparison between shared and separate current actor-critic architecture with all the other hyper- parameters same, on Semi-Circle, toy meta RL environ- ment. We show the performance metric (left) and also gra- dient norm of the RNN encoder(s) (right, in log-scale). For the separate architecture, critic and aactor refer to separate RNN in critic and actor networks, respectively.
Policy Input Space. The 1st row of Table 3 shows the effect of policy input space in POMDP environment Walker-P. The reward signals could help reveal the missing information of the velocity of the robot base. which is occluded in Walker-P, Similarly, the single factor analysis policy input in Fig. 12 shows that oar is among the best in ".P" environments. Therefore, it is reasonable that adding previous rewards into policy inputs can increase the performance.
8

---

Under review as a conference paper at ICLR 2022
Table 3: Ablation results in our implementation of recurrent model-free RL. In this table, we show how a single change in one decision factor from the variant that is best on average in that subarea, could significantly increase the performance. The first column shows how change the single decision factor, and the last column shows the performance comparison between the best variant that subarea (left) and ablated one (right). For robust RL and generalization in RL, we show the performance metric in worst returns and extrapolation success rates, respectively.
Change in one decision factor
Subarea
Env
Performance comparison
snputs: oa + oar
"Standard"' POMDP "Standard" POMDP
Walker-P
981.6 + 1345.0 (1.3x)
RL: TD3 -> SAC
Ant-P
310.7 - 2123.5 (6.8x)
Kncoder: LSTM + GRU
Robust RI.
Walker-Robust
765.9 - 931.3 (1.2x)
Len: 64 + 400
Meta RL
Cheetah- Vel
-85.2 + -74.6 (+14%)
Len: 64 -> 5
Generalization
Hopper-
0.292 - 0.415 (1.4x)
Len: 64 -> 5
"Standard" POMDP
Walker-V
121.4 - 264.3 (2.2x)
Model-free RL Algorithms. Table 2 shows that TD3 dominates all the best variants in each sub- area, which may be due to dense reward setting in most environments. However, the 2nd row of Table shows the effect of RL algorithm in POMDP environment Ant-P. SAC is significantly better than TD3 (increase by 6.8x, surpassing the PPO-GRU (Kostrikov, 2018) in Fig. 1), possi- bly due strategic exploration where the action noise conditions the history instead of being independent. This is prominent mainly in Ant-P as it might be much harder than the others.
RNN Variants and Context Length. Generally, there no significant difference between LSTM and GRU (see the single factor analysis in Appendix E.2), which is understandable as both are designed for general purpose. Howover, the 3rd row of Table 3 shows the effect of RNN encoder in robust RL environment. We can see replacing LSTM with GRU can increase the worst-case metric in Valker-Robust, For the context length in RNNs, a medium length (64) dominates in trade-off between
all the best variants in each subarea (see Table 2). which could be viewed memory capacity and computation costs. However, the remaining rows of Table 3 show the mixed effects of context length in RNNs. Both increasing and decreasing the context length performance in different environments. Specifically, decreasing the length from 64 to 5 makes our method surpass VRM in Walker-V (increase by 2.2x). ). This might explain why the prior methods adopt a wide range of context lengths from I to 2048 (see Table 2). Therefore, the choice of context length is problem- specific and can require tuning.
boost the
Summary. We summarize the main findings of our experiments:
1. Using separate weights for the recurrent actor and recurrent critic boosts performance, likely because it avoids gradient explosion (Fig. 5 and Fig. 17 in Appendix).
2. Using state-of-the-an off oof-po RL algorithms as the backbone in recurrent model-free RI. can improve asymptotic performance (Fig. I and Figures in Appendix E.
3. The context length for the recurrent actor and critic has a large influence on task performance, but the optimal length is task-specific. Reasonable values are 5 to 500, and 64 is good start (Rows 4-6 in Table and Figures in Appendix E.2).
4. is important that the inputs to the recurrent actor and critic, such as past observations and past returns, contain enough information to infer the POMDP hidden states (Row 1 in Table 3 and Figures in Appendix E.2).
These findings may provide useful initialization for researchers studying recurrent model-free RL.
CONCLUSION
In this paper, we show that carefuly-dessinnel implementation of recurrent model-free RL can perform well across a range of POMDP domains, often on par with (if not significantly better than) prior methods that are specifically designed for specific types of POMDPs. Our ablation experiments demonstrate the importance of key design decisions, such as the underlying RL algorithm and RNN context length. While the best choices for some decisions (such as using separate RNNs for the actor and the critic) are consistent across domains, the best choices for other decisions (such as RNN context length) are problem--ependent.
9

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We release our code at the url for reproducibility. In this code repository, we provide instructions on how to run our method and the compared methods in this paper all the environments involved in the experiment section. We provide the default configuration files for training and evaluating these algorithms that are adopted in our experiments. We also attach the numeric results of all the bar charts shown in the experiment section.
ETHICS STATEMENT
We do not believe that our work has direct ethical or societal implications.
REFERENCES
Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk. Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? A large-scale study. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Karl Johan Aström. Optimal control of markov processes with incomplete state information i. Journal of Mathematical Analysis and Applications, 10:174-205, 1965.
J Andrew Bagnell, Andrew Y Ng, and Jeff Schneider. Solving uncertain markov decision processes. 2001.
Bram Bakker. Reinforcement learning with long short-term memory. In Advances in Neural Information Pro- cessing Syxtems 14 [Neural Information Processing Syxtems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada], 2001.
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, 6(5):679-684, 1957.
Yoshua Bengio, Patrice Y. Simard, and Paolo Frasconi, Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Networks, 1994.
Anthony R. Cassandra, Leslic Pack Kaelbling, and Michael L. Littman. Acting optimally in partially observable stochastic domains. In Proceedings of the 12th National Conference Artificial Intelligence, Seartle, WA, USA, July 31- August 4, 1994, Volume 2, 1994.
Min Chen, Stefanos Nikolaidis, Harold Soh, David Hsu, and Siddhartha S. Srinivasa. Planning with trust for human-robor robot collaboration. Proceedings of the 2018 ACM/IEEE International Conference on Human- Robot Interaction, HRI 2018, Chicago, II, USA, March 05-08, 2018, 2018.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçchre, Dzmitry Bahdanau. Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encodor-decoder for statisti- cal machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, Special Interest Group of the ACL, 2014.
Junyoung Chung. Çaglar Gülçehre, KyungHyun Cho. and Yoshua Bengio. Empirical evaluation of gated current neural networks on sequence modeling, arXiv preprint arXiv:1412.3555, 2014.
Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization reinforcement learning. Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, 2019.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to bench- mark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Viriual Event, 2020.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta learning of exploration. arXiv preprint arXiv:2008.02590 2020.
Yan Duan, John Schulman, Xi Chen, Peter I. Bartlent, llya Sutskever, and Picter Abbeel. RI2: Fast reinforce- ment learning via slow reinforcement learning. arXiv preprint orXiv:1611 I. 1.02779, 2016.
10

---

Under review as a conference paper at ICLR 2022
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos. Larry Rudolph, and Alek- sander Madry. Implementation matters deep RL: A case study on PPO and TRPO. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander Smola. Meta-q-learning In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and regularizzation in DQN. arXiv preprint arXiv:/810.00122 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, 2017.
C. Daniel Freeman, David Ha, and Luke Metz. Learning to predict without looking ahead: World models without forward prediction. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, 2018
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell, Adversarial policies: Attacking deep reinforeement Icarnigg. In 8th International Conferencee on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Con- ference on Machine Learning, (CML.2018, Stockholmsmütssan, Stockholm, Sweden, July 10-15, 2018, 201 8a.
Tuomas Haarnoja. Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1el.:05983.0 2018b.
Dongqi Han, Kenji Doya, and Jun Tani. Variational recurrent models for solving partially observable control tasks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposia, Arlington, Virginia, USA, November 12-14, 2015, 2015.
Nicolas Heess, Jonathan J. Hunt, Timothy P. Lillicrap, and David Silver. Memory-based control with recurrent neural networks. arxiv preprint arXiv:1512.04455 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 78):1733- 1997.
Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. In Artificial Neural Networks ICANN 2001, International Conference Vienna, Austria, August 21-25, 2001 Proceedings, 2001.
Sandy H. Huang, Nicolas Papernot. Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017.
Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A. Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement leaming as task inference. arXiv preprint arXiv:1905.0644 2019.
Maximilian Igl, Luisa M. Zintgraf, Tuan Anh Lc, Frank Wood, and Shimon Whiteson. Deep variational rein- forcement learning for pomdps. Proceedings of the 35th International Conference on Machine Learning, JCML 2018, Stockholmsmässsan Stockholm, Sweden, July 10-15, 2018, 2018
Maximilian Igl. Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottle- neck. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019.

---

Under review as a conference paper at ICLR 2022
Masoumeh T. Izadi and Doina Precup. Using rewards for belief state updates in partially observable markov decision processes. Machine Learning: ECML 2005, 16th European Conference on Machine Learning. Porto, Portugal, October 3-7, 2005, Proceedings, 2005.
Yuankun Jiang. Chenglin Li, Wenrui Dai, Junni Zou, and Hongkai Xiong. Monotonic robust policy optimization with model discrepancy. Proceedings of the 38th International Conference on Machine Learning, ICML 2021. 18-24 July 2021, Virtual Event, 2021.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint 1806. 10729, 2018.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially ob- servable stochastic domains. Artif. Intell., 1998.
Islam SM Khalil, JC Doyle, and Glover. Robust and optimal control. prentice hall, new jersey, 1996.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. httpp://gi http://tttttthccccee com/ 2018.
ikostrikev weatrrikov/pytrrcchhhh a2c- appo-acktr-
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Michael Lederman Littman. Algorithms for sequential decision-makinng Brown University, 1996.
Xiao Ma, Peter Karkus, David Hsu, and Wee Sun Lee. Particle filter recurrent neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5101-5108, 2020a.
Xiao Ma, Peter Karkus, David Hsu, Wee Sun Lee, and Nan Ye. Discriminative particle filter reinforcement learning for complex partial observations. arXiv preprint arXiv:2002.98884 2020b.
Lingheng Meng, Rob Gorbet, and Dana Kulic. Memory- deep reinforcement learning for POMDP. arXiv preprint arXiv:2102. 12344, 2021.
Volodymyr Mnih, Adrià Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Pro- ceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, 2016.
Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335-35, 2005.
Arnab Nilim and Laurent Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Oper. Res., 2005.
Charles Packer, Katelyn Gao, Jemnj Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv: 12282, 2018
Christos H. Papadimitriou and John N. Tsitsiklis. The complexity of markov decision processes. Math. Oper. Res., 1987.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep inforcement learning with adversarial attacks. Proceedings of the 17th International Conference on Au- tonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, July 10-15, 15, 2018, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learn ing. Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW; Australia, 6-11 August 2017, 2017.
Pascal Poupart and Craig Boutilier. Value-directed compression pomdps. In Advances in Neural Informa- tion Processing Systems 15 (Neural Information Processing Systems, NIPS 2002, December 9- 9-14, 2002, Vancouver; British Columbia, Canada], 2002.
Martin L Puterman. Markov decision processex: discrete stochastic dynamic programming. John Wiley and Sons, 2014.
Antonin Raffin. RI baselines3 zoo. https: www.:ll.oooooo//oeoeeo.ee htw.-oa----ee----aaeaeeeeeeeeeee 2020.
Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In 5th
Annual Conference on Robot Learning, 2021.
12

---

Under review as a conference paper at ICLR 2022
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, 2021.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017a.
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham M. Kakade. Towards generalization and simplicity in continuous control. In Advances in Neural Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, 2017b.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In Proceedings of the 36th International Confer- ence on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, 2019.
Alessio Russo and Alexandre Proutière. Towards optimal attacks reinforcement learning policies. In 2021 American Control Conference, ACC 2021, New Orleans, LA, USA, May 25-28, 2021, 2021.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
Jürgen Schmidhuber. Reinforcement learning in markovian and non-markovian environments. In Advances in Neural Information Processing Systems 3, NIPS'3, pp. 500-506, 1991.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06346 2017.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 30, 2020, 2020.
Mario Srouji, Jian Zhang, and Ruslan Salakhutdinov. Structured control nets for deep reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssson Stockholm, Sweden, July 10-15, 2018, 2018.
Freek Stulp, Evangelos A. Theodorou, Jonas Buchli, and Stefan Schaal. Learning to grasp under uncertainty. In IEEE International Conference on Robotics and Automation, ICRA 201 Shanghai, China, 9-13 May 2011, 2011.
Hao Sun, Ziping Xu, Meng Fang, Zhenghao Peng, Jiadong Guo, Bo Dai, and Bolei Zhou. Safe exploration by solving carly terminated mdp. arXiv preprint arXiv:2107.04200 2021.
Chen Tessler, Yonathan Efroni, and Shic Mannor. Action robust reinforcement learning and applications in continuous control. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, 2019.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science and Business Media, 2012.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel. Domain ran- domization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RS International Conference on Intelligent Robots and Systems, IROS 2017, Vancouver, BC, Canada, Septem- ber 24-28, 2017, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoca: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, 2012.
Trieu Trinh, Andrew Dai, Thang Luong, and Quoc Le. Learning longer-term dependencies in mns with auxil- iary losses. In International Conference on Machine Learning, pp. 4965-4974, PMLR, 2018.
Haonan Wang, Ning Liu, Yiyun Zhang, Dawei Feng, Feng Huang, Dong Sheng Li, and Yiming Zhang. Deep reinforcement learning: survey. Frontiers Inf. Technol. Electron. Eng., 2020.
13

---

Under review as a conference paper at ICLR 2022
Jane Wang, Zeb Kurth-Nelson. Hubert Soyer. Joel Z. Leibo, Dhruva Tirumala, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matt M. Botvinick. Learning to reinforeement learn In Proceedings of the 39th Anmual Meeting of the Cognitive Science Society, CogSci 2017, London, UK, 16-29 July 2017, 2017.
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin A. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, 2015.
Steven D Whitehead and Dana H Ballard. Active perception and reinforcement learning. In Machine Learning Proceedings 1990, pp. 179-188. Elsevier, 1990.
Shimon Whiteson, Brian Tanner, Matthew E. Taylor, and Peter Stone. Protecting against evaluation overfitting in empirical reinforcement learning. In 2011 IEEE Symposium Adaptive Dynamic Programming And Reinforcement Learning, ADPRL 2011, Paris, France, April 12-14, 2011, 2011.
Daan Wierstra, Alexander Förster, Jan Peters, and Jürgen Schmidhuber. Solving deep memory pomdps with recurrent policy gradients. In Artificial Neural Newworks ICANN 2007, 17th International Conference, Porto, Portugal, September 9-13, 2007, Proceedings, Part I, 2007.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: hierarchi- cal bayesian approach. In Machine Learning, Proceedings of the Twenty-Fourth International Conference (ICMI. 2007), Corvallis, Oregon, USA, June 20-24, 2007, 2007.
Zhihan Yang and Hai Nguyen. Recurrent off off-policy baselines for memory- based continuous control. arXiv preprint arXiv:21 1o..1.0.2.222 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In 3rd Anmual Conference on Robot Learning, CoRL 2019, Osaka, Japan. October 30 November 1. 2019, Proceedings, 2019.
Amy Zhang, Nicolas Ballas, and Joelle Pineau, A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv: 0793777937 2018a.
Chiyuan Zhang, Oriol Vinyals, Rémi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:/804 06893, 2018b.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan I. Liu, Duane S. Boning, and Cho-Jui Hsich. Robust deep reinforcement learning against adversarial perturbations state observations. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Huan Zhang, Hongge Chen, Duane S. Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Chenyang Zhao, Olivier Sigaud, Freek Stulp. and Timothy M. Hospedales. Investigating generalisation in continuous deep reinforcement learning, arXiv preprint 1cXxv::902.0701 2019.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta. Li Fei-Fei, and Ali Farhadi. Target- driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, Singupore, Singapore, May 29 June 3, 2017, 2017.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shi- mon Whiteson. Varibad: A very good method for bayes-adaptive deep RL via meta-leamiigg In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
14

---

Under review as a conference paper at ICLR 2022
A CODE-LEVEL DETAILS
In this section, we first introduce the outline of code design, especially the replay buffer for se- quences, and then compare the system usage, including computing speed, RAM, and GPU memory with previous POMDP methods.
A.1
CODE DESIGN
Easy to use. Our code can be either used as an API to call the recurrent model-free RL class or framework tune the details in the class. The recurrent model-free RL class takes the hyper- parameters of RNN encoder type, shared or separate actor-critic architecture, and whether include previous observations, and/or actions, and/or rewards into the inputs, generate different instances. The details of the hyperparameter tuning set are shown in Sec. A.3.
Memory-efficiet replay buffer for sequences. Moreover, we design an efficient replay buffer for off- off-policy RL methods to cope with sequential inputs. Previous methods (Han et al., 2020; Yang & Nguyen, 2021) mainly use three-dimensional replay buffer to store sequential inputs, with the di- mensions of (num epi episodes, max epi sode length, observas rvatrvatin in dimens on), taking observation storage as example. This kind of implementation becomes memory-ineffiiien the actual episode length is far smaller than the max episode length (then there will be many padded zeros (Dorfman et al., 2020)). Instead, we manage to implement a two-dimensional replay buffer observation
of shape (num transitions
dimension, on) for observation storage, which also records the locations where each stored episode ends. In case of actual episodes that are shorter than provided sampled sequence length, the buffer also generates on-the-fly masks to indicate if the corresponding transitions are valid, so that we do not need to save sero-padded observations in the buffer. This enables the policy to receive a batch of previous experiences in a tensor-like data struc- ture when sampling from the replay buffer. To sum up, replay buffer can support varying-leng sequence inputs and subsequence sampling without zero padding.
Flexible training speed. Finally, our code supports flexible training speed by controlling the ratio of the numbers of gradient updates in RL w.r.t. the environment rollout steps. The training speed is approximately proportional to the ratio if the simulator speed much faster than the policy gradient update. Typically, the ratio is less than or equal to 1.0 to enjoy higher training speed.
A.2 SYSTEM USAGE
Table 4 shows the typical system usage of our method and the compared specialized methods on different environments. The time cost for our method and VariBAD depends on how many processes in parallel are run on a single GPU our method is run with 8 processes on single GPU while VariBAD run with one process due to large GPU memory usage. From the results we can see that our method is memory-efficient in both RAM and GPU, and has an acceptable training speed with default hyperparameters. The computer system we used during the experiments includes a GeForce RTX 2080 Ti Graphic Card (with 1 1GB memory) and Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz (with 250GB RAM and 80 cores).
Table Comparison between our method and specialized methods in system usage. The time costs are evaluated within IM environment steps. Both VRM and MRPO are run on CPUs and MRPO does not have replay buffer (shown in N/A). VariBAD requires the assumption of fixed episode length for the RAM cost.
Method
Environment
Time cost
RAM
GPU memory
Ours (GRU)
Hopper-V V
22.5 h
O(1)
1.2 GB
VRM (Han et al., 2020)
Hopper-V V
102 h
0(200)
N/A
Ours
Semi-Circle
12 h
O(1)
1 GB
VariBAD (Dorfman al., 2020)
Semi-Circle
2.3 h
O(1)*
9.5 GB
Ours
Cheetah-Robust
7 h
O(1)
1.1 GB
MRPO (Jiang et al., 2021)
Cheetah-Robust
0.4 h
N/A
N/A
15

---

Under review as a conference paper at ICLR 2022
A.3 OUR HYPERPARAMETER TUNING SET
Our proposed method has the following decision factors (introduced in Sec. 4) to tune in the exper- iments with the following options (the names in brackets abbreviated ones):
Actor-Critic architecture (Areh): share the encoder weights between the recurrent actor and recurrent critic or not, namely shared and separate.
Model-free RL algorithms (RL): td3 (Fujimoto et al., 2018) or sac (Haarnoja et al., 2018a)
Encoder architecture (Encoder): 1stm (Hochreiter & Schmidhuber, 1997) or gru (Cho et al., 2014)
Policy input space (Inputs): o, oa, or, oar, oard (the notation is introduced in Sec. 4; depending on the POMDPs, see "Policy input" in Table 5)
Context length (Len): short (5), medium (64), long (larger than 100, depending the POMDPs).
For each instance, we label it with the names of all the hyperparameten'' it used in lowercase as notation. For example, td3-1stm- 54-or-ssppa ser-or--parree rate in Fig. 5 refers the instance that uses the separate actor-critic architecture, TD3 RL algorithm, LSTM encoder, the policy input space of previous observations and reward sequences, and RNN context length 64.
B
TRAINING DETAILS
Observ:
Observ. Encoder A
at
-Action
Encoder
00
D2
Observ. Encoder
Observ. Encoder
MLP
q
Prev.
Prev. Action
a
Action
MLP
a
RNN
Encoder
MLP
Reward
Reword
Encoder
Actor Network
Encoder
Critic Network
Figure 6: The network architecture of our implementation on recurrent model-free RL. The left part shows the actor network, and the right shows the critic network. Each block shows trainable module, with independent weights. We italicize the previous action and reward encoders as they are optional. By default, each encoder has one hidden layer, each RNN is one-layer LSTM GRU, each MLP has two hidden layers.
Fig. shows our (separate) recurrent actor-critic architecture. Table shows the main hyperparam- eters we adopt for each subarea. We did not tune these hyperparameters, except that we adjusted the number of gradient steps so that all the experiments could be completed in 36 hours.
For Markovian policies (SAC and TD3), we remove the encoders and RNNs from the actor-critte ector-critic architecture, and train them with same hyperparameters we report the results of either SAC or TD3, whichever achieves higher returns. For Oracle policies, we use the well-tuned results from Table ("SAC w/ unstructured row") in Raffin al. (2021), which is based on Stable Baseline3 (Raffin, 2020), for "standard" POMDPs. For the other subareas, we have to run the Markovian policies (SAC and TD3) with access to the hidden states, using the same training hyperparameters as those of recurrent policies. But these Oracle policies might be not well-tuned given same environment and gradient steps, especially in robust RL and generalization in RL.
those of recurrent policies. For each task,
We also show the settings of the specialized methods we compared in the main paper in Table 6. Note that recurrent model-free RI. share the exactly same settings as (off-policy) VariBAD (Dorfman
16

---

Under review as a conference paper at ICLR 2022
Table 5: Hyperparameter summary in our implementation of model-free recurrent RL. For each subarea, we report the hidden layer size of each module, RL and training hyperparameters. For Meta-RL, we take the model on Cheetah- Vel example, which follows the architecture design of off-policy VariBAD (Dorfman et al., 2020). The hidden size observation-action encoder is the sum of that of observation encoder, previous action encoder (if exists), and reward encoder (if
exists).
Meta-RL*
POMDP
Robust RL
Gen. in RL
Observ. encoder Prev. Action encoder* Reward encoder*
[32]
Hidden layer
[16]
[16]
size
RNN ML.P
[128]
[128,128,128]
[256, 256]
Learning rate Discount factor y
3e-4
0.99
RL
Smoothing coef
0.005
hparams
SAC temperature
automatically updated by Haarnoja et al. (2018b)
TD3 noises
default values from Fujimoto et al. (2018)
Replay buffer size
1e6
Batch size
32
64
Training
Environment steps
5M
1.5M
3M
hparams
Gradient steps
0.
1.5M
0.6M
Policy
Largest input space
oard oard
oar
oa
oar
inputs
Best input space
Table 6: Settings of the specialized methods compared in the main paper. For Meta-RL, take the model on Cheetah- Vel as example.
Meta-RL*
POMDP
Robust RL
Gen. in RL
Approach Memory-based? Off-policy?
(off-policy) VariBAD
VRM
MRPO
EPOPT
X
X
/
X
Input space Access to hidden states?
0a rd
car
X
et al., 2020) and VRM (Han et al., 2020). For MRPO (Jiang et al., 2021) and EPOPT (Rajeswaran et al., 2017a), they adopt totally different settings, i.e. on-policy Markovian approaches to MDPs (with access to the ground-truth state (s) of environment). Thus, in fact, MRPO and EPOPT should be more viewed as oracle policies as upper bounds of recurrent model-free RL.
C EVALUATION DETAILS
The bar charts in Fig. and 18 and Table 3 adopt the final performance of each method. We each instance/variant in our method and each compared method with 4 random seeds. The final performance calculated by the average performance the last 20% environment steps across the 4 seeds.
In terms of selecting the best variant in our method for each subarea, we first compute the final performance of each variant, then normalize the final performance into [0, 1]. and finally select the best variant the average of the normalized final performance across all the environments in each subarea.
For the bar charts Fig. and 18, we show the normalized returns of each method, calculated by R- Ruiin Emax Kamn [0, 1]. where R is the raw average return of that method and Rmax and Rmin the maximum and minimum of all the methods including Oracle policy and Random policy.
17

---

Under review as a conference paper at ICLR 2022
D ENVIRONMENT DETAILS
D.1 "STANDARD" POMDP
Except for the classic Pendulum environment, use PyBullet (Coumans & Bai, 2016) as the sim- ulator for "standard" POMDP environments. As the practice in VRM (Han et al., 2020), we remove all the position/angle-relatee entries the observation space for "-V" environments and velocity- related entries for "-P" environments, to transform the original MDP into POMDP.
(Pendulum, Hhheetah,Hoppeee -alker)-P. The "-P" stands for the environments that keep position-related entries by removal of velocity- entries. Thus, the observed state s" includes positions p. while the hidden state s is the velocities u.
(Pendulum,Ants Cheetah,Hopper Walker)-V. The "-V" stands for the environments that keep reelated entries by removal of position-related entries. Thus, the observed state s" includes positions e, while the hidden state s is the velocities p.
D.2 MRTA RL.
Semi-Cirele. We directly follow off-policy version of VariBAD (Dorfman et al., 2020). The ob- served state s includes the agent's 2D position and the hidden state sh is referred to the goal state Pg. The goal state only appears in reward function: R(s 1 at s²) R(pt-1,Pg) (llpt+1 Po 2 S r). The dynamic function I' is independent of the goal state.
(2020) using MuJoCo (Todorov et al., 2012) simulator of Ha fCheetah-v2. The hidden state s is the target velocity "y and observed state 8 includes the velocity u. Reward function includes both the hidden state and action: R(s 58 de, sh) Rivr,, Ug. az) -lle, uglli 0.05||aall2. The dynamic function T is inde- pendent of the goal state.
Cheetah-Vel We directly follow Dorfman
Wind. We modified the parameters of Wind environment in Dorfman et al. (2020) to make it harder to solve. The agent must navigate to a fixed (but unknown) goal Py within a distance of D from its fixed initial state. Similarly Semi-Cirele, the reward function is goal conditioned but without hidden state: R(s s4+1, Gr s²)) = R(pt+1. Pg) = 1[1 - Poll2 S r). The hidden state 4 appear in the deterministic dynamics noise term, i.e. 8 = of + + 8², where 8 is sampled from U-00.88 0.08) at the initial time-step and then kept fixed.
D.3 ROBUST RL
(Hopper, Walker, Cheetah), )heetah)-Robust We directly adopt the environments used in MRPO (Jiang et al., 2021). In each environment, the hidden state is the dynamics parameters including the density and friction coefficients of the simulated robot in roboschool, adapted from the SunBlaze (Packer et al., 2018). The exact ranges of the hidden states in each environment can be found in Table I of MRPO (Jiang et al., 2021). We evaluate the algorithms with 100 tasks in each environment, and use the average of them as average returns, and the average of the worst 10% of them as worst returns, following MRPO paper.
D.4 GENERALIZATION IN RL
(Hopper, (rrcetah)-Generatinn Generalizze. We directly adopt the environments used in SunBlaze (Packer et al., 2018), 8). In each environment, the hidden state is the dynamics parameters including the den- sity, friction coefficients, and the power of the simulated robot in roboschool. The exact ranges of both interpolation and extrapolation in the hidden state distribution for each environment can be found in Table I of SunBlaze (Packer et al., 2018). We follow the practice of SunBlaze evaluate interpolation and extrapolation success rates.
18

---

Under review as a conference paper at ICLR 2022
E FULL EXPERIMENTAL RESULTS
E.1 LEARNING CURVES OF ALL THE COMPARED METHODS
In this subsection, we show all the learning curves of all the compared methods (including Oracle policy as upper bound, Markovian and Random policies as lower bounds) in each subarea, namely "Standardd' POMDPs (Fig. 7 and Fig. 8), meta RL (Fig. 9), robust RL (Fig. 10), and generalization RL (Fig. 11). The final performance these learning curves generate the bar charts Sec. 5.1.
3000
ppo_gru
Ours: recurrent model-free
3000
aZc_gru
2000
ppo_gru
modal-frre
a2c gru
retur
2000
VRM
return
1000
VRM
Markavian
Markavian
Rancom
Rendom
-1000
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
env_steps
1e6
env_steps
1eb
Ant-P
Cheetah-P
Ours: recurrent model-free RI
Ours: recurrent model-free Ri
ppo gru
ppo gru
2000
retur
1500
azc
Gracle
return
1500
Oracie
azc aZc_gru
1000
2000
Markovian
Random
Random
500
0.6
0.8
0.0
0.4
0.6
env. steps
env. steps
1e6
Hopper-P
Walker-P
Learning curves on "standard" POMDP environments that preserve positions & angles but occlude velocities in the states in PyBullet (Coumans & Bai, 2016) (namely ".P"). We show the results from the single best variant of our implementation on recurrent model-free RL, the popular recurrent model free on-policy implementation (PPO-GRU, A2C-GRU) (Kostrikov, 2018), 8), and also model-based method VRM (Han et al., 2020). Note that VRM is around 5x slower than ours, so we have to run 0.5M environment steps for it.
Figure 7:
E.2 SINGLE FACTOR ANALYSIS ON OUR METHOD
Our analysis will focus on ablating the important design decisions: the actor-critic architecture (Arch), the policy input space (Inputs), the underlying model-free RL algorithm (RL), the RNN encoder (Encoder), and the RNN context length (Len), As there are several decision variables in our method, we could only show the results of each variable in the plots by averaging the perfor- mance over the other variables. which we call single factor analysis. In this kind of analysis, we can only say one value is better than another in one factor in the average sense (not the maximal sense); therefore it can show the robustness of each factor, but cannot tell the best choices (showed in See. 5.1). We show single factor analysis plots in each subarea, namely "Standard" POMDPs (Fig. 12 and Fig. 13), meta RL (Fig. 14), robust RL (Fig. 15), and generalization in RL (Fig. 16).
From these plots, we can see that each decision factor can make a difference some environments. For example, the choice of RL algorithm is crucial in Ant-P (Fig. 12), Cheetah-V (Fig. 13), Wind (Fig. 14) and ments (Fig. 12), Cheetah- Vel (Fig. 14), and both the generalization environments (Fig. 16). The policy input space can make a difference most "_p" environments (Fig. 12) possibly because oar contains the information of missing velocities.
(Fig. 16). The context length is essential all the ".P" environ-
19

---

Under review as a conference paper at ICLR 2022
model-free
3000
Ours: recurrent model-free
3000
ppo_gru
2000
ppo_ge
VRM
a7c_gru
return
2000
Oracle
return
1000
VRM
Oracle
Markovian
1000
Random
-1000
-2000
env steps
env_steps
1e6
Ant-V
Cheetah-V
2500
Ours: recurrent macel-free
ppo_gru
2000
2000
ppo_gru gru
VRM
Ours: recurrent madel-free RL
1500
a2c gru
return
1500
a2c_gru
Oracle
1000
Markovian
Markovian
Rancom
Random
500
0
0.0
0.2
0.4
0.6
0.8 env_stee steps
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
144
1e6
svesstepp
1e6
Hopper-V V
Walker-V
Figure 8: Learning curves on "standard" POMDP environments that preserve velocities but occlude positions & angles in the states in PyBullet (Coumans & Bai, 2016) (namely "-V"). We show the results from single best variant of our implementation on recurrent model-free RL, popular recurrent model-free on-policy implementation (PPO-GRU A2C-GRU) (Kostrikov, 2018), and also model-base method VRM (Han et al., 2020). Note that VRM is around 5x slower than ours, so have run 0.5M environment steps for it.
E.3 ADDITIONAL RESULTS ON SEPARATE VS SHARED RECURRENT ACTOR-CRITIC ARCHITECTURE
In Fig. of Sec. 5.2, we show the importance of selecting a separate recurrent actor-critic architec- ture meta RL environment, Semi-Cirele. Now we show the result in another POMDP environ- ment, Pendulum-V, V, which occludes the positions and angles, in Fig. 17. We can see that the shared encoder architecture is also worse than the separate one, possibly due to the different gradient scales in actor and critic losses w.r.t. the encoder.
E.4 ADDITIONAL RESULTS ON COMPARISON WITH VRM
Both Fig. and Fig. 18 shows the final performance the same single variant of our implementa- tion, but the former shows results with 1.5M simulation steps while the latter shows our results with 0.5M simulation steps to match with those of VRM due to the time budget.
20

---

Under review as a conference paper at ICLR 2022
100
Ours: recurrent model-free RL
varibad
80
Oracle
Markovian
return
Random
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
env_ stssps
1e6
Semi-Cirele
60
Ours: recurrent.poodekfeeeeRt
varibad
Oracle
40
Markovian
return
Random
20
0
10000000000000000000000000000000
env_steps
Wind
Ours: recurrent model-free RL
-200
varibad
Oracle
Markovian
return
-400
Random
-600
-800
2
3
5
env nnv_steps
1e6
Cheetah-Vel
Figure 9: Learning curves meta RL environments. We show the results from the single best variant of our implementation on recurrent model-free RL, the specialized meta RL method VariBAD (Dorfman et al., 2020)
21

---

Under review as a conference paper at ICLR 2022
Dufs: model-free
2000
model-free
MRPO
MRPO
return, 3Vg
2000
Markovian
Oracle
retum wors
1500
Markovian Random
Oracle
1500
Rendom
1000
1000
500
0.00
0.25
0.50
0.75
1.00
1.50
1.75
2.00
0.00
0:25
0.50
0.75
1.00
1.50
1.75
2.00
env steps
1e7
eny steps
1e7
Hopper- Robust
2000
2500
Ours: recurrent model free RL
Ours: recurrent model free RL
MRPG
MRPO
2000
Oracle
Dradie
retum avg
Markovian
Markovian
1500
Random
1000
Random
retur
1000
0.0
0.2
0.4
0.6
0.8
1.0
1.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Steps
env_ steps
1e7
Cheetah-Robure
2000
Outs:: model-free
1250
model-free
MRPO
MRPO
neturn, 3Vg
1500
Oracle
Markovian Rendom
Markovian
Oracle
Random
1000
retur,
500
500
250
0.00
0.25
0.50
0.75
1.00
1.75
2.00
0.00
0.25
0.50
1.75
2.00
env_ sne_steps
1e7
eniv env__step stepst
167
Walker-Robust
Figure 10: Learning curves robust RL environments. We show the average returns (left fig- ures) and worst returns (right figures) from the single best variant of our implementation on re- current model-free RL, the specialized robust RL method MRPO (Jiang et al., 2021). Note that our method is much slower than MRPO, so we have to run our method within 3M environment steps. But the results show that our method have much better sample efficiency over MRPO.
22

---

Under review as a conference paper at ICLR 2022
0.8
Ours: recurrett model-free RI
A2C
0.3
A2C RC
Ours: recurrent model-free
Interpelation
0.6
EPOpt-PPO-FF
Extrapolation
EPOpt-PPO-FF
Cracle
Oracle
Markovion
Markgvian
0.4
Random
Random
0.1
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
1.0
2.0
3.0
env steps
1e6
env niv_steps
1e6
Hepper-Generalize
1.0
0.6
recurrent
0.8
A2C.80 BC
A2C.
Interpolation
EPOpt-PPO.FF PPO-FF
Extrapolation
EPOpt EPOpt-PPO.FF
PPO-FF
0.6
Cracte
0.4
Markovian
Markovian
0.4
Random
Random
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
steps
le6
steps
1e6
Cheetah-Generaan
Figure 11: Learning curves on generalization in RL environments. We show the interpolation success rates (left figures) and extrapolation success rates (right figures) from the single best vari- ant of our implementation on recurrent model-free RI. We also show the final performance of the specialized method EPOPttPPO-FF (Rajeswaran al., 2017a) and another recurrent model-free (on-policy) RL method (A2C-RC) copied from the Table 7 & 8 in Packer et al. (2018).
2500
gru
Encoder
2000
1d5
RL
2000
Encoder
30000
50C
fctm
500
0.5
3.0
1.5
env stops
1e6
om stops
166
Inputs
1500
2000
3000
500
3.0
1.5
stops
1ed
onv, stops
1
emy: steps
Twb
6mV steppt
166
Ant-P P
Cheetah-P
RL
Encoder
Encoder
103
12:m
3000
S2C
1000
tstm
1000
105
return
500
500
0.0
0.5
1.0
3.5
B.C
0.5
1.0
0.0
0.5
1.0
1.5
0.0
0.5
3.0
15
stops
CRv stops
CITy,
Stoos
Lon
Inputs
Lon
Inputs
2000
3000
3000
reourn
500
0.5
1,0
3.5
1.5
Hopper-P
Walker-P
Figure 12: Ablation study of our implementation on "standard" POMDP environments that preserve positions & angles but occlude velocities the states pybullet (Coumans Bai, 2016) (namely "<p"). We show the single factor analysis on the 4 decision factors including RL, Encoder, Len, and Inputs for each environment.
23

---

Under review as a conference paper at ICLR 2022
600
600
Encoddr
2000
RL
1000
Encoder
400
400
200
1000
1.0
1.3
166
pupss
Inputs
600
retum
2000
retur
-100
steps
steos
106
106
Ant-V V
Cheetah V
RL
Encoder
RL
Encodor
1000
183
bt.m
103
gru tem
500
500
200
200
100
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.5
0.0
0.5
1.5
1.5
stops
cnv staps
stops
6TV staps
Lee
Inputs
Len
Inputs
1000
300
GM
500
64
300
Aar
00
200
200
08
100
0.0
0.5
3.5
0.5 steps
1.0
0.0
0.5
3.5
0.0
0.5
1.5
env_steps
cnv steps
Hopper-V V
Walker-V V
Figure 13: Ablation study of our implementation on "standard" POMDP environments that preserve velocities but occlude positions & angles the states in pybullet (Coumans Bai, 2016) (namely ".V"). We show the single factor analysis the 4 decision factors including RL, Encoder, Len, and Inputs for each environment.
24

---

Under review as a conference paper at ICLR 2022
Arch
RL
Shared
50
sac
separate
retur
td3
0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
env_ steps
env @nv_steps
Encoder
Len
Ismm
retum
120
25
25
0.0
0.5
1.0
0.0
0.5
1.5
env stussttess
1e6
env stv_steps
1e6
Semi-Circle
RL
Encoder
50
sBc
td3
retum
1stm
25
200000 400000 600000
200000 400000 600000 env env_steps
env snv_steps
Inputs
50
50
return
64
retum
25
25
200000 400000 600000
0
200000 400000 600000 steps
steps
Wind
RL
Encoder
-250
-250
gru
Id3
returm
Istm
-500
-500
-750
-750
2
env iiv_steps
1e6
env stv_steps
1e6
Len
-250
64
-500
400
-750
env siv_stteps__
1e6
Cheetah-Vel
Figure 14: Ablation study of our implementation on meta RL environments, We show the single factor analysis covering all the decision factors
25

---

Under review as a conference paper at ICLR 2022
RL
Encoder
Len
retum av8
2000
avg
2000
td3
gru
retum
1000
1000
3
2
1e6
cnvy steps
1e6
env _tvps
1e6
Inputs
RL
Encader
return worst
sBc
1000
ltt
1000
gnu
10000
500
relur
500
3
0
1
3
eriv sie_steep
eriy steps
eni sieps
1e6
Inputs
5
return worrs
1000
1000
64
return
500
500
env_steps
steps
106
Hopper-Robust Encoder
2000
sac
return 30D
2000
Isam
relurn avg
2000
Id3
1000
1000
1000
env jiv_sttps
cnv steps
env jiv_stops
Inputs
Encoder
2300
rotum worst
return return_au
1000
td3
1000
gru
1300
return
steps
186
steps
166
1e6
Len
Inputs
5
return warst
1000
1000
1
3
0
2
env, mrw_steps
1e6
env. sheps
106
Cheetah-Robur Encoder
RL
return avg
return
sac
$1000
gru
1000
env int_sstep
1e6
env_ stepsttepssss
100
env_stops
1eb
Inputs
RL
Encadier
return worst
td3
Istm gru
1.000
300
5000
3
0
3
env. steps
env. steps
146
env. steps
1e6
Inputs
5
return worst
500
64
500
return
env_steps
steps
166
Talker-RRbbst
Figure 15: Ablation study of our implementation robust RL environments. We show the single factor analysis on the 4 decision factors including RI, Encoder, Len, and Inputs for each environment in both average returns and worst returns.
26

---

Under review as a conference paper at ICLR 2022
RL
Len
Inputs
Interpolation
0.5
td3
SBC
Interpolation
0.5
64
Interpol reerooatton
0.50
128
0.25
08r
0.0
0.0
0.00
2
2
env_steps
1e6
niv__teps
1e6
env_steps
1e6
RL
Len
Inputs
Extrapolation
Sac
Extrapolation
Extrapolation
0.2
0.0
0.0
0.0
env_steee steps
1e6
env_ @nv__seep
env_steps
1e6
Hopper-Gennrann
RL
Len
Inputs
1.0
Interpolation
sac
Interpolation
5
Interpolation
0.5
Id3
0.5
0.5
03
oar
0.0
0.0
0.0
env jtv_sttpp
env_steps
env_ @nv__tepp
RL
Len
Inputs
Extrapolation
sac
Extrapolation
0.50
5
Extrapolation
0.50
0.25
0.25
128
0.00
0.00
0.00
2
3
2
2 cnv_steps
s__psttpp
1e6
stv_steps
1e6
1e6
Satetth--Generale
Figure 16: Ablation study of our implementation on generalization in RL environments. We show the single factor analysis on the decision factors including RI, Len, and Inputs for each environment both interpolation and extraploation success rates.
-500
mn grad norm
return
-1000
sac-Istm-2008 secclsst--20--ar-hhie 200-ooa-aa-d
sac-Istm-22
aec-istm-200-aar-eeee 200-oar-seearate
-1500
sac-Istm-200 200m-22000aarsareees
sac- lett-ssm-2000 200.aaaaaassepratteaatto
10000
20000 30000
40000
50000
10000
20000 30000
40000 50000
ste_steep
Figure 17: Comparison between shared and separate recurrent actor-critic architecture with all the other hyperparameters same, on Pendulum- V, toy "standard" POMDP environment. We show the performance metric (left) and also the gradient norm of the RNN encoder(s) (right, in log-scale). For the separate architecture, :rr t ic and ar refer to the separate RNN in critic and actor networks, respectively.
27

---

Under review as a conference paper at ICLR 2022
model- free
VRM
0.8
Markovian
Normalizer lized return
0.5
0.4
0.3
0.2
0.0
Ant-P
Ant-V
Cheetah-P
Cheetah-V
Hopper-P
Hopper-V
Walker-P
Wa Iikker/V
Env
Figure 18: Final normalized returns of our implemented recurrent model-free RL algorithm with the same hyperparameters, and the prior method VRM (Han et al., 2020) across the eight environ- ments in "standard" POMDPs, each which trained in 0.5M simulation steps.
28

---

Under review as a conference paper at ICLR 2022
F ABLATION STUDY ON RNN ARCHITECTURES
Our implementation of recurrent model-free RL uses the popular 1-layer LSTM or GRU. To in- vestigate the effect of RNN architecture, we ablate RNN variants: one is 2-layer stacked LSTM/GRU, the other is Particle-Filter RNN (PF-RNN (Ma et al., 2020a)). PF-RNN maintains stochastic belief (posterior distribution) through set of weighted particles better capture the uncertainty and multi multi-modality, compared to classic RNN's deterministic belief. DPFRL (Ma et al., 2020b) applies PF-GRU to POMDP tasks, using the mean of particles and MGF features of parti- cles as the belief state. We follow the implementation of PF-GRU in DPFRL (Ma et al., 2020b) to replace regular GRU, keeping the other model components the same.
We try 2-layer LSTM/GRU, on our best variant across all the environments. We run PF-GRU on "standard" POMDP environments where the best variant also uses GRU for fair comparison, and do not try PF-LSTM as it is not adopted in DPFRL (Ma et al., 2020b).
Since 2-layer LSTM/GRU doubles the training time of 1-layer LSTM/GRU given the same gradient updating frequency, have to decrease the frequency from 1.0 to 0.6 in "standard" POMDPs. Similarly, PF-GRU costs 15x than 1-layer GRU, so we also have to decrease the frequency from 1.0 0.6. All the other hyperparameters remain the same.
Figs. 19 to 23 show all the learning curves of the ablation study on RNN architectures. Except for cases (e.g. Ant-P, Walker- V), the RNN variants perform worse than are par with 1-layer LSTM/GRU, although in most cases the RNN variants can still outperform the Markovian policies. Possible reasons that the RNN variants need hyperparameter tuning and more training samples to converge.
3000
2gru
Ours: recurrent model-free
3000
td3_ptgru
td3_2gru
model-free
2000
td3 td3_pfgru
Oracle
Opacle
retum
2000
Markovian
return
1000
Markovian
Random
Random
1000
-1000
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
env_steps
1e6
env_steps
1e6
Ant-P
Cheetah-P
2500
Ours: recurrent model--fee
Ours: recurrent model--fre RL
td3 2gru
2000
td3 2gru
2000
pfgru
td3 _00_pfrru
Ojacle
2500
Oracle
return
1500
Markovien
return
Markovian
1000
1000
500
1.4
env. steps
env. env_steps
Hopper-P
Walker-P
Figure 19: Ablation study on RNN architecture in "standard" POMDPs ("-P"), We show two models that are only different from our best single variant in the RNN architectures, namely using 2-layer stacked GRU (td3-2gru) and PF-GRU (Ma et al., 2020a;b) (td3-pfgru), instead of 1-layer GRU.
29

---

Under review as a conference paper at ICLR 2022
model-free
3000
Ours: recurrent model-free
3000
td3_pfgru
td3_2gru
2000
td3_ td3_2gru
td3 td3_pfgru
Oracle
return
2000
Markovian
return
Markovien
Random
Roaddom
1000
-1000
-2000
0.8
1.2
1.2
env stops
1e6
env nnv_steps
1e6
Ant-V
Cheetah-V
2500
Ours: recurrent macell-ree
td3_ptgru
2000
Edr_2gru
2000
recurrett model--fre
td3_pfgru
td3, 2gru
ebum
1500
Oracle
return
1500
Oracle
Markovian
Random
Random
500
0
0.0
0.2
0.4
0.6
0.8 env_steps
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1e6
env_stepps
1e6
Hopper-V V
Walker-V
Figure 20: Ablation study on RNN architecture "standard" POMDPs ("-V"), We show two models that are only different from our best single variant in the RNN architectures, namely using 2-layer stacked GRU (td3-2gru) and PF-GRU (Ma et al., 2020a;b) (td3-pfgru), instead of 1-layer GRU.
30

---

Under review as a conference paper at ICLR 2022
100
td3 21stm
Tecurrent model-free RL
Oracle
Markovian
return
Random
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
env_ jiv_steps
1e6
Semi-Cirele
60
Ours: recurrentt.podeeffeeeR
td3 ___llssmm
Oracle
40
Markovian
return
Random
20
0
0
1000:000000000000000000000000000
env_steps
Wind
td3_2lstm
Ours: rede-fen--mdellfree RL.
-200
Oracle
Markovian
return
-400
Random
-600
-800
3
env hnv_steps
1e6
Cheetah-Vel
Figure 21: Ablation study on RNN architecture in Meta RL. We show one model that are only different from our best single variant in the RNN architecture, namely using 2-layer stacked LSTM (td3-21stm) instead of 1-layer LSTM.
31

---

Under review as a conference paper at ICLR 2022
2500
Ours: recurrent model-free
2000
Ours: recurrent model-free RL.
2000
Oracle
1500
Orecle
return avg
Markovisn
Markovian
Ramdom
11000
retum
1000
500
0.0
0.5
1,0
1.5
2.0
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
stv_steps
env_ env_steps
1e6
Hlopper-Robust
model-free
model-free
2500
td3 21stm
td3 td13_2lstm 21stm
Oracle
Oracle
return 3V9
retum worst
1500
1500
Random
Random
1000
1000
500
500
0.0
0.5
1.5
2.0
3.0
0.0
1.0
3.5
2.0
3.0
env_steps
1e6
BTI steps
1e6
Cheetah-Robust
Ours: recurrent model-free
1250
Ours: recurrent model-free RL
return Dvi
3500
Oracle
Markovion
retum werst
1000
Markovian
Oracle
750
1000
250
0.0
0.5
1.0
1.5
2.0
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
steps
1e6
env_ stepsteep
1e6
Walker-Robust
Figure 22: Ablation study on RNN architecture in robust RL. We show one model that are only different from our best single variant in the RNN architecture, namely using 2-layer stacked LSTM (td3-21stm), instead of l-layer LSTM.
32

---

Under review as a conference paper at ICLR 2022
0.8
recurrent model-free
modal-free
td3, 2istm
td3;2/stm
Interpolation
0.6
Cracle
Extrapolation
Oracle
Markovian
0.2
Markovian
Random
Random
0.4
0.2
0.0
0.0
0.0
0.5
1.0
1.5 env_steps
2.0
2.5
3.0
0.0
0.5
1.0
1.5 env_steps
2.0
2.5
3.0
1e6
1e6
Hopper- Generalize
1.0
Ours: recurrent model-free RL
Ours: recurrent model-free RL
0.8
td] 21stm
td3] 21stm
Interpolation
Cracle
Extrapalation
Oracle
0.6
Markovian
0.4
Markovian
Random
Random
0.4
0.2
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5 env env_steps
2.0
2.5
3.0
stepstep
1e6
1e6
Cheetah-Gennrrlize
Figure 23: Ablation study on RNN architecture in generalization in RL. We show one model that are only different from our best single variant in the RNN architecture, namely using 2-layer stacked LSTT (td3-21stm), instead of 1-layer L.TM.
33

---

Under review as a conference paper at ICLR 2022
G ADDITIONAL RESULTS ON COMPARISON WITH OFF-POLICY VARIBAD
In our experiment section (Sec. 5) and Fig. 9, we show the learning curves of our method and our re-implemented off-policy VariBAD (Dorfman et al., 2020). To rule the possibility implementation bugs in the off-policy VariBAD, we ran the official off-policy VariBAD implemen- tation", denoted as VariBAD-BOReL.
In Fig. 24, we show official off-policy VariBAD (VariBAD-BORK) B-ReL in short), our re-implemented off-policy VariBAD (VariBAD in short), and one variant of our recurrent model-free RL, which uses the same decision factors as VariBAD-BOReL. (SAC as RI. algorithm, GRU as encoder, oar as input, 400 as context length) for fair comparison. We also try Ant-Dir from Dorfman al. (2020) paper, more challenging meta RL environment than Cheetah-Ve Vel.
In Cini-Circle and Wind, VariBAD-BOReL has similar performance as our re-implemented version (VariBAD).
In Cheetah- Vel, we find VariBAD-BORGS outperforms our re-implemented version, but still has gap from what they reported in Fig. 11 in the appendix of Dorfman et al. (2020). Our implementation still outperforms VariBAD-BOReL, supporting our claim in the main paper.
Finally, in the newly added and more challenging Ant-Dir, found our recurrent model-free RL can even greatly surpass Oracle (trained with same environment and gradient steps), while VariBAD- BOReL somehow has extremely low performance (worse than Random) and suffers from numerical issue to terminate early.
100
60
80
50
40
retum
Ours: recurrent model-free RL
Ours: recurrent model-free
varibad
30
varibad
varibad-BOReL.
varibad-8ORel
20
20
Markovian
10
Markovian
Random
Random
0.0
0.4
0.6
0.8
1.0
1.2
1.6
env_slt srv__sepss
1e6
env env_stesp steps
Semi-Circle
Wind
1100
200
1000
500
400
varibed BÖReL.
Ours: recurrent model-frea RI
5500
varibad
varibad-8ORee
Oracle
-500
Oracle
Markovian
Markovion
-700
Random
-100
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.1
0.6
0.8
1.0
srv_steps
1e7
prre_sleps
1e7
Cheetah- Vel
Ant-Dir
Figure 24: Additional learning curves meta RL environments. We show the results from the single variant of our implementation on recurrent model-free RL saa-grr-oaar4000,, and the spe- cialized meta RL method off- off-policy VariBAD (Dorfman et al., 2020) (their official implementation VariBAD-BOReL- and our re-implementation VariBAD).
Phttps:/ / thub. wtom///onorooooooooooooooooooooe lOORel.
34


Reviews:
[{"review_id": "6f_SahjhGz", "reviewer": "Reviewer_3HH1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n  - A few grammatical mistakes aside, the paper is well-organized and understandable. It sets out a clear picture of the current POMDP landscape, making clear the distinctions between different classes of problem (such as stationary vs nonstationary, across episode, policy inputs) and the concerns that each raise. It describes in detail the process by which the experiments were chosen and performed.\n  - It's surprising to me how novel the contribution is. This is not because any new methods are proposed, but rather because so little previous work has really considered the question of actually getting recurrent algorithms to work well. I think an analysis like this is long overdue, even if I may have some concerns over the experiments themselves. \n  - A good amount of effort clearly went into reproducibility and efficient, readable code\n\nWeaknesses:\n  - I'm not as convinced as I'd like to be by the experiments, which are ultimately what this empirical study rest on. I'll detail a few issues:\n    1) In each class of problem, the tuned RNN is only compared against one other method. In general, these appear to be recent, SOTA, and well-chosen, but between that and the use of bar graphs instead of learning curves, it's hard to really sanity check. How would something like PFRNNs or DRBPNs compare?\n    2) As a follow-on, I'm not sure that these are really fair comparisons. For the comparisons to other methods, the authors use an off-policy algorithm (SAC) with access to information that is not necessarily common in POMDP literature (rewards and done signals aren't often counted in the stored history). VRM follows a relatively similar training procedure, as does the modified VariBAD, but the other two algorithms are quite different. Can these methods really be compared in terms of simulation steps?\n    3) While the set of all possible RNN architecture choices is enormous, I do think that there were a few more that could have been considered. Most importantly, I'm not fully convinced by the shared vs separate argument. The hypothesis of exploding gradient should be relatively simple to empirically log and demonstrate, and it also heavily depends on the shape of the environment rewards. Most of these environments have large, dense reward signals which need normalization. Another consideration would be whether the RNN should be before or after feature extraction. Finally (though this is more something that I've come across in my own work), layer-normalized RNNs often vastly outperform their standard counterparts.\n    4) I preferred the learning curves found in the appendix to the bar chart comparisons in the main body. I think it would be more informative to compare the methods with some error bars and a better idea of the process. Also, it seemed like some environments (like Cheetah-Vel) didn't learn? A couple baselines (optimal upper-bound, random policy lower-bound) would give a better idea of how well the methods are actually doing.", "summary_of_the_paper": "This paper primarily provides an empirical analysis of recurrent model-free RL on several classes of POMDP, showing that if parameters are well-chosen, the basic approach of just applying a recurrent layer is not only competitive with but often outperforms specially-designed methods for those problem classes. The authors consider different architectures, algorithms, types of RNN, and other parameters and fill in a significant gap left by previous papers, which didn't consider most of the important details of getting model-free RNN methods to work well", "summary_of_the_review": "I'm recommending this paper with some reservations. Overall, I really think work like this is valuable and illustrates a large gap in knowledge that was previously unaddressed. I've always found it frustrating that the approaches to partial observability are so ad-hoc. I also appreciate the clarity with which this paper addresses the classes of POMDP and the considerations required for each. That said, I have some concerns with the thoroughness of the experiments and would like to see more baselines and fairer comparisons.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3OAvZjnWGvo", "reviewer": "Reviewer_2WFY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work is quite straightforward and simple - which is good. The authors state that an end-to-end approach can be competitive with sota methods on tailored problems, if only specific considerations are taken into account.\n\nThe main drawback of this work is that the results are inconclusive.\n1. There is no theoretical reasoning behind the authors claims. No motivating principles. It is hard for me to assess the correctness of their claims solely from their experiments on 4 mujoco tasks.\n2. While the authors did run extensive experiments on the 4 mujoco tasks, these are definitely not enough. For the claims of this paper, I would at least want to see extensive experiments on another large domain (e.g., 4 other environments in atari, and perhaps at least 1 hard task in another domain).\n3. When do things fail? Results that only show successes usually feel cherry picked. I assume the authors can find environments and/or algorithms for which their design choices fail. It is important to show these as well to understand the tradeoffs better.", "summary_of_the_paper": "This paper is concerned with understanding when RNNs are useful in POMDPs. They should that, for a class of POMDPS (e.g., meta RL, robust RL), standard RNNs can be competitive with solutions that are tailored to the given POMDP structure. The authors discuss four design considerations that they claim are essential for performance: (1) decoupling actor and critic networks, (2) using off-policy instead of on-policy algorithms, (3) context length of RNN, and (4) using rewards as historical input. They should on 4 mujoco baselines that their results are competitive with sota methods whenever these design choices are taken into account.", "summary_of_the_review": "Good direction, results are inconclusive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "28TKrkSsEDm", "reviewer": "Reviewer_iXRL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although this paper does not propose any new algorithms or theories, I think these implementation details with RNN in RL are very practically useful experiences for solving RL problems. These introduced tricks, designments, and techniques on the architecture, input space, RL algorithms, context length, etc., are all valuable conclusions for the readers.\n\nMoreover, the perspective viewing meta RL, robust RL, and generalization or transferability in RL as POMDP problem is also very interesting and reasonable. From this view, I can connect many of these different tasks from a principled way.\n\nIn the experiments, could the authors provide details on the network structures used for different tasks? Since efficiency of the implementations are discussed and compared with prior works. Also, instead of comparing with some SOTA methods in specific tasks, would the simplest baseline, i.e., the plain network by just removing the RNN layer, be compared as a reference in the curves? This would provide a most straightforward view that how much benefit RNN brings.\n\nFor all these tasks, will stacking RNN layers be better than a single layer?\n\nAnother question is that as demonstrated in the experiments, a carefully designed RNN implementation can always result in much better performance to solve POMDP problem, does there indeed exist theoretical explanations connecting the specific architecture of RNN and the analysis based on belief in previous POMDP theories?\n\nOverall, although this paper focuses on RNN implementations, instead of proposing new algorithms and methods, I think the contexts discussed in the paper can contribute to the practical RL domain.\n", "summary_of_the_paper": "This paper revisits recurrent neural network based model-free RL methods with carefully tuning on various aspects of the learning details. Results on a large number of tasks demonstrate that with a well-tuned RNN implementation of RL methods, it is enough to achieve sufficient performance compared with many state-of-the-art competitors on various types of tasks. \n", "summary_of_the_review": "The paper provides many useful implementation details for RNN based RL methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper recognizes that several common sub-problems studied in RL, such as meta RL and generalization in RL, can be cast as POMDPs. Using this observation, the authors evaluate how a straightforward approach to deal with POMDPs---using a recurrent neural network---compares to more specialized approaches. The reviewers agree that the research question studied in this paper is very interesting. However, after careful deliberation, I share the view of reviewer 2WFY that the results insufficiently support the claims made in the paper. In particular, I view the main claim from the abstract "We find that a careful architecture and hyperparameter decisions yield a recurrent model-free implementation that performs on par with (and occasionally substantially better than) more sophisticated recent techniques in their respective domains."  as insufficiently supported. The main issue with the experiments is that only a small number of simple domains are considered. As Luisa points out in the public comments, variBAD dominates recurrent baselines when more complex tasks are considered, while on simpler domains such as the Cheetah-Vel domain considered in this paper, it performs similar to a recurrent model-free baseline. In the rebuttal the authors have added a more complex domain to address this, showing that a recurrent model-free baseline outperforms an off-policy version of variBAD. However, I view these results as inconclusive, as only a single complex domain is considered and they appear to contradict previous results with on-policy variBAD. For these reasons, I don't think the work in its current form is ready for publication at ICLR. But I want to encourage the authors to work out this direction further. In particular, adding more complex domains and also considering the on-policy variBAD method, can make this work stronger.
Decision: Reject