Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
Authors: Avi Singh
Huihan Liu
Gaoyue Zhou
Albert Yu
Nicholas Rhinehart
Sergey Levine
Venue: ICLR 2021 Oral (2021)
Keywords: reinforcement learning
imitation learning

Abstract: Reinforcement learning provides a general framework for flexible decision making and control, but requires extensive data collection for each new task that an agent needs to learn. In other machine learning fields, such as natural language processing or computer vision, pre-training on large, previously collected datasets to bootstrap learning for new tasks has emerged as a powerful paradigm to reduce data requirements when learning a new task. In this paper, we ask the following question: how can we enable similarly useful pre-training for RL agents? We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials from a wide range of previously seen tasks, and we show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent's ability to try out novel behaviors. We demonstrate the effectiveness of our approach in challenging robotic manipulation domains involving image observations and sparse reward functions, where our method outperforms prior works by a substantial margin. Additional materials can be found on our project website: https://sites.google.com/view/parrot-rl

Published as a conference paper at ICLR 2021
PARROT: DATA-DRIVEN BEHAVIORAL PRIORS FOR REINFORCEMENT LEARNING
Avi Singh", Huihan Liu", Gaoyue Zhou, Albert Yu, Nicholas Rhinchart, Sergey Levine University of California, Berkeley
ABSTRACT
Reinforcement learning provides general framework for flexible decision mak- ing and control, but requires extensive data collection for each new task that agent needs to learn. In other machine learning fields, such as natural lan- guage processing or computer vision, pre-training on large, previously collected datasets to bootstrap learning for new tasks has emerged to reduce data requirements when learning a new task. In this paper, we ask the following question: how can enable similarly useful pre-training for RL agents? We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials from a wide range of previously seen tasks, and we show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent's 's abil- ity to try out novel behaviors. We demonstrate the effectiveness of our ap- proach in challenging robotic manipulation domains involving image observa- tions and sparse reward functions, where our method outperforms prior works by a substantial margin. Additional materials can found on our project website: https://sites: google. com/viiiiiioooooo hommmview//eee wwwe//wwwwrot./.. http//parotrrr
powerful paradigm
INTRODUCTION
Reinforcement Learning (RI.) is an attractive paradigm for robotic learning because of its flexibility in being able to learn diverse range of skills and its capacity to continuously improve. However, RL algorithms typically require a large amount of data to solve each individual task, including simple ones. Since an RL agent is generally initialized without any prior knowledge, it must try many largely unproductive behaviors before it discovers a high-reward outcome. In contrast, humans rarely attempt solve new tasks in this way: they draw their prior experience of what is useful when they attempt new task, which substantially shrinks the task search space. For example, faced with a new task involving objects on a table, a person might grasp an object, stack multiple objects, or explore other object rearrangements, rather than re-learning how to move their arms and fingers.
Can we endow RL agents with a similar sort of behavioral prior from past experience? In other fields of machine learning, the use of large prior datasets to bootstrap acquisition of new capabil- ities has been studied extensively to good effect. For example, language models trained on large, diverse datasets offer representations that drastically improve the efficiency of learning downstream robotics
tasks (Devlin et al., 2019). What would be the analogue of this kind of pre-training and RI.? One way we can approach this problem is to leverage successful trials from wide range of previously seen tasks to improve learning for new tasks. The data could come from previously learned policies, from human demonstrations, et al., 2019). In this paper, we show that behavioral priors can be obtained through representation learning, and the representation in question must not only be a representation of inputs, but actually a representation of input-output relationships a space possible and likely mappings from states actions among which the learning process can interpolate when confronted with a new task.
even unstructured teleoperation of robots (Lynch
What makes for a good representation for RL. Given a new task, a good representation must (a) pro- vide effective exploration strategy, (b) simplify the policy learning problem for the RL. algorithm, and (c) allow the RL agent to retain full control over the environment. In this paper, we address
Equul contribution. Correspondence Avi Singh (avi ingh@bbe r wek.eyyeeee..

---

Published as a conference paper at ICLR 2021
Grapp
candi
Scene
oite
Scene N
Training Data
Policy Ti 9
Prior Julie
Exploration RL: Use our behavioral prior learn specific task via RL. For example, pick and place the chair.
Figure I: Our problem setting. Our training dataset consists of near-optimal state-action trajectories (without reward labels) from a wide range of tasks. Each task might involve interacting with a different set of objects. Even for same set objects,
task can be different depending on our objective. For example, the upper right corner, the objective could picking up a cup, or it could be to place the bottle on the yellow cube. We learn a behavioral prior from this multi-task dataset capahle of trying many different useful hehaviors when placed in new environment, and can aid RI. agent to quickly learn specific task in this new environment.
all of these challenges through learning an invertible function that maps noise vectors to complex, high-dimensiooal environment actions. Building prior work in normalizing flows (Dinh et al., 2017), we train this mapping to maximize the (conditional) log-likelinn lig-liied of actions observed in successful trials from past tasks. When dropped into from unit Gaussian, and use the learned mapping (which we refer to as the behavioral prior) to generate likely environment actions, conditional the current observation. This learned mapping essentially transforms the original MDP into simpler one for the RL agent, as long as the original MDP shares (partial) structure with previously seen MDPs (see Section 3). Furthermore, since this mapping is invertible, the RL agent still retains full control over the original MDP; for every possible environment action, there exists a point within the support of the Gaussian distribution that maps to that action. This allows the RI. agent to still try out new behaviors that are distinct from what was previously observed.
MDP, the RL agent
sample
Our main contribution is a framework for pre-training in RL from a diverse multi-task dataset, which produces a behavioral prior that accelerates acquisition of new skills. We present an instantiation of this framework in robotic manipulation, where we utilize manipulation data from a diverse range of prior tasks to train our behavioral prior, and then use it to bootstrap exploration for new tasks. By making it possible pre-train action representations large prior datasets for robotics and RL, we hope that our method provides a path toward leveraging large datasets in the RL and robotics settings, much like language models can leverage large text corpora in NLP and unsupervised pre- training can leverage large image datasets in computer vision. Our method, which we call Prior AcceleRated ReinfOrcemenT (PARROT), is able to quickly learn tasks that involve manipulating previously unseen objects, from image observations and sparse rewards, in settings where RL from scratch fails to learn a policy all. We also compare against prior works that incorporate prior data for RL, and show that PARROT substantially outperforms these prior works.
2 RELATED WORK
Combining RL with demonstrations. Our work is related to methods for learning from demon- strations (Pomerleau, 1989; Schaal et al., 2003; Ratliff et al., 2007; Pastor et al., 2009; Ho & Ermon, 2016; Finn al., 2017b; Giusti al., 2016; Sun et al., 2017; Zhang et al., 2017: Lynch al., 2019). While demonstrations can also be used to speed up RL (Schaal, 1996; Peters & Schaal, 2006; Ko- rmushev et al., 2010; Hester et al., 2017: Vecerík et al., 2017: Nair et al., 2018; Rajeswaran et al., 2018; Silver et al., 2018; Peng et al., 2018; Johannink et al., 2019: Gupta al., 2019), this usually requires collecting demonstrations for the specific task that is being learned. In contrast, we use data from wide range of other prior tasks to speed up RL for new task. As show in experiments, PARROT is better suited to this problem setting when compared to prior methods that combine imitation and RI. for the same task.
2

---

Published as a conference paper at ICLR 2021
Generative modeling and RL. Several prior works model multi-modal action distributions using expert trajectories from different tasks. IntentionGAN (Hausman et al.) and InfoGAIL (Li et al., 2017) learn multi-modal policies via interaction with an environment using adversarial imitation approach (Ho & Ermon, 2016), but we learn these distributions only from data. Other works learn these distributions from data (Xie et al., 2019; Rhinehart et al., 2020) and utilize them for planning at test time to optimize a user- user-provided cost function. In contrast, we use the behavioral prior to augment model-free RL of new task. This allows us to learn policies for new tasks that may be substantially different from prior tasks, since we can collect data specific to the new task at hand, and we do not explicitly need model the environment, which can be complicated for nigh-dimensional state and action spaces, such when performing continuous control from images observations. Another line of work (Ghadirzadeh et al., 2017; Hämäläinen et al., 2019; Ghadirzadeh et al., 2020) explores using generative models for RI., using a variational autoencoder (Kingma & Welling, 2014) to model entire trajectories in an observation irraatiooninddeedeeett manner, and then learning an open-loop. single-step policy using RL to solve the downstream task. Our approach differs in several key aspects: (1) our model is observation-conditoon to the current scene or environment, (2) model allows for closed-loop feedback control, and (3) our model is invertible, allowing the high-level policy to retain full control over the action space. Our experiments demonstrate these aspects are crucial for solving harder tasks.
allowing it to prioritize actions that are relevant
Hierarchical learning. Our method can be interpreted as training a hierarchical model: the low- level policy is the behavioral prior trained on prior data, while the high-level policy is trained us- ing RL and controls the low-level policy. This structure is similar to prior work in hicrarchical RL (Dayan & Hinton, 1992; Parr & Russell, 1997; Dietterich, 1998; Sutton al., 1999; Kulka- mi et al., 2016). We divide prior work in hierarchical learning into two categories: methods that seek to learn both the low- and high-level policies through active interaction with an environ- ment (Kupesik et al., 2013; Heess et al., 2016; Bacon et al., 2017: Florensa et al., 2017: Haarnoja et al., 2018a; Nachum et al., 2018: Chandak et al., 2019; Peng et al., 2019), and methods that learn temporally extended actions. also known options, from demonstrations, and then recompose them to perform long-horizer horizon tasks through RL or planning (Fox et al., 2017: Krishnan et al., 2017; Kipf et al., 2019; Shankar et al., 2020; Shankar & Gupta, 2020). Our work shares similarities with the data-driven approach of the latter methods, but work on options focuses on modeling the temporal structure in demonstrations for a small number of long-horizon tasks, while our behavioral prior is not concerned with temporally-extended abstractions, but rather with transforming the original MDP into one where potentially useful behaviors are more likely, and useless behaviors are less likely.
Meta-lcarning, Our goal in this paper is to utilize data from previously seen tasks to speed up RL for new tasks. Meta-RL (Duan et al., 2016; Wang et al., 2016; Finn et al.. 2017a; Mishra et al., 2017: Rakelly et al., 2019; Mendonca et al., 2019; Zintgraf et al., 2020; Fakoor et al., 2020) and meta-imitation methods (Duan et al., 2017: Finn et al., 2017c; Huang et al., 2018; James et al., 2018; Paine et al., 2018; Yu et al., 2018; Huang et al., 2019 Zhou et al., 2020) also seek to speed up learning for new tasks by leveraging experience from previously seen tasks. While meta-learning provides an appealing and principled framework to accelerate acquisition of future tasks, we focus on more lightweight approach with relaxed assumptions that make our method more practically applicable, and we discuss these assumptions in detail in the next section.
3 PROBLEM SETUP
Our goal is to improve an agent's ability to learn new tasks by incorporating a behavioral prior, which it can acquire from previously seen tasks. Each task can be considered a Markov decision process (MDP). which is defined by a tuple (S, T, r, r),,, where S and AA represent state and action spaces, T(s' s, a) and r(s, a) represent the dynamics and reward functions, and y E (0, 1) represents the discount factor. Let p(M) denote a distribution over such MDPs, with the constraint that the state and action spaces are fixed. In our experiments, we treat high-dimension which means that this constraint is not very restrictive in practice. In order for the behavioral prior to be able to accelerate the acquisition of new skills, we assume the behavioral prior is trained on data that structurally resembles potential optimal policies for all or part of the new task. For example, if the new task requires placing a bottle in a tray, the prior data might include some behaviors that involve picking up objects. There are many ways to formalize this assumption. One way to state this formally is to assume that prior data consists of executions of near-optimal policies for MDPs
images as s,
3

---

Published as a conference paper at ICLR 2021
Transformed MDP
Behavioral Prior
Data from many tasks
f(o
Image observation
Comvolutinal Neural Network
m[=|s]
$
N(
RL Policy
Original
Figure 2: PARROT. Using successful trials from a large variety of tasks, we learn an invertible mapping fo that maps noise to useful actions This mapping is conditioned on the current observation, which in our case is RGB image. The image is passed through stack of convolutional layers and flattened to obtain an image encoding (s)). and this image encoding is then used to condition each individual transformation fa of our overall mapping function fe. The parameters of the mapping (including the convolutional encoder) are learned through maximizing the conditional log-likellhond of state-action pairs observed in the dataset. When learning new task, this mapping randomly initialized policy to actions that are likely to lead to useful hchavior in the current scene. Since the mapping is invertible, the RI. agent still retains full control over the action space of original MDP, simply the likelihood of executing a useful action is increased through use of the pre-trained mapping.
simplify the MDP for an RI. agent by mapping actions sampled from
drawn according to M p(M), and the new task M is likewise drawn from p(M). In this case, the generative process for the prior data can be expressed as:
M p(M), TM (T) arg max Ex, M [Ra], TM (T)(),
(1)
where TM = (81, 01; $2, 02 .2...... ................................ ST, aT) is sequence of state and actions, (T) denotes near- optimal policy (Kearns & Singh, 2002) for MDP M and Rı = Eieo 7'rz. When incorporating the behavioral prior for learning new task M our goal is the same standard RL: find a policy that maximizes the expected return arg max, E.., R. Our assumption on tasks being drawn from distribution p(M) shares similarities with the meta-RL problem (Wang et al., 2016; Duan et al., 2016), but our setup is different: it does not require accessing any task in p( M) except the new task we are learning, M Meta-RI. methods need to interact with the tasks in p (M) during meta-training, with access to rewards and additional samples, whereas we learn our behavioral prior simply from data, without even requiring this data to be labeled with rewards. This of particular importance for real-world problem settings such robotics: it is much easier to store data from prior tasks (e.g., different environments) than to have a robot physically revisit those prior settings and retry those tasks, and requiring known rewards makes it possible use data from a variety of sources, including human-provided demonstrations. In our setting, RL is performed in only one environment, while the prior data come from many environments.
Our setting related to meta-iminne ietaa-mitaaion learning (Duan et al., 2017; Finn et al., 2017c), as we speed up learning new tasks using data collected from past tasks. However, meta-imitation learning methods require at least one demonstration for each new task, whereas our method can learn new tasks without any demonstrations. Further, our data requirements are less stringent: meta-imitation learning methods require all demonstrations to be optimal, require all trajectories in the dataset to have a task label, and requires "paired demonstrations". i.e. at least two demonstrations for each task (since meta-imitation methods maximize the likelihood of actions from one demonstration after conditioning the policy on another demonstration from the same task). Relaxing these requirements increases the scalability of method: we can incorporate data from wider range of sources, and we do not need to explicitly organize it into specific tasks
4 BEHAVIORAL PRIORS FOR REINFORCEMENT LEARNING
Our method learns a behavioral prior for downstream RL by utilizing a dataset D of (near-optimal) state-action pairs from previously seen tasks. We do so by learning a state-conditioned mapping fø: Z x + A (where ó denotes learnable parameters) that transforms a noise vector 2 into an action that is likely to be useful in the current state This removes the need for exploring via "meaningless" random behavior, and instead enables an exploration process where the agent attempts behaviors that have been shown to be useful in previously seen domains. For example, if
4

---

Published as a conference paper at ICLR 2021
robotic arm is placed in front of several objects. randomly sampling 2 (from simple distribution, such as the unit Gaussian) and applying the mapping a = fo(z:s) should result in actions that, when executed, result in meaningful interactions with the objects. This learned mapping essentially transforms the MDP experienced by the RL agent into a simpler one, where every random action executed in this transformed MDP is much more likely to lead to useful behavior.
How can we learn such a mapping? In this paper, we propose to learn this mapping through state- conditioned generative modeling of the actions observed in the original dataset DD and we refer to this state-conditioned distribution over actions as the behavioral prior (prior(als), A deep generative model takes noise as input, and outputs a plausible sample from the target distribution, i.c. it can represent Pptior (a]s) as a distribution over noise using a deterministic mapping fo Z x S + A When learning new task, we can use this mapping to reparametrize the action space of the RI. agent: if the action chosen by the randomly initialized neural network policy is then we execute the action a fo [2; s) in the original MDP, and learn a policy I (2|s) that maximizes the task reward through learning to control the inputs to the mapping fø. The training of the behavioral prior and the task-specific policy is decoupled, allowing us to mix and match RL algorithms and generative models to best suit the application of interest. An overview of our overall architecture is depicted in Figure 2. In the next subsection, we discuss what properties we would like the behavioral prior to satisfy, and present one particular choice for learning prior that satisfies all of these properties.
4.1 LEARNING A BEIIAVIORAL PRIOR WITH NORMALIZING FLOWS
For the behavioral prior to be effective, it needs to satisfy certain properties. Since we learn the prior from a multi-task dataset, containing several different behaviors even for the same initial state, the learned prior should be capable of representing complex, multi-modal distributions. Second, should provide a mapping for generating "useful" actions from noise samples when learning a new task. Third, the prior should be statecconditioned, so that only actions that are relevant to the current state are sampled. And finally, the learned mapping should allow easier learning in the reparameterized action space without hindering the RL agent's 's ability to attempt novel behaviors, including actions that might not have been observed in the dataset DD Generative models based on normalizing flows (Dinh et al., 2017) satisfy all of these properties well: they allow maximizing the model's exact log-likelihood of observed examples, and learn a deterministic, invertible mapping that transforms samples from a simple distribution pz to examples observed in the training dataset. In particular, the realev non-volume preserving (real NVP) architecture introduced by Dinh et al. (2017) allows using deep neural networks to parameterize this mapping (making it expressive) While the original real NVP work modelled unconditional distributions, follow-up up work has found that it can be easily extended to incorporate conditioning information (Ardizzone et al., 2019). We refer the reader to prior work (Dinh et al., 2017) for a complete description of real NVPs, and summarize its key features here. Given an invertible mapping a = fo (z; s). the change of variable formula allows expressing the likelihood of the observed actions using samples from D in the following way:
Perior (als) p= (f
(a;:)) |det (o:))ao))))
(2)
Dinh et al. (2017) propose a particular (unconditioned) form of the invertible mapping fa. called an affine coupling layer, that maintains tractability of the likelihood term above, while still allowing the mapping fo to be expressive. Several coupling layers can be composed together to transform simple noise vectors into samples from complex distributions, and each layer can be conditioned on other variables, as shown in Figure 2.
4.2 ACCELERATED REINFORCEMENT LEARNING VIA BEHAVIORAT. PRIORS
After we obtain the mapping (2; s) from the behavioral prior learned by maximizing the likelihood term in Equation 2, we would like to use it to accelerate RL. when solving a new task. Instead of learning a policy To (alss) that directly executes its actions in the original MDP, we learn policy To(2[s), and execute an action in the environment according to a = fz(z; s). As shown in Figure 2, this essentially transforms the MDP experienced for the RL agent into one where random actions 2 ~ P3 (where P= is the base distribution used for training the mapping fe) are much more likely to result in useful behaviors. To enable effective exploration at the start of the learning period, we initialize the RL policy to the base distribution used for training the prior, so that at the beginning of training, T=(2|)) := p= (2). Since the mapping S& is invertible, the RI agent still retains full control
5

---

Published as a conference paper at ICLR 2021
(a; over the action space: for any given a, it can always find 2 that generates fz fa; s) in the original MDP. The learned mapping increases the likelihood of useful actions without crippling the RL agent, making it ideal for fine-tuning from task-specific data. Our complete method is described in Algorithm I in Appendix A. Note that we need to learn the mapping fo only once, and it can be used for accelerated learning of any new task.
4.3 IMPLEMENTATION DETAILS
We use real NVP to learn fe, and as shown in Figure 2, each coupling layer in the real NVP takes as input the the output of the previous coupling layer, and the conditioning information. The con- ditioning information in our case corresponds to RGB image observations, which allows us to train single behavioral prior across wide variety of tasks, even when the tasks might have different underlying states (for example, different objects). We train a real NVP model with four coupling layers; the exact architecture, and other hyperparameters, are detailed in Appendix B. The behav- ioral prior can be combined with any RL algorithm that is suitable for continuous action spaces, and we chose to use the soft actor-critic (Haarnoja al., 2018b) due to its stability and ease of use.
5 EXPERIMENTS
Our experiments seek to answer: (1) Can the behavioral prior accelerate learning of new tasks? (2) How does PARROT compare to prior works that accelerate RL with demonstrations? (3) How does PARROT compare to prior methods that combine hierarchical imitation with RL?
Domains. We evaluate our method on suite of challenging robotic manipulation tasks, a subset of which are depicted in Figure 3. Each task involves controlling 6-DoF robotic arm and its gripper, with 7D action space. The observation is a 48 48 RGB image, which allows servation representation across tasks, even though each underlying task might have different under- lying state (e.g., different objects). No other obser- vations (such as joint angles or end-effector posi-
use the same ob-
tions) are provided. In each task, the robot needs
interact with one two objects in the scene to achieve its objective, and there are three objects in each scene. Note that all of the objects in the test scenes are novel - the dataset DD contains no inter- actions with these objects. The object positions at the start of each trial are randomized, and the policy must infer these positions from image observations in order to successfully solve the task. A reward of +1 is provided when the objective for the task is achieved, and the reward is zero otherwise. De- tailed information on the objective for each task and example rollouts provided in Appendix C. and on our anonymous project website'.
Figure 3: Tasks. A subset of our evaluation tasks, with one task shown in each row. In the first task (first row), the objective is to pick up can and place in the pan. In the second task, the robot must pick up the vase and put it in the basket. In the third task, the goal is to place the chair on top of the checkerboard. In the fourth task, the robot must pick up the mug and hold it above certain height. Initial positions of all objects are randomized, and must be inferred from visual observations. Not all objects in the scene are relevant to the current task.
Data collection. Our behavioral prior is trained on a diverse dataset of trajectories from a wide range of tasks, and then utilized to accelerate reinforcement learning of new tasks. As discussed in Sec- tion 3, for the prior to be effective, it needs to be trained on dataset that structurally resembles the behaviors that might optimal for the new tasks. our case, all of the behaviors involve reposi- tioning objects (i.e., picking up objects and moving them to new locations), which represents very general class of tasks that might be performed by a robotic arm. While the dataset can be collected in many ways, such as from human demonstrations prior tasks solved by the robot, collect it using a set of randomized scripted policies, see Appendix C.2 for details. Since the policies are ran-
hhttps:/siit tes. wowg.goooll......oo.oooo lel www.c///wwwe///..ttto.

---

Published as a conference paper at ICLR 2021
domized, not every execution of such a policy results in useful behavior, and we decide to keep or discard a collected trajectory based on a simple predefined rule: if the trajectory collected ends with successful grasp or rearrangement of any one of the objects in the scene, we add this trajectory to our dataset. We collect dataset of 50K trajectories, where each trajectory is of length 25 timesteps (R:5-6 seconds), for total of 1. 1.25m observation-action pairs. The observation is 48 x 48 RGB image, while the actions are continuous 7D vectors. Data collection involves interactions with over 50 everyday objects (see Appendix C.3); the diversity of this dataset enables learning priors that can produce useful behavior when interacting with a new object.
5.1 RESULTS, COMPARISONS AND ANALYSIS
To answer the questions posed at the start of this section, we compare PARROT against a of prior works, as well as ablations of our method. Additional implementation details and hyperpa- rameters can be found in Appendix B.
number
Soft-Actor Critic (SAC), For a basic RL comparison, we compare against the vanilla soft-actor critic algorithm (Haarnoja et al., 201 (0),,,, which does not incorporate any previously collected data.
SAC with demonstrations (BC-SAC). We compare against a method that incorporates demonstra- tions to speed up learning of new tasks. In particular, we initialize the SAC policy by performing behavioral cloning on the entire dataset D, and then fine-tune using SAC. This approach similar to what has been used in prior work (Rajeswaran et al., 2018), 8), except we use SAC as our RL algo- rithm. While we did lest other methods that are designed to use demonstration data with RL such as DDPGfD (Vecerík et al., 2017) and AWAC (Nair et al., 2020), we found that our simple BC SAC variant performed better. This somewhat contradicts the results reported in prior work (Nair et al., 2020), but believe that this is because the prior data is not labeled with rewards (all transitions are assigned a reward of O), and more powerful demonstration RL methods require access to these rewards, and subsequently struggle due to the reward misspecification.
Transfer Learning via Feature Learning (VAE-features). We compare against prior methods for transfer learning (in RL) that involve learning robust representation of the input observation. Similar to Higgins et al. (2017b), we train a 8- V-E using the observations in our training set, and train a policy on top of the features learned by this VAE when learning downstream tasks.
Trajectory modeling and RL (TrajRL.). Ghadirzadeh et al. (2020) model entire trajectories using VAE, and learn a one-step policy on top of the VAE to solve tasks using RL. Our implementation of this method uses VAE architecture identical to the original paper's, s. and we then train a policy using SAC solve new tasks with the action space induced by the VAE. We performed additional hyperparameter tuning for this comparison, the details of which can be found in Appendix B.
Hierarchical imitation and RL (HIRL). Prior works in hierarchical imitation learning (Fox et al., 2017; Shankar Gupta, 2020) train latent variable models over expert demonstrations to discover op- tions, and later utilize these options to learn long- horizon tasks using RL. While PARROT can also be extended to model the temporal structure in trajecto- ries through conditioning
past states and actions, by modeling Ppeior (42, 18r, sy.., Ot-1 .... 6o) in- focus on a simpler version of the model in this paper that does not condition on the past. In order provide fair comparison, we modify the model proposed by Shankar & Gupta (2020) to remove the past conditioning, which then reduces to training a conditional VAE, and perform- ing RL on the action space induced by the latent space of this VAE. This comparison is similar to our proposed approach, but with one crucial difference: the mapping we learn is invertible, and allows the RL agent to retain full control over the final actions in the environment (since for every a E A, there exist some
stead of Pprior(Ge] (Grlsz).
Without Behavioral Prior with Behavioral Prior
We plot trajectories from executing random policy, with and without the behavioral prior. We see that the behavioral prior substan- tially increases the likelihood of executing an ac- tion that is likely to lead meaningful interac- tion with an object, while still exploring a diverse set of actions.
Figure 4:
f (a; 8), while latent space learned by VAE provides no such guarantee).
7

---

Published as a conference paper at ICLR 2021
Place Can
Place Sculpture Basket
place Chair Checkerboard Table
Baseball Black
1.4
Soccoss Rato
Success Rate
04-
3.2
6.c
6.-
<boc
Jblc
Timestegs
Timestegs
Timesteps
Timesleps
Pick up Bar
Pick Up Soulpture
Pick up Cup
Pick up Baseball Cap
Succoss Rate
Success Rate
Success Rate
Rate
JBSC
5404
1004
25. Timestops
2006 <0oc
06
2004
403c
Seat
22.8 Timestaps
Timesteps
Timesteps
PARROT
BC+SAC
TrajRI
VAE- VAE-features
Prior-expibre
Figure 5: Results. The lines represent average performance across multiple random seeds, and the shaded areas represent the standard deviation. PARROT able to leam much faster than prior methods on a majority of the tasks, and shows little variance across runs (all experiments were run with three random seeds, compu- tational constraints of image-based RL make it difficult to run more seeds). Note that some methods that failed make any progress certain tasks (such as "Place Sculpture in Basket") overlap each other with success of zero. SAC and VAF-fcatures fail make progress on any of the tasks.
Exploration via behavioral prior (Prior-explore). We also experiments with ablation of our method: instead of using a behavioral prior to transform the MDP being experienced by the RL agent, we use it to simply aid the exploration process. While collecting data, an action is executed from the prior with probability e, else an action is executed from the learned policy. We experimented with e 0.1, 0.3, 0.7, 0.9, and found 0.9 to perform best.
Main results. Our results are summarised in Figure 5. We see that PARROT is able to solve all of the tasks substantially faster and achieve substantially higher final returns than other methods. The SAC baseline (which does not use any prior data) fails to make progress on any of the tasks, which we suspect is due to the challenge of exploring in sparse reward settings with a randomly initialized policy. Figure 4 illustrates a comparison between using behavioral prior and a random policy for exploration. The VAE-features baseline similarly fails to make any progress, and due to the same reason: the difficulty of exploration in sparse reward setting. Initializing the SAC policy with behavior cloning allows it to make progress on only two of the tasks, which is not surprising: Gaussian policy learned through behavior cloning loss is not expressive enough represent the complex, multi-modal action distributions observed in dataset D. Both TrajRL and HIRL perform much better than any of the other baselines, but their performance plateaus lot earlier than PAR- ROT. While the initial exploration performance of our learned behavioral prior is not substantially better from these methods (denoted by the initial success rate in the learning curves), the flexibility of the representation it offers (through learning an invertible mapping) allows the RL agent to im- prove far beyond its initial performance. Prior-explore, an ablation of our method, is able make progress on most tasks, but is unable to learn as fast as our method, and also demonstrates unstable learning on some of the tasks. We suspect this is due to the following reason: while off-policy RI. methods like SAC aim to learn from data collected by any policy, they are in practice quite sensitive to the data distribution, and can run into issues the data collection policy differs substantially from the policy being learned (Kumar et al., 2019).
Impact of dataset size on performance. We conducted additional experiments on a subset of tasks to evaluate how final performance is impacted result of dataset size, results from which are shown in Figure As one might expect, the size of the dataset positively correlates with performance, but about 10K trajectories sufficient for obtaining good performance, and
8

---

Published as a conference paper at ICLR 2021
up Cup
Pick Baseball
Pick Sculpture
2.-
1.0
0.9
Success Rate
Success Rate:
Success Rate
0.4-
0.
<<
1008
2016 vecc
visc
199k
2004
wid
1956
2006 Timastops
123r
cree
seer
1006
20.0
1996
199
Timestops
Timesteps
Timestops
SK
10K
25K
50K
Figure 6: Impact of dataset size on performance. We observe that training 10K, 25K or 50K trajectories yields similar performance.
collecting additional data yields diminishing returns. Note that initializing with even a smaller dataset size (like 5K trajectories) yields much better performance than learning from scratch.
Mismatch between train and test tasks. We experiments in which we deliberately bias the training dataset so that the training tasks and test tasks are functionally different (i.e. involve substantially different actions), the results from which are shown in Figure We observe that if the prior is trained on pick and place tasks alone, it can still solve downstream grasping tasks well. However, if the prior is trained only unable to perform well when solving pick and place tasks. We suspect this is due to the fact that pick and place tasks in- volve completely new action (that of open- ing the gripper), which is never observed by the prior if it is trained only grasping, making it difficult to learn this behavior from scratch for downstream tasks.
Prior grasping
Success Bore
2004
2008
2004
on grasping,
POLLB Soidhal
Plat Stalportt
Flci Chae
Figure 7: Impact train/test mismatch on perfor- mance. Each plot shows results for four tasks. Note that for the pick and place tasks, the performance is close zero, and the curves mostly overlap each other the x-axis.
CONCLUSION
We presented PARROT, method for learning behavioral priors using successful trials from wide range of tasks. Learning from priors accelerates RL on new incksiin manipulating previously unseen objects from high-dimensional image observations-which RL from scratch often fails to learn. Our method also compares favorably to other prior works that use prior data to bootstrap learning for new tasks. While our method learns faster and performs better than prior work in learning novel tasks, it still requires thousands of trials to attain high success rates. Improving this efficiency even further, perhaps inspired by ideas meta-learning, could be a promising direction for future work. Our work opens the possibility for several exciting future directions. PARROT provides a mapping for executing actions in new environments structurally similar to those of prior tasks. While primarily utilized this mapping accelerate learning of new tasks, future work could investigate how it can also enable safe exploration of new environments (Hunt et al., 2020; Rhinehart et al., 2020). While the invertibility of our learned mapping ensures that it is theoretically possible for the RL policy to execute any action in the original MDP, the probability of executing an action can become very low if this action was never seen in the training set. This can be issue if there is a significant mismatch between the training dataset and the downstream task (as shown in our experiments), and tackling this issue would make for an interesting problem. Since our method speeds up learning using problem setup that takes into account real world considerations (no rewards for prior data, no need to revisit prior tasks, etc.), we are also excited about its future application to domains like real world robotics.

---

Published as a conference paper at ICLR 2021
REFERENCES
Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. In ICLR, 2019.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. AAAI, 2017.
Yash Chandak, Georgios Theocharous, James Kostas, Scott M. Jordan, and Philip S. Thomas. Learn- ing action representations for reinforcement learning. In ICML, 2019.
Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. CoRR, abs/ 0bb//1512.03011 2015.
E Coumans and Y Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. GitHub repository, 2016.
Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In NIPS, 1992.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Thomas G. Dietterich. The MAXQ method for hierarchical reinforcement learning. In ICMI., 1998.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In ICLR, 2017.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.02779, 2016.
Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Neural Information Process- ing Systems (NIPS), 2017.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-learning. In ICLR, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint ar.iv::703..m 000000000000.. 2017a
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (ICML), 2017b.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imita- tion learning via meta-learning. Conference Robot Learning (CoRL), 2017c.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. In ICLR, 2017.
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. CoRR, 011117033.99999 2017.
Ali Ghadirzadeh, Atsuto Maki, Danica Kragic, and Märten Björkman. Deep predictive policy train- ing using reinforcement learning. In International Conference on Intelligent Robots and Systems, 2017.
Ali Ghadirzadeh, Petra Poklukar, Ville Kyrki, Danica Kragic, and Märten Björkman. Data- efficient visuomotor policy training using reinforcement learning and generative models. CoRR, 13332007.13134, 2020.
Alessandro Giusti, Jérôme Guzzi, Dan C Ciresan, Fang-Lin He, Juan Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, et al. A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters (RA-L)), 2016.
10

---

Published as a conference paper at ICLR 2021
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving ong-horizon tasks via imitation and reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Conference on Robot Learning. 2019.
Tuomas Haarojja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference Machine Learning (ICML), 2017.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), ICMI., 201 8a.
Tuomas Hаarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Vikash Kumar Jie Tan, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. Technical report, 2018b.
Aleksi Hämäläinen, Karol Arndt, Ali Ghadirzadeh, and Ville Kyrki. Affordance learning for end- to-end visuomotor robot control. In IROS, 2019.
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav S. Sukhatme, and Joseph Lim. Multi- modal imitation learning from unstructured demonstrations using generative adversarial nets. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), NIPS.
Nicolas Heess, Gregory Wayne, Yuval Tassa, Timothy P. Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. CoRR, abs/1610.051122 2016.
Todd Hester, Matej Vecerík, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Amold, Ian Osband, John P. Agapiou, Joel Z. Leibo, and Au- drunas Gruslys. Learning from demonstrations for real world reinforcement learning. CoRR, abs/1704.03332, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017a.
Irina Higgins, Arka Pal, Andrei A. Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: improving zero-shot transfer in reinforcement learning. In ICML, 2017b.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances Neural Information Processing Systems (NIPS), 2016.
De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, and Juan Carlos Niebles. Neural task graphs: Generalizing to unseen tasks from a single video demon- stration. 2018
De-An Huang, Danfei Xu, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei, and Juan Carlos Niebles. Continuous relaxation of symbolic planner for one-shot imitation learning. In IROS, 2019.
Nathan Hunt, Nathan Fulton, Sara Magliacane, Nghia Hoang. Subhro Das, and Armando Solar-Lezama. Verifiably safe exploration for end-to-end reinforcement learning. CoRR, abs/2007.01223, 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, 2015.
Stephen James, Michael Bloesch, and Andrew J Davison. Task-embedded control networks for few-shot imitation learning. arXiv preprint arXiv:1810.032221 2018.
Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea. Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control. In ICRA, 2019.
11

---

Published as a conference paper at ICLR 2021
Michael J. Kearns and Satinder P. Singh. Near-optimal reinforcement learning in polynomial time. Machine Learning, 49(2-3):209 232, 2002.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations (ICLR), 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and
Yann LeCun (eds.), ICLR, 2014.
Thomas Kipf, Yujia Li, Hanjun Dai, Vinícius Flores Zambaldi, Alvaro Sanzhez-Gonzalez, Edward Grefenstette, Pushmeet Kohli, and Peter W. Battaglia. Compile: Compositional imitation learning and execution. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), ICML, 2019.
Petar Kormushev, Sylvain Calinon, and Darwin G. Caldwell. Robot motor skill coordination with em-based reinforcement learning. IROS, 2010.
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. DDCO: discovery of deep continuous options for robot learning from demonstrations. In CoRL, 2017.
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Sacedi, and Josh Tenenbaum. Hicrarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in Neural Information Processing Systems, 2016.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. NeurlPS, 2019.
Andras Gabor Kupesik, Marc Peter Deisenroth, Jan Peters, and Gerhard Neumann. Data-efficient generalization of robot skills with contextual policy search. In AAAI, 2013.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems, 2017.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and
Pierre Sermanet. Learning latent plans from play. In Conference Robot Learning, 2019.
Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. arXiv preprint
2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. Meta-learning with temporal convolutions. 17Xiv:1707.03141 2017.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances Neural Information Processing Systems, 2018.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. ICRA, 2018
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. CoRR, abs/2006.0933 09955, 2020.
Tom Le Paine, Sergio Gómez Colmenarejo, Ziyu Wang, Scott Reed, Yusuf Aytar, Tobias Pfaff, Matt W Hoffman, Gabriel Barth-Maron Serkan Cabi, David Budden, et One-shot high-fidelity imitation: Training large-scale deep nets with rl. arXiv preprint arXiv:1810.05017, 2018.
Ronald Parr and Stuart J. Russell. Reinforcement learning with hierarchies of machines. In Advances in Neural Information Processing Systems, 1997.
Peter Pastor, Heiko Hoffmann, Tamim Asfour, and Stefan Schaal. Learning and generalization of motor skills by learning from demonstration. In International Conference Robotics and Automation (ICRA). 2009.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 2018.
12

---

Published as a conference paper at ICLR 2021
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. MCP: learning composable hierarchical control with multiplicative compositional policies. In NeurlPS, 2019.
Jan Peters and Stefan Schaal. Policy gradient methods for robotics. In IROS, 2006.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Neural Information Processing Systems (NIPS), pp. 305-313, 1989.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy meta-reinnfrrcement learning via probabilistic context variables. In ICML, 2019.
Nathan Ratliff, Andrew Bagnell, and Siddhartha $ Srinivasa. Imitation learning for locomotion and manipulation. In International Conference Humanoid Robots, 2007.
Nicholas Rhinchart, Rowan McAllister, and Sergey Levine. Deep imitative models for flexible inference, planning, and control. In ICLR, 2020.
Stefan Schaal. Learning from demonstration. In Michael Mozer, Michael I. Jordan, and Thomas Petsche (eds.), NIPS, 1996.
Stefan Schaal, Auke Ijspeert, and Aude Billard. Computational approaches motor learning by imitation. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 2003.
Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. 2020.
Tanmay Shankar, Shubham Tulsiani, Lerrel Pinto, and Abhinav Gupta. Discovering motor programs by recomposing demonstrations. In ICLR, 2020.
Tom Silver, Kelsey R. Allen, Josh Tenenbaum, and Leslie Pack Kaelbling. Residual policy learning. CoRR, abs/ 181211881....,,,, 2018.
Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In Proceedings of the 34th International Conference Machine Learning-Volunee Volume 70, pp. 3309-3318. JMLR. org, 2017.
Richard S. Sutton, Doina Precup, and Satinder P. Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial Intelligence, 1999.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In NeurIPS, 2017.
Matej Vecerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin A. Riedmiller. Leveraging demon- strations for deep reinforcement learning on robotics problems with sparse rewards. CoRR, 2017.
Jane Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv.1611.0566. 2016.
Annie Xie, Frederik Ebert, Sergey Levine, and Chelsea Finn. Improvisation through physical under- standing: Using novel objects tools with visual foresight. In Robotics: Science and Systems, 2019.
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain domain-adaptive meta-learning. Robotics: Science and Systems (RSS), 2018
13

---

Published as a conference paper at ICLR 2021
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Ken Goldberg, and Pieter Abbeel. Deep im- itation learning for complex manipulation tasks from virtual reality teleoperation. arXiv preprint arXiv:1710.04615, 2017.
Allan Zhou, Eric Jang, Daniel Kappler, Alexander Herzog, Mohi Khansari, Paul Wohlhart, Yunfei Bai, Mrinal Kalakrishnan, Sergey Levine, and Chelsea Finn. Watch, try, learn: Meta-learning from demonstrations and reward. 2020.
Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep RL via meta- learning. ICLR, 2020.
14

---

Published as a conference paper at ICLR 2021
Appendices
A ALGORITHM
Algorithm RL with Behavioral Priors
I: Input: Dataset D of stae-acction pairs (s, a) from previous tasks, new task M
2: Learn fo by maximizing the likelihood term in Equation 2
3: for step & [I,...;N, NJ do
4 current observation
5: Sample 2 T=(2)))
6:
7: s'.r Execute in M
Update m=(=))) with 5/,,,)))))
9: for
10: Return: Policy (2(s) for task M"
B IMPLEMENTATION DETAILS AND HYPERPARAMETER TUNING
We now provide details of the neural network architectures and other hyperparameters used in our experiments.
Behavioral prior. We use conditional real NVP with four affine coupling layers our behavioral prior. The architecture for single coupling layer shown in Figure 8. We use learning rate of 1e-4 and the Adam (Kingma & Ba, 2015) optimizer to train the behavioral prior for 500K steps.
Adeivagee
soe
2-l con
Satan
Ereer
Bnear
Inear
1236
2001
21
(
(b)
(()))())))
2d
2dt+1
200
256.0
(a)
Figure 8: Coupling layer architecture. A computation graph for a single affine coupling layer is shown in (a). Given input noise the coupling layers transform into through the following operations: æí.d 21d and 2'21.DD 2d+1.D exp(v(#1.0) ((s)) - f(21.4; (())), where the t and are functions implemented using neural networks whose architectures are shown in (b) and (c). Since and have the same input, they are implemented using a single fully connected neural network (shown in (b)), and the output this network split into two. The image encoder, (i(s) implemented using a convolutional neural network with parameters shown in (c).
TrajVAE. For this comparison, we use the same architecture as Ghadirzadeh et al. (2020). The decoder consists of three fully connected layers with 128, 256, and 512 units respectively. Batch- Norm (loffe & Szegedy, 2015) and ReLU nonlinearity are applied after each layer. The encoder is symmetric: 512, 256, and 128 layers, respectively. The size of the latent space is 8 (same as the behavioral prior). We sweep the following values for the parameter (Higgins 0.01, 0.005, 0.001, 0.0005, and find 0.001 to be optimal. We initialize 3 to zero at the start of train- ing, and anneal it to the target value using logistic function, achieving half of the target value in
al., 2017a): 0.1,
15

---

Published as a conference paper at ICLR 2021
25K steps. We use learning rate of 1e-4 and the Adam (Kingma & Ba, 2015) optimizer train this model for 500K steps.
HIRL. This comparison is implemented using conditional variational autoencoder, and uses architecture that is similar to the one used by the TrajVAE, but with two differences: since this comparison uses image conditioning, we use the same convolutional network e the behavioral prior to encode the image (shown Figure 8), and pass as conditioning information to both the encoder and decoder networks. Second, instead of modeling the entire trajectory single forward pass, it instead models individual actions, allowing the high-level policy to perform closed-loop control, similar the behavioral prior model. We sweep the following values for the / parameter: 0.1, 0.01, 0.005, 0,001, 0.0005, and find 0.001 to be optimal. We found the annealing process to be essential for obtaining good RI performance using this method. We use a learning rate of 4e-4 and the Adam (Kingma & Ba, 2015) optimizer to train this model for 500K steps.
Behavior cloning (BC). We implement behavior cloning via maximum likelihood with a Gaussian policy (and entropy regularization (Haarnoja et al., 2017)). For both behavior cloning and RL with SAC, we used the same policy network architecture as shown Figure We train this model for 2M steps, using Adam with learning rate of 3e-
VAE-features. For this comparison, we use standard VAE architecture used for CIFAR-10 experiments (van den Oord et al., 2017). The encoder consists of two strided convolutional layers (stride 2, window size x 4), which is followed by two residual 3 3 blocks, all of which have 256 hidden units. Each residual block is implemented ReLU, 3x3 conv, ReLU, 1x1 conv. The decoder symmetric to the encoder. We train this model for 1.5M steps, using Adam with a learning rate of 1e-8 and a batch size of 128.
Soft Actor Critic (SAC). We use the soft actor critic method (Haarnoja et al., 2018b) as our RI. algorithm, with the hyperparameters shown in Table 1. We the same hyperparameters for all of our RL experiments (our method, HIRL, TrajRL, BC+SAC, SAC).
Table I: Hyperparameters for soft-actor critic (SAC)
Hyperparameter
value used
Target network update period discount factor ?
1000 steps
0.99
policy learning rate Q-function learning rate reward scale automatic entropy tuning number of update steps per env step
3e
3e
1.0
enabled
madd
Keds
Dtait
Reil
RW
(2)1
512
(16. 51
118 26 2K
178 26 24
2305
Figure 9: Policy and Q-function network architectures. use a convolutional neural network to represent the Q-function for SAC,, shown in this figure. The policy network is identical, except it does not take an action as an input and outputs 7D action instead of scalar Q-value.
16

---

Published as a conference paper at ICLR 2021
C EXPERIMENTAL SETUP
C.I TASKS
We provided visual depiction of 4 of our 8 evaluations tasks in Figure and the remaining tasks are shown here in Figure 10.
Figure 10:
the first row, the objective is to grasp a can three are similar, except the objective is to grasp a vase and a baseball cap, respectively. The final row depicts a task where the goal is to pick the baseball cap and place it on the marble cube.
above certain height. Rows two and
17

---

Published as a conference paper at ICLR 2021
C.2 DATA COLLECTION
We collected our dataset using scripted policies detailed Algorithms 2 and 3.
Algorithm 2 Scripted Grasping
Algorithm 3 Scripted Pick and Place
I: threshold 0.02
1: threshold <- 0.02
2: numTimestepss 25
2: numTTimesteps +- 25
3: targetPoint + object position
placeAttempped
False
4: for in (0, numTimesteps))
4: dropPos +- point above container
5: eePos &- end effector position
5: for in (0, nommnnmmmeeee (iumsttsssps)) do
6: largetEEDist <- distance eePos)
6: eePos <- end effector position
7: targetEEDDstt threshold then
7: objectDropDist distance(eePos, dropPos)
action + targetPoint cePos
8: if placeAttempted then
9:
else if gripperOpened then
9: action t
10:
action + close gripper
10:
else if object not grasped AND objectDropDist threshold then
11:
else if object not raised high enough then
12:
action lift upward
11:
Execute grasp using Algorithm 2 else oroect > threshold then action + dropPos ccPos
13:
else
12:
14:
action + 0
13:
15:
end if
14:
else
16:
noise (0,
15:
action open gripper placeAteemteed True
action +- action + noise
16:
18:
s' + cnvisppppacai))
17:
else
19:
end for
18:
action + 0
20:
19:
end if
20:
noise (/ 0.1)
21:
action <- action noise
22:
< cnw.step(acttinn)
23:
end for
18

---

Published as a conference paper at ICLR 2021
C.3 SIMULATION OBJECTS
To collect data in diverse environments, we used 3D object models from the ShapeNet dataset (Chang al., 2015) and the PyBullet (Coumans & Bai, 2016) object libraries.
Figure 11: Train objects.
Figure 12: Test objects
19


Reviews:
[{"review_id": "_C-tNq6Iatp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a method, PARROT, to learn data-driven priors for deep reinforcement learning agents. Motivated by the idea of pre-training with existing data of similar tasks, the authors propose to learn state-conditional behavioral priors from a set of similar tasks for reinforcement learning agents, such that a learning agent explores its environment in a meaningful way. Successful trials from past tasks are used as training data to learn a mapping from pixel-level state input to actions. Experiments in simulated robotic manipulator domain demonstrate the benefit of learning behavioral priors, comparing PARROT against algorithms learning from scratch as well as agents pre-trained with behavioral cloning.\n\nPros:\nThe problem setting tackled in this paper is important for applying deep reinforcement learning algorithms on real world robotic tasks; \nTraining the behavioral prior with PARROT is fully offline, which makes it favorable for practical concerns, as existing meta learning algorithms often require online data (interacting with the environment);\nThe selection of baseline algorithms in this paper are satisfactory, showcasing the benefits of each design choices made with PARROT;\nThe paper is well-written and easy to follow in general; the related work section clearly identifies the novelty of PARROT comparing with existing literature\n\nCons:\nPARROT only conducted experiments in a simulated robotic environment, possibly due to the fact that the RL step still takes a considerable amount of data to converge; ideas from works in the imitation learning literature such as GTI by Mandlekar et al. can be leveraged for improving the overall data-efficiency of PARROT\n\n\nMandlekar, Ajay, et al. \"Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations.\" Proceedings of Robotics: Science and Systems (RSS), July 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XjRrcnvD2W6", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces PARROT, a novel approach for pretraining a reinforcement learning agent on near-optimal trajectories by learning a behavioral prior. Essentially, the authors learn a word2vec style embedding of actions for a simple virtual single-arm environment. This embedding will naturally place more common examples from its training data towards the center of the gaussian, making sampling them during training time more likely. The authors demonstrate that this approach outperforms existing pretraining methods in this domain. \n\nThis paper presents an interesting novel approach and presents strong support for its value. In particular I appreciate that the basic intuition is relatively straightforward and therefore should be relatively easy to test in new domains. In addition, the evaluation results are impressive. \n\nThe paper claims in two instances that human examples would be appropriate for the training data. However, this is not confirmed in the current evaluations to my understanding. The paper indicates that the training data should be “near optimal” multiple times but never explicitly defines what is meant by “near optimal”. Is it that actions must be “useful” (also vague)? More clarity on this would be appreciated. \n\nI have some concerns around certain claims or arguments made in the paper (more on that below). However, this paper still ticks all the boxes for me in terms of novelty and value, and so I would argue for its acceptance. \n\nSome questions for the authors: \n1. What is meant by near optimal? Can this be more formally defined?\n2. The constraint that the action dimensionalities are fixed seems like a large one. I understand that it’s necessary for the invertible requirement, but I could imagine that a similar approach to this could still be helpful in sim2real problems without this constraint. Would the authors agree?\n3. The paper domain is still relatively simple compared to real world problem domains, to what extent do you expect this approach to generalize? \n\nSome smaller points/additional feedback: \n- The introduction and abstract are a bit vague, and I didn’t grasp the approach intuition until figure 1. I’d appreciate if some of this information could be moved up into paragraph 3 of the introduction. \n-I’d replace “large such datasets” with “such large datasets” or just remove “such”\n-I recognize that theoretically that the approach should be able to represent every possible environment action, but it would seem to me still possible that good/useful actions could be placed so far from the center of the Gaussian as to make them very unlikely, depending on the training data. Some discussion about how to avoid this/whether it is a concern would be helpful. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "7gNNmIwfbx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis paper proposes PARROT, a method for learning a policy prior from a dataset of expert state-action pairs that have been derived from multiple similar tasks. The policy prior is parameterized as a deep conditional generative model that maps a noise input and a state to an action. The latter map can be inverted, which is important to guarantee that the prior assigns nonzero probability to the full action space for all states. Given a new task, the policy prior is used to parameterize a new policy; the new policy outputs noise inputs to the policy prior’s invertible mapping, which in turn outputs an action in the original action space. This parameterization of the new policy leads to much more targeted exploration versus sampling actions uniformly from the original action space. Experiments are on a suite of pick-and-place robotic tasks in simulation.\n\n### Pros\n\n- The writing is overall very clear and persuasive. The introduction is especially compelling. The sections are well organized.\n- Related work is quite thorough, with just a few potential omissions (see Cons)\n- The problem setup section is very much appreciated. It is difficult to formalize or even discuss the assumptions behind an approach like this, but I thought the authors did a very good job.\n- The problem setting in general is very motivating. It does seem realistic to suppose that a set of expert trajectories from related tasks are available, but that they are not necessarily annotated with rewards. The authors do a good job explaining how this differs from other problem settings like LfD, meta-learning and meta-imitation learning.\n- The whole method is elegant. It is a sensible integration of existing techniques to address a well-motivated problem.\n- The experimental results are strong (though there are many opportunities for additional experiments)\n- The baselines are well chosen\n- The pseudocode in appendix A is clean and clear. All of the notation throughout the paper is too.\n- The paper provides a good level of detail in appendix B for reproducibility\n\n### Cons\n\n- Missing references: \n    -  “Learning Action Representations for Reinforcement Learning” Chandak et al. ICML 2019\n    - “Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning” Siegel et al. ICLR 2020\n    - “Residual Policy Learning” Silver et al. (2018) and “Residual Reinforcement Learning for Robot Control” Johannink et al. (2018)\n- The empirical results are very encouraging, but I would like to have a better understanding of when the overall approach might fail. Some experiments in this direction would include:\n    - Evaluate test-time performance as a function of the number of training tasks\n    - Deliberately bias the training tasks so that they non-uniformly sample from the set, and see how much of an effect that has on the downstream performance. For example, train only on pick tasks and test only on pick-and-place tasks, or vice versa.\n    - More domains beyond the single one considered\n- It would also be nice to see some ablations. In particular, I am curious about the extent to which parameterizing the new task-specific policy with the behavior prior is important, versus just using the behavior prior to gather data. A simple comparison would be to use the behavior prior as an “exploration policy” that is called epsilon% of the time during RL.\n- I highly encourage the authors to release code for the paper\n- Three random seeds is not enough\n\n### Detailed Comments\n\n- Figure 1 is visually compelling, but a little hard for me to follow. I am most thrown off by the “exploration” part, which says: “attempt all possible tasks in a new scene.” I am interpreting this to be somewhat metaphorical -- it is not as though there is actually a list of tasks, and you are attempting each one in a new scene. Also, as far as I understand, there is not a strict separation between exploration and “learn a specific task via RL” -- exploration is part of that RL process. Let me know if my understanding here is correct. If so, I would recommend perhaps cutting the exploration part of this figure.\n- Figure 2 is very helpful and clear\n- In the problem setup, there is discussion of fixed state and action space dimensionalities, but it is not stated that the spaces are vector spaces. Would it suffice to instead say that $\\mathcal{S}$ and $\\mathcal{A}$ are assumed fixed?\n- In the problem setup, in two places, there is an expectation over single step rewards that should really be an expectation over temporally discounted returns. \n- In the problem setup, it says: “Note, however, that the optimality assumption is not very restrictive: every trajectory can be interpreted as an optimal trajectory for some unknown reward function.” I think this is misleading. It is true that there is some reward function for every trajectory, but the whole point of the problem setup is that there is assumed to be a nontrivial distribution over MDPs, and the important question is whether the observed data are representative of this distribution.\n- Please indicate in the caption what the lines and shaded areas represent in Figure 5. I assume they are means and standard deviations, but other choices are plausible too.\n- Typos:\n    - “Each task can considered a Markov decision processes” (two typos)\n    - “(making it expressive)” is missing a period\n    - “each coupling layer in the real NVP take as input” (should be “takes”)\n    - “hyparparameters”\n    - “This comparisons”\n    - “behavioural” (I know this is an alternative spelling, but the paper should be internally consistent)\n\n### Questions\n\n-  How does the asymptotic performance of PARROT compare to vanilla RL? Is there a point at which RL starts to outperform PARROT in the tasks considered in this work? It looks like this happens in “Pick up Baseball Cap”.\n-  What steps would need to be taken to try PARROT in an environment with discrete actions?\n- How might PARROT be used in a continual/lifelong learning setting? My concern is that if the behavior prior is changing over time, all of the policies that have been parameterized using the behavior prior would also change, and would potentially need to be relearned. (This question does not necessarily need to be addressed in the submission -- I am just curious.)\n-  What would it take to apply PARROT in a domain where it is not possible to write down good scripted policies (like the ones in Appendix C :) )?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5j2TU7N-_R", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the paper \"Parrot: Data-Driven Behavioral Priors for Reinforcement Learning\", the authors propose a new approach to leverage previously acquired data to learn a new conditional search space (called representation) that accelerate the convergence of RL algorithm.   \n\nThe paper is clearly written and well structured. The figures provide a nice illustration of the algorithm and the results are appropriately reported in the graphs. I am just annoyed to see so many important technical details, like the exact definition of tasks, the networks and the different parameters relegated to the appendix, while they are essential to understand the work. However, I understand that is sadly now customary in this venue. \n\nThe presented results demonstrate the benefits of the proposed approach, which looks to be at the same time simple and yet very effective. That's great. \nHowever, I have two main issues with the paper:\n1) This paper is directly related to the research domain of transfer learning or knowledge transfer, but this is completely ignored in the related work section. In particular, concepts like learning transferable features would appear quite relevant to this work and should also be considered in the experimental reason. \n2) I was particularly disappointed by the experimental analysis. More precisely, by the 13 lines of the result section that briefly summarise the content of figure 5. In particular, it shows that PARROT outperforms all the other baselines, great, but there is no further comment/discussion/analysis than that. Ideally, a scientific paper should explain (or at least try to) the observed phenomenon, and thus I was expecting some further analysis that would demonstrate the main hypothesis of the paper, for instance via an ablation study. \n\nOverall, I think that this is an interesting paper which shows promising results, while there are a couple of sections in the paper where the same concepts are repeated multiple times and space could be saved and reused to expand the technical description and the result analysis. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an elegant and effective approach to knowledge transfer in RL by learning a policy prior from expert data. The paper is generally well structured and well written.
Generally, all the reviewers were favourable about this paper, with its simple idea and convincing results.
It was thought that the paper would benefit from the addition of more discussion around related work, and more experimental results, but it remains a strong paper.
Decision: Accept (Oral)