Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper introduces PARROT, a novel approach for pretraining a reinforcement learning agent on near-optimal trajectories by learning a behavioral prior. Essentially, the authors learn a word2vec style embedding of actions for a simple virtual single-arm environment. This embedding will naturally place more common examples from its training data towards the center of the gaussian, making sampling them during training time more likely. The authors demonstrate that this approach outperforms existing pretraining methods in this domain. 

This paper presents an interesting novel approach and presents strong support for its value. In particular I appreciate that the basic intuition is relatively straightforward and therefore should be relatively easy to test in new domains. In addition, the evaluation results are impressive. 

The paper claims in two instances that human examples would be appropriate for the training data. However, this is not confirmed in the current evaluations to my understanding. The paper indicates that the training data should be “near optimal” multiple times but never explicitly defines what is meant by “near optimal”. Is it that actions must be “useful” (also vague)? More clarity on this would be appreciated. 

I have some concerns around certain claims or arguments made in the paper (more on that below). However, this paper still ticks all the boxes for me in terms of novelty and value, and so I would argue for its acceptance. 

Some questions for the authors: 
1. What is meant by near optimal? Can this be more formally defined?
2. The constraint that the action dimensionalities are fixed seems like a large one. I understand that it’s necessary for the invertible requirement, but I could imagine that a similar approach to this could still be helpful in sim2real problems without this constraint. Would the authors agree?
3. The paper domain is still relatively simple compared to real world problem domains, to what extent do you expect this approach to generalize? 

Some smaller points/additional feedback: 
- The introduction and abstract are a bit vague, and I didn’t grasp the approach intuition until figure 1. I’d appreciate if some of this information could be moved up into paragraph 3 of the introduction. 
-I’d replace “large such datasets” with “such large datasets” or just remove “such”
-I recognize that theoretically that the approach should be able to represent every possible environment action, but it would seem to me still possible that good/useful actions could be placed so far from the center of the Gaussian as to make them very unlikely, depending on the training data. Some discussion about how to avoid this/whether it is a concern would be helpful. 
review_title: Novel and valuable work

final_score: rating: 6: Marginally above acceptance threshold
review_content: In the paper "Parrot: Data-Driven Behavioral Priors for Reinforcement Learning", the authors propose a new approach to leverage previously acquired data to learn a new conditional search space (called representation) that accelerate the convergence of RL algorithm.   

The paper is clearly written and well structured. The figures provide a nice illustration of the algorithm and the results are appropriately reported in the graphs. I am just annoyed to see so many important technical details, like the exact definition of tasks, the networks and the different parameters relegated to the appendix, while they are essential to understand the work. However, I understand that is sadly now customary in this venue. 

The presented results demonstrate the benefits of the proposed approach, which looks to be at the same time simple and yet very effective. That's great. 
However, I have two main issues with the paper:
1) This paper is directly related to the research domain of transfer learning or knowledge transfer, but this is completely ignored in the related work section. In particular, concepts like learning transferable features would appear quite relevant to this work and should also be considered in the experimental reason. 
2) I was particularly disappointed by the experimental analysis. More precisely, by the 13 lines of the result section that briefly summarise the content of figure 5. In particular, it shows that PARROT outperforms all the other baselines, great, but there is no further comment/discussion/analysis than that. Ideally, a scientific paper should explain (or at least try to) the observed phenomenon, and thus I was expecting some further analysis that would demonstrate the main hypothesis of the paper, for instance via an ablation study. 

Overall, I think that this is an interesting paper which shows promising results, while there are a couple of sections in the paper where the same concepts are repeated multiple times and space could be saved and reused to expand the technical description and the result analysis. 
review_title: Overall, I think that this is an interesting paper which shows promising results, but that deserves a better discussion about the related work and the result analysis. 

final_score: rating: 7: Good paper, accept
review_content: This work proposes a method, PARROT, to learn data-driven priors for deep reinforcement learning agents. Motivated by the idea of pre-training with existing data of similar tasks, the authors propose to learn state-conditional behavioral priors from a set of similar tasks for reinforcement learning agents, such that a learning agent explores its environment in a meaningful way. Successful trials from past tasks are used as training data to learn a mapping from pixel-level state input to actions. Experiments in simulated robotic manipulator domain demonstrate the benefit of learning behavioral priors, comparing PARROT against algorithms learning from scratch as well as agents pre-trained with behavioral cloning.

Pros:
The problem setting tackled in this paper is important for applying deep reinforcement learning algorithms on real world robotic tasks; 
Training the behavioral prior with PARROT is fully offline, which makes it favorable for practical concerns, as existing meta learning algorithms often require online data (interacting with the environment);
The selection of baseline algorithms in this paper are satisfactory, showcasing the benefits of each design choices made with PARROT;
The paper is well-written and easy to follow in general; the related work section clearly identifies the novelty of PARROT comparing with existing literature

Cons:
PARROT only conducted experiments in a simulated robotic environment, possibly due to the fact that the RL step still takes a considerable amount of data to converge; ideas from works in the imitation learning literature such as GTI by Mandlekar et al. can be leveraged for improving the overall data-efficiency of PARROT


Mandlekar, Ajay, et al. "Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations." Proceedings of Robotics: Science and Systems (RSS), July 2020.

review_title: good method for pre-training for RL; well-written paper

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ### Summary

This paper proposes PARROT, a method for learning a policy prior from a dataset of expert state-action pairs that have been derived from multiple similar tasks. The policy prior is parameterized as a deep conditional generative model that maps a noise input and a state to an action. The latter map can be inverted, which is important to guarantee that the prior assigns nonzero probability to the full action space for all states. Given a new task, the policy prior is used to parameterize a new policy; the new policy outputs noise inputs to the policy prior’s invertible mapping, which in turn outputs an action in the original action space. This parameterization of the new policy leads to much more targeted exploration versus sampling actions uniformly from the original action space. Experiments are on a suite of pick-and-place robotic tasks in simulation.

### Pros

- The writing is overall very clear and persuasive. The introduction is especially compelling. The sections are well organized.
- Related work is quite thorough, with just a few potential omissions (see Cons)
- The problem setup section is very much appreciated. It is difficult to formalize or even discuss the assumptions behind an approach like this, but I thought the authors did a very good job.
- The problem setting in general is very motivating. It does seem realistic to suppose that a set of expert trajectories from related tasks are available, but that they are not necessarily annotated with rewards. The authors do a good job explaining how this differs from other problem settings like LfD, meta-learning and meta-imitation learning.
- The whole method is elegant. It is a sensible integration of existing techniques to address a well-motivated problem.
- The experimental results are strong (though there are many opportunities for additional experiments)
- The baselines are well chosen
- The pseudocode in appendix A is clean and clear. All of the notation throughout the paper is too.
- The paper provides a good level of detail in appendix B for reproducibility

### Cons

- Missing references: 
    -  “Learning Action Representations for Reinforcement Learning” Chandak et al. ICML 2019
    - “Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning” Siegel et al. ICLR 2020
    - “Residual Policy Learning” Silver et al. (2018) and “Residual Reinforcement Learning for Robot Control” Johannink et al. (2018)
- The empirical results are very encouraging, but I would like to have a better understanding of when the overall approach might fail. Some experiments in this direction would include:
    - Evaluate test-time performance as a function of the number of training tasks
    - Deliberately bias the training tasks so that they non-uniformly sample from the set, and see how much of an effect that has on the downstream performance. For example, train only on pick tasks and test only on pick-and-place tasks, or vice versa.
    - More domains beyond the single one considered
- It would also be nice to see some ablations. In particular, I am curious about the extent to which parameterizing the new task-specific policy with the behavior prior is important, versus just using the behavior prior to gather data. A simple comparison would be to use the behavior prior as an “exploration policy” that is called epsilon% of the time during RL.
- I highly encourage the authors to release code for the paper
- Three random seeds is not enough

### Detailed Comments

- Figure 1 is visually compelling, but a little hard for me to follow. I am most thrown off by the “exploration” part, which says: “attempt all possible tasks in a new scene.” I am interpreting this to be somewhat metaphorical -- it is not as though there is actually a list of tasks, and you are attempting each one in a new scene. Also, as far as I understand, there is not a strict separation between exploration and “learn a specific task via RL” -- exploration is part of that RL process. Let me know if my understanding here is correct. If so, I would recommend perhaps cutting the exploration part of this figure.
- Figure 2 is very helpful and clear
- In the problem setup, there is discussion of fixed state and action space dimensionalities, but it is not stated that the spaces are vector spaces. Would it suffice to instead say that $\mathcal{S}$ and $\mathcal{A}$ are assumed fixed?
- In the problem setup, in two places, there is an expectation over single step rewards that should really be an expectation over temporally discounted returns. 
- In the problem setup, it says: “Note, however, that the optimality assumption is not very restrictive: every trajectory can be interpreted as an optimal trajectory for some unknown reward function.” I think this is misleading. It is true that there is some reward function for every trajectory, but the whole point of the problem setup is that there is assumed to be a nontrivial distribution over MDPs, and the important question is whether the observed data are representative of this distribution.
- Please indicate in the caption what the lines and shaded areas represent in Figure 5. I assume they are means and standard deviations, but other choices are plausible too.
- Typos:
    - “Each task can considered a Markov decision processes” (two typos)
    - “(making it expressive)” is missing a period
    - “each coupling layer in the real NVP take as input” (should be “takes”)
    - “hyparparameters”
    - “This comparisons”
    - “behavioural” (I know this is an alternative spelling, but the paper should be internally consistent)

### Questions

-  How does the asymptotic performance of PARROT compare to vanilla RL? Is there a point at which RL starts to outperform PARROT in the tasks considered in this work? It looks like this happens in “Pick up Baseball Cap”.
-  What steps would need to be taken to try PARROT in an environment with discrete actions?
- How might PARROT be used in a continual/lifelong learning setting? My concern is that if the behavior prior is changing over time, all of the policies that have been parameterized using the behavior prior would also change, and would potentially need to be relearned. (This question does not necessarily need to be addressed in the submission -- I am just curious.)
-  What would it take to apply PARROT in a domain where it is not possible to write down good scripted policies (like the ones in Appendix C :) )?

review_title: Very nice work, with some room for additional experiments

Meta-review: This paper presents an elegant and effective approach to knowledge transfer in RL by learning a policy prior from expert data. The paper is generally well structured and well written.
Generally, all the reviewers were favourable about this paper, with its simple idea and convincing results.
It was thought that the paper would benefit from the addition of more discussion around related work, and more experimental results, but it remains a strong paper.
Decision: Accept (Oral)