Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper introduces a meta-learning approach that can learn from demonstrations and subsequent reinforcement learning trials. The approach works in two phases. First, multiple demonstrations are collected for each task and a meta-learning policy is obtained by embedding the demonstrations of a single task in a context vector that is given to the actor policy.  Subsequently, the meta-policy is used to collect trajectories which are also evaluated with a (sparse) reward. These trajectories, along with the demonstrations, are again used by a second meta-learning policy (with a concatenated context from demonstrations and trajectories) to obtain the final policy. The algorithm is tested on a simple point reaching task as well as on a more complex 3d physics simulation that contains pick-and-place tasks, pushing tasks and button pressing.

The approach seems really interesting and I think combining demonstrations with reinforcement learning for meta learning is a very promising approach. This is also underlined by the experimental results presented in the paper. The only concern I have is the presentation of the paper. The algorithmic description is very short and many things are left unclear. I would also like to see a bit more ablation studies in the paper. More comments below:

- Just from the text it is not clear how the meta-learning actually works and you have to study the figures in detail to understand that a context is extracted from the demonstrations as well as the RL trajectories. The context should be better introduced (even though this has been shown already in different papers, I think it is a good strategy to write the  
in a self-contained way).
- It is unclear to me how the reward signal is used to learn the meta-policy is phase 2. I understand that it is concatenated to the embedding before forming the context (again this is not really described in text but only in the figure), but it is quite unclear to me how the reward is used as optimality criteria. How do you reinforce good trajectories and decrease the influence of poor trajectories? This is maybe one of the most important parts of the paper and needs to be described in much more detail.
- The reward functions are not properly explained in the paper, even not for the toy task (reaching). This should at least be done in the appendix.
- I would like to see more ablation studies. For example, it is mentioned that 40 observations are used from the demonstrations and the trajectories. How is this number picked and how does it influence the performance? Also, typically, how long is a single trajectory? It is also unclear to me, if we take 40 random observations and the reward signal is sparse, wouldn't it be quite likely that the observations do not contain any reward values?
- How does K influence the performance of the algorithm?

 




review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This work focuses on meta learning from both demonstrations and rewards. The proposed method has certain advantages over previous methods:
(1) In comparison to meta-imitation, it enables the agent to effectively and efficiently improve itself autonomously beyond the demonstration data. 
(2) In comparison to meta-reinforcement learning, it can scale to substantially broader distributions of tasks, as the demonstration reduces the burden of exploration.

I am a bit confused by the writings and don't clearly understand the algorithm.
1. In eq. 4, should \theta be \phi?

2. What does \pi_\theta(a_t|s_t, {d_i,k}) mean? I'd like to see an example formulation of the policy. When the parameter \theta is well trained, does the policy still take demonstrations {d_i,k} as inputs? If yes, why are demonstrations needed? 

3. In meta-testing (Algo. 2), will \theta and \phi be updated? Currently it looks like that the two parameters are fixed in meta testing. If so, this is a bit strange. Why not update the policies after sampling demonstrations (step 4) and collecting trials (step 5)?

4. "The state space S, reward ri, and dynamics Pi may vary across tasks." I think all the tasks should share the same input space; otherwise, the meta-trained two policies cannot be applied to a test task with different input space.  For example, if the states are 100x100 images in meta training, how to apply the policies to a meta-testing task with 100x200 images as states? 

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes an approach for combining meta-imitation learning and learning from trial-and-error with sparse reward feedback. The paper is well-written and the experiments are convincing. I found the idea of having separate networks for the two phases of the algorithm (instead of recollecting on-policy trial trajectories) interesting and possibly applicable in other similar settings. 

The paper has a comprehensive analysis of the advantages of the method over other reasonable baselines which do not have the trial-and-error element. However, in terms of the limitation of the method, the paper only has a small failure analysis section in the appendix; adding a more detailed discussion on the limitations can improve the impact of the paper further. 
review_title: Official Blind Review #1

Meta-review: The paper proposed a meta-learning approach that learns from demonstrations and subsequent RL tasks.
The reviewers found this work interesting and promising. There have been some concerns regarding the clarity of presentation, which seems to be addressed in the revised version. Therefore, I recommend acceptance for this paper.
Decision: Accept (Poster)