Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary

The authors introduce a simple extension of PPO that uses a single expert demonstration and a modified sampling algorithm to show better performance than vanilla PPO in a difficult 3D setting.

## Strengths & Weaknesses

#### Strengths

- The paper is well-written and the method is simple, yet powerful. Learning something from a single demonstration is no easy feat.
- The videos shown in the submission look impressive in how difficult the setup is and how the agent manages to learn complex strategies like.
- Overall, the paper is very "short & sweet" in that it's not a groundbreaking new technique, but a small change to PPO but it's well explained, and the results that _are_ in the paper are good.
  
#### Weaknesses

- The main problem I have with this is actually the lack of further experiments. For such an easy extension of PPO, I would've expected you to have no problem running this on an Atari environment and at least one MuJoCo environment too (where it's also easy to gather a human demonstration, like controlling the reacher via inverse-kinematics or the tricky Pusher). Compared to vanilla PPO we should see improvements across the board, no?
- Similarly, you just arrived at the hyperparameters $p = 0.1, \phi=0.3$ without explanation or ablation. Do you maybe want to justify how these parameters came to be and what happens if either parameter is higher or lower?

**TL;DR how to make me raise my score:** Include at least one Atari and one Mujoco/Robot environment (since Ilya Kostrikov's implementation that you use supports these out of the box) and either add an ablation study on a single env over p/phi or explain the importance of these values.

## Impact & Recommendation

This is fundamentally good material. It's not groundbreakingly new but I think it could make for an easy-to-use imitation learning baseline that would help in a lot of scenarios get the method off the ground. However, the current paper doesn't show the rigour and depth of analysis that I would expect from an ICLR paper. I hope the authors can make up for that in the rebuttal week and then I'm happy to up my score. Currently, my recommendation is borderline. If the present method was really an all-around improvement over PPO, why did the authors not show it on a tried-and-true OpenAI Gym task but only in their own made-up setting? 

## Minor Nitpicks

- I'd report a few more seeds - I think 5 seeds is a good starting point.
- Your plotting of runs is uncommon - usually, you either plot the mean and standard deviation or the mean and min/max. 
- In Fig. 3 and 4, the fonts in the legend need to be bigger 
- On page 6 you're twice in a row weirdly enthusiastic for Unity-ML / the "flexibility allowed by this environment". These are odd things to say in a research paper unless you're an employee of Unity
- Algorithm 1 is a bit verbose but great. Makes it very clear. On the flip side of that, Figure 1 is a bit redundant. These two things communicate the same idea and I like Algo 1 better.
- There are a few missing commas, like "In our approach,", bottom of the first page.


review_title: Great work, but missing experiments or more depth in analysis

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################

Summary:

The paper proposes a modification of the PPO algorithm which can accommodate a single task demonstration with the goal of faster learning in sparse-reward tasks. There are 4 tasks proposed by the paper, inspired by Animal-AI Olympics. The baselines are Behavioural Cloning (BC) and vanilla PPO. The proposed method outperforms those baselines.


##########################################################################

Reasons for score:

From the text it seems like BC baseline doesn't have access to rewards. If that is the case, the baselines are very weak - they either have access to demonstrations only or sparse rewards only.

The most basic fair baseline would be doing BC gradient updates to policy network from time to time during PPO training - should be a reasonable amount of work to implement. Ideally, comparison to relevant prior work should be at least attempted: Paine et al (R2D3) or some other whichever is easier to implement (see below for related work comments).


##########################################################################

Pros:

1. Nice research direction: combining demonstrations with sparse rewards.

##########################################################################

Cons:

1. More baselines are needed to evaluate the value of the proposed method. Prior work does exist (e.g., Paine et al.) and warrants more extensive comparisons.
2. Related work needs to mention some relevant papers:
- "One-Shot Imitation Learning" https://arxiv.org/pdf/1703.07326.pdf
- "Watch, try, learn: meta-learning from demonstrations and rewards" https://arxiv.org/pdf/1906.03352.pdf
3. There are some established benchmarks for sparse-reward tasks - it might be more productive to attack those as they already have some baselines. Introducing new tasks requires some discussion on why the established tasks are unsuitable for the goals of the paper. For example:
- Vizdoom navigation by Pathak et al https://arxiv.org/pdf/1705.05363.pdf
- https://openai.com/blog/ingredients-for-robotics-research/
- https://aihabitat.org/

##########################################################################

Questions during rebuttal period: 

Please address and clarify the cons above.
review_title: Baselines are very weak

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary
This paper looks at problems that have sparse rewards, are partially observable, and have
variable initial conditions.  Previous work [R2D3, Paine et al 2019] tackles this problem using 
off-policy recurrent Q-learning. Instead, this work proposes to use a PPO+D, an on-policy method (PPO)
for a policy with memory (GRUs). This relies on storing successful trajectories (demonstration(s) 
and discovered), and replaying these as experiences using importance sampling. Other details
such as a value-prioritized buffer, annealed use of that buffer, an entropy term, and more
are given.  The method is tested on four environment variations, as constructed in the Animal-AI Olympics challenge env.
Exact comparisons with R2D3 are not possible due to missing initial condition ranges. 
It is compared to a behavior cloning baseline, vanilla PPO, and a value-buffer ablation,
It is shown that it can succeed with just one demonstration, although it fails to generalize
to all many of the variations for the problems.

## Strengths
- interesting problem setting (although this is not novel, e.g., R2D3)
- interesting exploration beyond R2D3, i.e., to on-policy methods
- use of value buffer as a means to allow for progress on harder problems

## Weaknesses
- could have compared against an implementation of some facsimile of R2D3
- lack of generalization to wider initial conditions means that the results may not be as interesting as they first seem
- behavior cloning is, unsurprisingly, a poor baseline for a problem with variable initial conditions and with partial observability
- the main result is really: "sometimes it works!".  That is actually of interest, although really a "proof by example" result.
- only tested on limited tasks, although similar to R2D3.  Thus it is unclear how generalizable the method and results will be;  it may overfit in some respects to the task at hand.

## Recommendation
Overall, while the results come with some real caveats, the paper may inspire further work in the use of demonstrations in sparse-reward settings, in conjunction with on-policy methods such as PPO. The use of the value-prioritized buffer is also a feature
that could see further adoption. 

## Questions
- why could behavior cloning realistically be expected to work as a baseline, given the variable initial states?

## Additional Feedback
- DR and DV: it's unclear what the 'R' stands for in DR.  It's always helpful to define the origins of the chosen nomenclature for cases where it is not obvious.
- please tell the reader where the value estimates come from for the value buffer for unsuccessful trajectories.  Why aren't these all simply zero?
- Figure 1 is really helpful;  place it earlier in section 3?
review_title: Progress on a difficult class of RL problem

Meta-review: There was quite a bit of internal discussion on this paper. To summarize:
- The idea is very neat and interesting and likely to work
- The paper is likely to inspire future work
- There are still serious doubts  about the experimental evaluation that is not entirely up to par with current standards
  - The reviewers were not convinced 100% by the arguments about the 'custom' environments
  - The reviewers were not convinced 100% that the baselines were given their best shot

While the paper has potential to provide valuable input for the community, it needs a bit more work before being presentable at a highly competitive venue like ICLR.

Decision: Reject