Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an approach for imitation learning from parametric experts, by augmenting the data and then querying the experts for action labels on the new states. The authors emphasize two practical settings where learning from parametric experts is useful - expert compression and learning from privileged experts. Includes evaluation on continuous control mujoco tasks (humanoid and peg insertion).

strengths_and_weaknesses: Originality - 

-> The proposed idea is very simple and intuitive. Data augmentation is commonly used for machine learning approaches and recently also in RL for improved performance [1,2]. Given parametric experts, querying them for action labels on the new augmented states seems like it would give rise to better performance, since more information is being extracted from the expert. This is similar to how in dagger when new states are encountered, the expert is queried for labels, except here the new states don’t require a policy rollout but rather a direct perturbation of the recorded states. While the paper does include experiments on dagger with data augmentation, I’d be curious to see how regular dagger compares to the proposed approach, in terms of policy performance and overall sample efficiency.
 
-> The paper is missing a related works section that goes into depth about prior work in the area, but the structure of the paper does include some discussion on prominent imitation learning methods (dagger, kickstarting, privileged experts) but doesn’t mention specific works adequately. 
 
[1] : Reinforcement Learning with Augmented Data (Lashkin et al.)
[2] : Regularizing Deep Reinforcement Learning from Pixels (Kostrikov et al.)
 
Significance - 

-> The proposed method can be of interest to a large section of the community interested in imitation learning, and is quite simple to adopt and implement. Sample efficiency is critical especially for robot learning, and the proposed approach shows the ability to learn control policies in the very low data regime. The only caveat is that the method needs parametric experts, but the authors are very clear about this and point out practical areas where the proposed approach is relevant - i.e expert compression and learning from privileged experts. The latter is specifically relevant for robotics, since sim to real pipelines [3,4] can adopt this idea and become more efficient. 
 
[3] : Rapid Motor Adaptation (Kumar et al.)
[4]: Solving Rubik’s Cube with a Robot Hand (OpenAI)
 
Quality -

-> The experimental evaluation seems quite thorough (performance is reported on 150 random environment instantiations). The authors show that the proposed approach fits the expert data much better than the baselines given an offline dataset, and the results on learning from privileged experts do seem to show that the proposed approach is effective
 
-> However, on the expert compression task, the only environment where there’s a clear advantage for the proposed method is humanoid run. On peg insertion, the baseline’s high capacity model has quite poor performance so it’s difficult to judge compression ability, and on humanoid walls the relative compression of the proposed approach and the baseline appear similar. I would recommend defining a new metric for compression (i.e ratio of performance of a low capacity model to that of the high capacity model) so that performance can be more accurately judged. 
 
Clarity - 

The paper is well written and motivated, and easy to follow. It includes clear algorithm boxes and equations that are sufficient to understand the details of the method. 
 
 

questions: -> How does the proposed approach compare to regular dagger.  Specifically, how much worse is the policy performance (if at all), if the policy isn’t rolled out to collect new states, but instead the recorded states were just perturbed to get new states, and the expert was then queried for new actions ? What are the sample efficiency gains ? 
 
-> What is the performance of the expert MPO policies on the domains reported? 
 
-> For expert compression results, please plot ratio of performance of a low capacity model to that of the high capacity model for each method for easier comparison of compression ability. 

limitations: The limitation are adequately discussed, primarily being requiring a parametric expert.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a data augmentation technique for behavioral cloning algorithms. It consists of adding Gaussian noise to the states used for training and storing the perturbed states and the action the expert policy would produce in such states. Using this augmented dataset, the proposed algorithm achieves better performance than the baselines over three complex continuous tasks.
strengths_and_weaknesses: The paper is well written and contains many implementation details and ablation studies.

The idea proposed is very simple and effective. Unfortunately, apart from the experimental results in complex environments, the novelty in this work is limited. It is well known that augmenting the dataset in Behavioral Cloning is useful and there are prior works applying observation perturbation and label corrections via experts queries [1,2,3].

The main motivation for using the proposed approach is the case when collecting more data in the environment is expensive, but querying the expert many times is not. The authors motivate this by saying that "an expert policy may be too large to execute due to memory considerations" and that "Many expert-driven learning approaches actually have access to an expert that can be queried", but no specific example is provided to the reader. 

The comparison with baselines BC and ABC does not seem very fair. If the policy has enough capacity and the distribution from which we sample the states covers the state-space enough, one could clone the expert by repeatedly querying it on such random states without even seeing a single trajectory. Therefore it is quite straightforward that the proposed method works better than the baselines.  

[1] M. Bojarski et al., “End to end learning for self-driving cars,” arXiv preprint, 2016

[2] L. George, T. Buhet, E. Wirbel, G. Le-Gall, and X. Perrotton, “Imitation learning for end to end vehicle longitudinal control with forward camera,” in NeurIPS, 2018.

[3] T. Buhet, E. Wirbel, and X. Perrotton, “Conditional Vehicle Trajectories Prediction in CARLA Urban Environment,” in ICCV, 2019



questions: While in continuous control environments adding Gaussian noise to a state often produces a perturbed state that could be reached by an agent (i.e. valid states), it is not the case when operating with visual observations. Is there an intuition why data augmentation would be useful in the visual domain?

Assume that, when the expert was trained, it observes in the environment states with mean $\mu$ and standard deviation $\sigma$ (this could be estimated from the last few trajectories during the training of the expert). What is the performance of APC when using zero trajectories and only synthetic data coming from a normal distribution with mean $\mu$ and standard deviation $\sigma$? 


Figure 1 and Algorithm 1 appear many pages before the first time they are mentioned in the paper. I suggest putting them when they are mentioned.
limitations: The authors discuss the possible limitations of this work. The method queries the expert many times, so if expert queries are expensive, their approach might suffer. I suggest providing specific examples for this setting.




---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Edit: updating my score after author-reviewer discussion
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: They use augmentatio to improve data efficiency with learning from parametric experts. In naive data augmentation approach, parametric experts act same on augmented states and original states. However, APC get different action on augmented states using parametric experts.
They add a samll Gaussian state perturbation to state, and parametric experts predict action on perturbed state. 
They experiment on continual control tasks, and show improved data efficiency. 
strengths_and_weaknesses: Strength
This paper is well written with proper experiments. 
APC significantly improve data efficiency, while naive ABC have almost similar data efficiency with BC. 
Their experiement section is well structured for application of their method APC such as Expert compression and Privileged experts. 

Weakness 
Baselines are not enough. They have to add other offline RL algorithms to compare with APC. 
The performance of experts is not revealed in the Figures. 
questions: 1. I want to check the change of APC performance according to experts level. 
2.  Ablation study of sigma for Gaussian perturbation. (If the perturbatsion is small, a new action sampling from experts can be not needed.) 
3. Do you try other perturbation methods on states? 
limitations: In my opinion, their contribution is not enough, because APC can only used with parametric experts. Thus, we have to need well trained experts. Therefore, the scope of utilization is bound to be limited.
Furthermore, baseline is not enough to assure the contribution of APC.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a simple yet effective way APC, which queries the parametric experts to augment the demonstrations for imitation learning. APC is shown to outperform naive behavior cloning in both expert compression and transferring from privileged expert settings.
strengths_and_weaknesses: Pros:
This paper focuses on a novel problem where a parametric expert can be queried. The paper writing is clear and easy to follow. The empirical results do demonstrate the advantage of APC over BC and naive ABC. 

Cons:
However, I don't feel this paper proposes any novel idea. One serious issue is that when compared with baselines, all these baselines don't query the expert as APC does. This means that the baselines have fewer demonstrations to imitate from. It is clear that APC will beat naive ABC since APC has more high-quality/ground-truth demonstrations. Therefore, I think this doesn't make enough technical contributions to the problem it studies.
questions: I think the key problem here is that the authors don't explore the methods of querying experts enough. From my point of view, the advantage of APC is totally brought by a new setting where the expert could be queried, which is not an innovation of the methods. I suggest the authors compare APC with more baselines that can query the experts. 

Although this method is reasonable, I'm afraid the current technical contribution is far from enough. Adding the Gaussian noise is the simplest way to make a data augmentation, and previous works (e.g., RAD, https://arxiv.org/pdf/2004.14990.pdf) also have systematically studied the data augmentations for general reinforcement learning problems. It could be better if the authors come up with novel methods for data augmentations in your settings.
limitations: The authors have addressed the limitations.
review_title: null

Meta-review: metareview: This paper studies an interesting problem, and overall the reviewers agreed the exposition and validation are sufficient. We encourage the authors to consider the issues raised by the reviewers and further improve the work in the final version.

confidence: Certain
Decision: Accept