Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an approach for distilling motion-planner augmented policies that operate in state space into visual control policies that operate without augmented actions from a motion planner. The first stage of this approach is to use Motion Planner Augmented Reinforcement Learning (MoPa-RL) to train an actor and critic model that operates in state space and has augmented actions from a motion planner to learn a policy to complete a task. The second stage of this approach uses behavioral cloning to train an actor network that operates in visual space and without a motion planner by using demonstrations from the expert MoPa-RL agent. The third stage trains an asymmetric actor-critic model to learn a visual motor policy and state-based critic that do not use the augmented action space by using demonstrations from the behavioral cloning agent and weight initializations from the MoPa-RL expert agent. The authors conduct experiments in 3 object manipulation domains (object pushing, object lifting, and object assembly) that involve obstacles and visual obstructions, and compare their full method to ablated versions of their method as well as other state-of-the-art RL and learning from demonstration approaches. This paper also demonstrates the robustness of their approach to domain randomization.
strengths_and_weaknesses: Strengths:
- This paper presents a novel and interesting approach for distilling a motion-planner augmented policy that operates in state space into a visual control policy that does not rely on motion planning at all. The method is also extremely clear and well-presented, and the reviewer commends the authors for their clarity. Overall, this method is a very good contribution for enabling motion planners to be useful for learning visual motor policies without requiring a motion planner or hand-crafted state space to be used at inference time. Overall this paper and method deserve praise and could provide a good contribution to the field, and the reviewer really enjoyed reading the paper.
- This paper includes various ablations of their method and plenty of comparisons to other state-of-the-art approaches for this robot policy learning task, and report quantitative results on average success rate, episode length, and the discounted returns.
-The figures in the paper are clear and useful for understanding the method, robot domains, and results.

Weaknesses:
- This proposed approached requires several stages of environment interactions that are unaccounted for the in quantitative results, which questions how sample efficient this method really is. The first stage requires MoPA-RL to interact and learn from the environment. The second stages requires behavioral cloning to occur (to get a smoother policy), and then they also collect more samples using the behaviorally cloned agent before doing the final stage of learning with the asymmetric actor, which further requires environment interactions. However, it appears none of the results reported in this paper appear to include how long these previous stages up to the last one take, or at least it is not clear from the graphs at which part each stage occurs. The authors should be explicit in their graphs about how many samples each of the stages require if they wish to make claims that their overall method is more sample efficient than other approaches. This is especially important since the authors emphasize how important the BC step is for good results (re: Sawyer assembly task)
- The authors claim in the related work that "However, using HER is not applicable for composite tasks involving goals associated with multiple sequential steps", but this statement needs to be justified since it is not clear why this would be true, since rewards can just be relabeled based on whether they achieve a subgoal (defined by composite task) or not. 
- The authors should clarify what the state space was used in the experiments, and they should also clarify the size of the images used for the observations.
- In section 3.3.1, the authors claim to store transitions from MoPA-RL into a demonstration dataset for later stages. Does this include transitions that include motion-planner augmented actions?
- The authors claim they conduct their experiments in a sparse reward setting, but in the the tasks they use a dense reward around an epsilon ball of the goal proportional to the distance, they should specify how large epsilon is. 
- The authors do not specify what the gamma value is, which is important for the sparse reward setting because if it is not less than 1, then there is no useful distinction between the ASR and ADR metric.
- The ASR metric for MoPA-RL against the proposed method across all three tasks is extremely similar (at max, a ~5% improvement in the lift task, and only a ~1% improvement in the other two tasks). Although AEL and ADR are improved, the ASR is the most interesting metric when considering whether these methods can operate in tasks with obstacles where vision is important for completing it. This makes the results much less convincing: the authors should choose a task where MoPA-RL is not able to reliably complete the task to demonstrate the importance of this distilling approach (although the reviewer recognizes it is useful regardless of success rate since it removes motion planning from the loop).
summary_of_recommendation: This paper presents a novel and interesting approach for distilling motion-planner augmented policies into visual motor policies, and is very easy to read and is overall enjoyable. The reviewer really enjoys this paper and the contribution it offers, but the the experimental validation of the sample and computational efficient of this method due to the MoPA-RL expert and BC stages is not included and brings into question the efficacy of this method. There are also other important experimental details that should be included. Overall this paper has the potential to be a good contribution to the field (and is very accessible and easy to read), but requires further experimental validation for each of the stages and needs other experimental details to be included before publication.
issues: The authors need to explicitly validate the sample complexity for each of the stages of the proposed approach, and they also need to include the experimental details mentioned in the "Weaknesses" previously mentioned. If these are addressed, then this paper stands to present a good contribution and deserve publication.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a learning approach for manipulation tasks in obstructed environments. 
It has a hybrid aspect, where a sampling-based (thus model/state-based) motion planning algorithm is combined with a model-free reinforcement learning formulation. 
As an extension of a prior work, here the focus is on obstructed environments, for which RL methods are prone to fail, whereas sampling-based motion planners are effective. 
However, switching to a sampling-based motion planner during execution or learning is costly and also using them in the real-world is harder due to their dependency on full state knowledge. 
With this motivation, a two-stage learning approach is proposed. 
Initially a policy is learned based on the state-space representation using motion planning with RL.
Then, an asymmetric actor-critic framework is used to train a policy that is now observation-based, i.e. visual perception data. 
The initially learned state-based policy is imitated by behavioral cloning to provide (smoothed) expert data for training the observation-based policy.
The results demonstrate the efficiency of the proposed framework on three manipulation tasks.
strengths_and_weaknesses: The paper showcases the strengths of combining model-based methods with model-free approaches. In that regard, the motivation of the proposed approach is clearly explained w.r.t. the target scenarios, and the execution of the framework is technically sound and justified by several manipulation tasks.

The only slight weakness is the domain randomization experiments, as the distractors seem to be weak, and there isn't much analysis on the performance.
summary_of_recommendation: The paper proposes a novel approach combining state-based motion planners with model-free reinforcement learning methods for tackling challenging robot manipulation tasks. Even though it is an incremental update on prior work, the extension is well motivated and justified by demonstrating its performance on a range of scenarios. Overall, the paper is well structured and written. It can be further improved by a critical discussion on the limitations of the proposed approach and a deeper analysis on the domain randomization experiments. 
issues: My comment on domain randomization mentioned above (Strengths & Weaknesses) could be an improvement point.

Another recommendation is to include some limitation and discussion of the proposed approach. Such a critical evaluation of the overall architecture, where it fails, for which tasks it might be applicable (or not), what are lessons learned, etc. would be valuable for the community. Especially, for example the dependency of the success of the whole framework to the 'weight initialization' is highly interesting, which might be further elaborated.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a technique for transferring a learned policy from privileged state information to images. The method can complete cluttered manipulation tasks after about 1 million training steps (a few days of real robot time), whereas appropriate baselines are unable to make any progress. This method combines two key ideas from prior work: the MoPA-RL framework which can solve cluttered manipulation tasks with privileged state information, and asymmetric actor-critic, which allows for efficient transfer to a visual policy.
strengths_and_weaknesses: Strengths:

-An elegant combination of existing ideas

-Compelling results on difficult manipulation tasks

-Addresses an important problem -- efficiently learning a visuomotor policy for complex manipulation tasks

Weaknesses:

-No real robot experiment. Although the pandemic made this difficult, without such an experiment the suggestions/claims about sim2real transfer are not supported. There are many things other than textures, colors, and lighting which vary between simulations and the real world, such as physical properties, geometric differences, controller differences, etc. So the results for evaluating transfer are not convincing.

-It seems the actor still requires privileged state information, even during distillation. This is potentially a big limitation, because further training on a real robot is not possible without training a visual critic from scratch. To address this, perhaps one could train a visual critic during this whole process (using both step 1 and step 2 data, via simulation rendering + D.R.)? Then at the end you could conceivably fine-tune in the real world with efficiency boosts from all the pretraining.

summary_of_recommendation: With the addition of a real robot result quantifying success on a task using the proposed method, I would change to a strong accept. This paper combines existing methods elegantly and has strong results. However, a real robot experiment is missing, and I do not see any important insights or theoretical advances. There are a few components of the method which I believe limit its usefulness, which I discuss further in the Issues section.
issues: -Since the RRT is going to produce different paths for the same/similar start & goal based on the random seed, it seems like you could run into issues with BC based on this, especially if the policy network outputs a single Gaussian over the action vector. Does the method have a way to compensate for this?

-Why does BC cause smoothing, and seemingly do something smarter than averaging over actions (which could cause issues in some environments)? Why isn’t multimodality of actions an issue? Are there cases where it is? Answering some of these questions would strengthen this paper.

-I find the description of the MoPA vs direct action spaces as “different spaces” misleading. They have different domains, but they aren’t different spaces. A reader might think you’re adding additional dimensions to the action vector space, rather than expanding the domain of the action vector. 

-I also do not entirely agree that the different action spaces “lead to different transition function … and reward function” (line 117). For the actions within the direct action subset, the transition and reward are the same, inputs and outputs shouldn’t change. In fact, that’s the reason why transferring the critic works, because the transition/reward (and hence Q-value) functions are unchanged. This is a minor point that could be clarified in the text.

Questions:

-For table 1, how many environment steps of training were used? I could not find that number. Perhaps put it in section 4.3. I also wonder, more generally, how much success in MoPA pretraining is needed? Does it still work if you stop pretraining before the task is “mostly solved”?

-The discussion of HER was not sufficient. I did not follow the explanation of why it’s not applicable. From a quick read of the HER paper, it seems like it should be applicable.

Minor comments:

-On line 261, consider removing “extremely”, I would not consider one million time steps extremely sample efficient. Perhaps replace with “far more sample-efficient than ...”

-Table 1: in the Sawyer Lift experiment, you bolded “Ours” but “Ours (w/o BC smoothing)“ had a better score, I think that should be bolded instead.

-Table 1&2: consider adding Up and Down arrows to indicate which metrics should be high and which should be low. That would save the reader from having to go back and re-read what the acronyms mean.

-Consider running more seeds, 3 seems like too small of a number to have much confidence in.

-There is a comma missing on line 61. I would have written it “in terms of success rate, sample efficiency, and average path length”

-Section A.3 is not particularly helpful because it’s hard to see visually which is smoother. Perhaps show only one trajectory, or split the graphs up more. You could also try connecting the points with lines. More importantly, what is being graphed? Is it end-effector position? That’s not explicitly state anywhere, I just guessed.
review_title: null

Meta-review: All three reviewers were generally positive about this paper. They agree that the method is novel and interesting, albeit somewhat incremental and an "elegant combination of existing ideas”. Reviewers comment that the paper is particularly well written. However, reviewers do raise some important points that should be addressed in a rebuttal. For example, further details on the sample and computational efficiency are important for a full evaluation, and some insights into the limitations and failure cases would be helpful. I would like to see the authors address the comment from Reviewer acdh about whether the actor still requires privileged state information during distillation, and how this relates to implementation on a real robot. Reviewers comment on the lack of real-world experiments, particularly given that sim-to-real transfer features prominently in this work, which is an important limitation. However, I do not necessarily expect authors to provide real-world results for the paper to be accepted. Overall, this is a good paper but I encourage authors to address the reviewers' criticisms and major queries in a rebuttal, in order to maintain good reviews.

------

Following the reviews, the authors have addressed a number of issues raised by the reviewers, and new experiments with a visual critic have been provided. There is still concern about the applicability of the method to real-world tasks, given the lack of real-world experiments and the sub-optimal results for the visual critic. However, the reviewers still found the overall idea appealing and with potential for interesting future work, and so all three reviewers recommended to accept the paper.
Decision: Accept (Poster)