Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this paper, the authors have tried to combine the advantage of imitation learning (IL) and reinforcement learning (RL) into a single framework. Mainly authors have focused on Advantage-Weighted Actor-Critic (AWAC), which learns good initial policy from demonstration, and QT-Opt, which can learn a task from scratch with a large amount of training data, to develop AW-Opt. To make AW-Opt fair well with IL plus RL, authors have modified data sampling, exploration strategy, and action selection in the Bellman equation. Authors have shown that these modifications helped robotic manipulators to solve 5 different robotic tasks, consist of both simulator and real-world, when provided with initial demonstration and then finetuned using RL.
strengths_and_weaknesses: Strengths
* Figured out the limitation and advantages of each technique and combined them well to figure out technique which works fairly well for IL and RL setup
* Provide ablation study to show the importance of each strategy
* Carried out experiment both in simulation and real world robotic platform.

Waekness
* It felt to me that method was overly optimized for the task in consideration. I would love to know how the method works for wide range of IL  + RL task, ranging from learning policy for solving video games to complex manipulation (like opening doors, cabinets, etc) 
* No good explanation for choosing some of the hyper parameters, like 20-80 % selection in Hybrid Actor-Critic Exploration
summary_of_recommendation: I am providing the weak accept based out of the following +ve and -ve points

### +ve points
* The paper proposes new way of performing IL and RL based out of vision inputs in sample efficient manner. 
* They have shown that the method they are proposing, is able to solve 3 robotic task in real world based out of demonstration collected for those tasks
* Detailed ablation study is provide which shows that each of the strategy is contributing significant amount to the overall success

### -ve points
* seems method is overly optimized for task in consideration
* explanation for some of the hyper parameters are missing
issues: One of the major concerns I have is that in this paper authors are proposing a new algorithm for solving IL + RL problem, but has only shown result based out of the one vision-based robotic grasping task with some amount of variation in what needed to be grasped. However, it would be interesting to know how this algorithm works for some complex manipulation tasks like opening microwave, cabinet, door or push stacking objects.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a hybrid method AW-Opt, built on two previous methods AWAC and Qt-Opt, for scalable robotic manipulation with imitation learning and reinforcement learning.
The resulting method is driven by empirically analyzing the design choices in both baseline methods and developing modifications that address the issues in the original components. Specifically, the proposed changes are balanced data for training both the critic and actor, improved exploration by switching between the actor and CEM policy, and bellman update target using maximum Q value optimized by CEM with "elites" from the actor. 

With extensive experiments in simulation and real-world, the paper demonstrates that the proposed method significantly improves over the baselines AWAC and Qt-Opt in challenging tasks where offline learning and online finetuning are needed for achieving high performance. 
strengths_and_weaknesses: Strengths
1. The presentation of the paper is clear
2. The modifications proposed in this paper are well motivated and they are effective as shown by extensive experiments
3. The complete algorithm archives strong empirical performance

Weaknesses/Suggestions
1. It missed some details that are important for interpreting the assessing the experimental results: 

  a. Most figures showed the success rate but did not give details on how many trials the rate was based on.  
  b. All learning curves were shown without mentioning what the line and shaded region represent. Is it the mean and standard deviation? If so, how many runs/random seeds were they based on?

2. Can the authors please clarify the following design choices in the paper?  
  a. The “switcher” exploration in Section 4.3 used 80% critic and 20% actor. I wonder if this choice is backed up by some experiment. How sensitive is the result to this choice of ratio?  
  b. The offline datasets for the five tasks contain different positive/negative ratios. I wonder if this is a somewhat arbitrary choice (in terms of that the extreme case of 100% negative is bad but other ratios are fine). How sensitive is the result to this choice of ratio when constructing the dataset?  
  c. The loss function shown in the appendix has weights for subactions. It would be great if the values of the weights are given.

3. The legend in Figure 6 does not quite match the text description. “AW-Opt - MaxQ” in the figure becomes “AW-Opt without ActorCandidate” in the text. It seems to me that they are not the same thing: we recover the AWAC target value without the MaxQ but “without ActorCandidate” can be interpreted as AW-Opt + MaxQ. It confuses me as to which one the paper meant to refer to. 

summary_of_recommendation: I base my recommendation mainly on the strong experimental results and the potential impact of the proposed ideas and insights. In the robotics domain, simple but effective methods often have great impacts. This paper falls into this category. 
issues: Please refer to the items discussed in the weaknesses/suggestions section. My main concern is in the missing details of the learning curves.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper, on the one hand, investigates experimentally why two existing RL methods have difficulties solving a bin picking task with a sparse reward for pick-up success.
On the other hand, the authors compare performance improvements by introducing small changes in three different aspects of the RL problem:
1) Data balancing for training the Q function and the policy.
2) Exploration with a learned actor policy vs optimizing the learned Q function. 
3) Different ways of calculating the Bellman update.

In the experiments, the influence of these changes both in terms of asymptotic performance and the capability of the system to include demonstration data is investigated.

strengths_and_weaknesses: Strengths:
- I appreciate the research question of figuring out why existing methods have difficulties in incorporating demonstrations for later fine tuning.
- Interesting ablation studies/comparisons. 
- Good video!

Weaknesses:
- No real methodological advancement. Proposed approach consists of only small tweaks to existing methods.
- Results don't allow a clear conclusion (QT-opt can also incorporate negative demonstrations).
- The choice of QT-opt and AWAC seem rather arbitrary.
summary_of_recommendation: I am not sure what to think about this paper.
The gained insights are not really surprising.
It is well-known that RL is difficult due to the exploration problem of collecting meaningful data, the credit assignment problem and the learning problem itself (whether it is Q functions or policies, even given optimal (but imbalanced) data).
What I have learned from the paper is that small modifications can make a large difference without changing the underlying learning methods significantly. But this also implies that the problem is not yet well understood yet and I am unsure if the findings of the paper are applicable to other domains/tasks as well.
Nevertheless, the approach of investigating the data balance problem, exploration and Bellman update is interesting and relevant.
I would recommend ICRA instead of CoRL.
issues: - "Then, during the online phase, QT-Opt is faced with a difficult exploration problem". Why QT-opt particularly?
- The fact that the episode-level random switcher is the best in Fig. 4 seems to be caused by the fact that it introduces the most useful randomness for exploration compared to CEM exploration only? How has the 80% 20% ratio been chosen?
- In Fig. 5 why is the grey curve better than the yellow one given the fact that the blue one outperforms all? Maybe a clarification how actor candidate differs from actor mean would be useful. (particularly I don't full get what it is meant with "maxQ + ActorCandidate combines the actor with CEM by using the actor-predicted action as an additional candidate in each round of CEM.").
- Why not include non successful demonstrations for QT-opt (as done in Tab. 1)? It seems arbitrary to not do that for most of the experiments and indeed in sec. 5.4 QT-opt performs best on task 3.
- Line 13 in Algo 1 does not make sense (typo, I think).
review_title: null

Meta-review: Reviewers are mixed. On the one hand, all reviewers appreciate the study on understanding why two well-known existing algorithms struggle to incorporate demonstrations well, and on proposing specific improvements to these algorithms accordingly. On the other hand, two reviewers question whether the conclusions would apply to other tasks, beyond the grasping task studied in this paper, and Reviewer qeFD notes a number of other minor potential weaknesses. Authors should address these issues in a rebuttal.

-------

Following the reviews, the authors have provided further experiments and have addressed the issues raised by the reviewers. All three reviewers now recommend acceptance of the paper. For the new navigation experiments (Figure 10) in the final paper, I encourage the authors to perform these over more than one random seed, as has been done for the manipulation experiments.
Decision: Accept (Poster)