Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The main contribution of the paper is to combine the skill-based RL with the demonstrations-based RL. The demo-based RL can suffer from the covariate shift problem. In contrast, the skill-based RL still needs to explore a relatively large options space. The idea is to combine both with a discriminator weight to avoid their disadvantages. Simply speaking, if the current state is covered by the demonstrations, the learned policy should behave like BC. If the current state is out of the demonstrations, the learned policy should behave like SPiRL.
strengths_and_weaknesses: The idea is interesting and clearly explained in the paper. The experiments are thorough. However, in the paper, the skill is learned beforehand and fixed during the downstream training. I think that all three experiments can be considered standard RL problems with a small set of "actions".  These kinds of problems are difficult for the agents without considering the hierarchy of tasks.  But the experiments are only skill sequencing problems if skills are well trained. I noticed that the low-level policy described in the paper is dependent on the state. According to the authors, the one in SPiRL is, however, not dependent on the state. Is this also one reason why the proposed method outperforms SPiRL? Furthermore, compared to SPiRL, the proposed method still needs a set of demonstrations for different tasks. 
summary_of_recommendation: Overall, the paper is well written. The idea is interesting. The question is whether this kind of method can be used or generalized for non-skill-sequencing problems. I think that it might be also difficult to extract skills automatically from demonstrations.
issues: It would be good if the authors can compare the proposed method with SPiRL with the same set of skills. They should all depend on the states. If this is already the case, the authors should clarify it in the paper. (If I missed some sentences in the paper, please just correct me.) 

I would suggest giving more details in the paper (but not only in supplementary materials) about the state and action definition for each experiment and what skills are learned. For example, for the maze navigation, it is not clear in the paper whether the goal position is in the state or not.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper describes two mechanisms, which when combined can provide a good imitation learning algorithm to quickly learn a skill-based policy for a novel skill. 
There are two main components:
A pretraining method (Skill representation Learning) for skill extraction from unstructured experience data.  This produces \pi(a|s,z) a skill-state-conditioned policy that will be used as the low-level controller later on.  This is learned through a variational learning objective with a regularization that forces the task embedding to be consistent during a particular trajectory.

A learning method train a policy purely from demonstrations (without a reward function) in a demonstration-data-efficient manner.  This method requires the training of:
D(s) - a demonstration/non-demonstration discriminator.
q_z(z|s), the task-based policy being learned.
P_a(z|s) - not entirely sure what this is, perhaps a model that provides z-probability conditioned on s_t but over the whole dataset? Or perhaps over a particular action sequence.... This could warrant being clarrified.

Once these are trained, a final policy pi_theta(z|s) is trained by maximizing a discriminator reward for a given task, and making sure that the policy is following the posterior predictor (q_z) when on-distrubition relative to task data, and following p_a(z|s) when out of distribution relative to task-specific data.

The authors show that this approach can find controllers that can complete more than 3 subtasks on average after 1-2 million training steps on the environment.  Thes tasks have no reward provided by the environment, and even if there was a sparse reward the exploration problem would render them unsolvable.  
The authors show that given the various models above one can learn a new policy pi_theta(z|s) which can operate in the skill-space MDP to solve demonstration-defined tasks.
strengths_and_weaknesses: I found the paper pretty clear to read, straight-to-the-point, and the experiments quite convincing.  The method makes intuitive sense, and although it has a lot of moving parts each one seems to be well-justified and not overdone.  I also found the related work well put together and providing sufficient context to situate the proposed work relative to other papers.

My main comments on weaknesses would be that because there are a lot of moving parts, I had to go back and forth quite a bit, and I still have a couple elements that I'm not quite sure about (p_a(z|s) for example).  I've put specific fixes like this in the 'Issues' section.

summary_of_recommendation: I recommend acceptance, but please try and fix the issues mentioned in the next section.
issues: 1. Although overall the paper was easy to read, it would be very helpful to have a small table or some form of recap of all the different models: skill prior, skill posterier, skill inference model, policy etc. and what exactly each one learns.  I spent a lot of time going back and forth between sections to try to decipher what is going on.  Could you provide some form of recap before or around Eq. 4 to summarize all these, perhaps in some sort of table?

2. Small nit but on l. 162 you introduce the skill posterior but don't define the zeta subscript, is zeta a skill?  That was my deduction but this cost me some concentration points and could be easily clarified.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work builds on top of SPiRL (Pertsch et al. 2020), with the insight that SPiRL encourages policies to explore many behaviors not relevant for the downstream task we actually want to solve. To address this, the authors propose a reasonable suggestion: fine-tune RL on the target task, but use the high-level skill spaces learned by SPiRL to benefit from the data efficiency and restricted search space. 

The recipe is as follows: just as in SPiRL, first learn a VAE for short-horizon actions. This yields a skill posterior q(z|s,a), and a skill-conditioned action decoder pi(a|s,z) (which is later re-used as an “action decoder” for the policy). There is a small modification where the state is also passed to the “low-level policy”. One can also train the task-agnostic prior p(z|s) in this step, by predicting the inferred posteriors directly from state.

Next, *repeat* this for the target demo data, learning a task-specific skill posterior q(z|s). The task-agnostic prior p(z|s) and posterior q(z|s) are used in regularizing the high-level policy pi(z|s) being trained with RL. The full RL objective adds 3 distribution matching losses on top of the existing RL reward optimization: the raw reward is using a GAIL-style discriminator reward against target task demonstrations, encouraging the high level policy to match the posterior, and encouraging the policy to also lie within the prior (because the “skill posterior will often provide incorrect guidance outside of the demonstrations’ support”). Some of the loss coefficients are tuned automatically via dual gradient descent. Learned discriminator used to distinguish whether state is within support of the target task demonstration, and is thus used to tune which loss (posterior or prior regularization) matters more. 

The authors evaluate this method on 3 simulated tasks, showing that it outperforms SPiRL and some other baseline algorithms (e.g. BC+RL) by a large margin.

strengths_and_weaknesses: 
Strengths:
1. Each of the pieces of the method (the use of high-level skills, the use of discriminator to detect OoD, the use of a task-agnostic skill prior to “catch” states outside of demonstration support, the use of a skill posterior to keep it close to the task-specific skill distribution) are reasonably motivated.
2. The re-use of various skill encoder components to form reward constraints and “loss switching” mechanisms is interesting. 

Weaknesses:
1. Although the extensions to SPiRL presented here are interesting, there are too many moving parts to this method to make it practical. Depending on how one counts the pieces of a VAE encoder/decoder and GAN as one or several models, there are at least 4 models that need to be trained for this to work, each with their own set of hyperparameters. Furthermore, certain pieces are re-used in different places (e.g. the state discriminator is used for both reward shaping and switching between the penalizing the skill prior or skill posterior), which sounds like it makes it hard to debug if something is not working.
2. No real robot results. Tuning on-policy RL algorithms on real robots is challenging, and often SAC alone has enough hyperparameters to make things quite difficult and collect sufficient data. A demonstration that this method can be plausibly scaled to real robots is what I’d like to see.
3. A discriminator is used to switch between penalizing posterior / prior, but a discriminator in and of itself does not guarantee out-of-distribution errors. In fact, early on in training the state discriminator model may be misspecified.
4. It is not easy to assess the effectiveness or completeness of the latent skills until one trains the policy itself. 
5. The paper’s admission that the task-specific skill posterior is not always accurate outside of the support of the task-specific demos seems like a major flaw in the method, requiring the user to bolt on several additional losses like the skill prior and discriminator loss. Going through this trouble of learning yet another couple models doesn’t imbue the robot with new capabilities beyond being able to solve the target task better, so it begs the question of whether one model can simply be tuned to solve the task well (for example, simply using PPO in simulation).


summary_of_recommendation: I do not spot major theoretical flaws with each of the components being proposed, but the method feels like 3 bolted-on learned components needed to get a specific idea ("fine-tune SPiRL with RL") to work well. It feels like to me that there can be a simplification of the method that makes it more practical for real robots. 
issues: The fact that SPiRL and the proposed method are the only methods that do okay on the Kitchen and Office environments suggest to me that the baselines have not been tuned very well. For instance, it’s surprising that SAC and GAIL + RL do so poorly on the Maze Navigation environments.. Relay Policy Learning (Gupta ‘19 CORL) achieved high success rates on the kitchen environment using what essentially amounts to behavior cloning, which suggests that there is something sub-optimal in the baseline implementations of this method. I would like to see some convincing analysis as to why the baselines do so abysmally, given that other papers have reported decent goal-conditioned behavior cloning baselines.

I would like the authors to address the problem setting and data requirements in which this method is an ideal choice over other learning approaches, in particular ones that are simpler like GCBC or PPO brute-forced at a sim environment. For instance, is this done in the context of solving in sim and then transferring to real? How much target task data is available? What is a passable amount of overlap between the task-agnostic dataset distribution and the target task distribution?
review_title: null

Meta-review: === comments before the discussion ===

The paper presents a demonstration-guided RL method, that leverages pre-trained skills. 
In the initial review, the scores are split. 
I recommend the authors to carefully address the concerns raised by reviewers. For example, following points seems to be important points:
- Reviewers 5LPL and fuNi shared the same impression that there are several moving parts in the proposed method. 
- The reviewer fuNi raised concerns related to baselines in experiments, which is important for evaluating the method. 
- The reviewer fuNi also raised concerns related to the out-of-distribution errors.

=== comments after the discussion ===

The author response clarified unclear points in the paper. Although there are several moving parts in the proposed method, the empirical results show the good performance in the kitchen task, and the pros outweigh the cons of the paper. Thus, the area chair recommends the acceptance of the paper.
Decision: Accept (Poster)