Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a hierarchical Reinforcement Learning method, where the high-level controller chooses skills that maximize the reward while staying close to a weighted combination of all skill priors. The skills and skill priors are obtained by fitting one skill prior to each labeled dataset using a VAE. A key contribution of this work is the adaptive skill prior, which learns to decide the weighting among skill priors that leads to the highest critic value. The method is evaluated in 3 simulated environments and is shown to be advantageous in terms of sample efficiency and final model performance.
strengths_and_weaknesses: Strengths:
- The components in the proposed approach are mostly well explained, and the whole idea is sound.
- The Adaptive Weight Module is a novel method to assign optimal weightings among priors to generate skills with high values.
- The visualization of weights along a rollout trajectory is very intuitive.

Weaknesses:
- The experiment environments are from a similar domain. In particular, the point mass and ant differ mainly in terms of low-level dynamics, which mainly challenges the skill learning stage (not the main contribution of this work). Thus, the conclusion drawn seems a bit weak.
questions: - As talked about in appendix A.2.1, the skill encoder and decoder are essentially learned from a single primitive and expected to generalize to other primitives. Would this make the skill latent space z more tailored towards the first task?
limitations: A major limitation of hierarchical learning with fixed primitives is that the task could be impossible to complete if the primitives are not comprehensive enough. Also as the authors have discussed in the paper, separate datasets of different primitives are required. It's unclear how this method will perform in harder downstream tasks that require more primitives.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper extends the prior work on skill prior-regularized RL by learning a skill prior for each skill. When K skill labels are available from the offline data, we can learn K skill polices using VAE as well as K skill priors, which tell an agent whether each skill is plausible to execute given a state. To intermingle K different skill priors into one composite skill prior for downstream RL, a weighting function of K skill priors is learned to maximize the estimated Q-value when following the composite skill prior. One benefit of the composite skill prior is that it enables simultaneous skill composition, e.g. navigating towards a goal while avoiding an obstacle. The experiments demonstrate that the proposed method can effectively combine up to three skill priors, and tackle downstream tasks by not only exectuing one skill at a time but also executing composite skills.

strengths_and_weaknesses: ### Strengths

- Figure 3 and 4 are very intuitive and well explain that how the proposed adaptive weight module learns to regulate the downstream task policy.

- The proposed method of weighting different skill priors is novel and the empirical results show the effectiveness of using multiple skill priors quantitatively and qualitatively.

### Weaknesses

- In L28-32, the claim about the single skill prior not being able to handle composition is not very intuitive. Also, Figure 1 does not seem relevant to this context. If this composition is about a mixture of two skills (simultaneous execution of two skills), it should be more explicilty described.

- "the learned policy" sounds weird in some contexts, e.g., L43, L46.

- The assumption of having task labels for trajectories can limit the applicability of the proposed method. The major benefit of the prior work, SPiRL, was in being able to leverage unstructured data.

- It might be good to show the performace of the uniform weights for multiple skill priors to see the effect of the proposed adaptive weight module. Given that only push skill prior can achieve comparable results with ASPiRe in Figure 6, this simple baseline may work as well.

- Is the MCP baseline identical to using the proposed composite skill prior as a policy? Otherwise, it would be important to include this baseline.
questions: - Most papers are cited as arXiv papers, which are conference or journal papers. Please use appropriate citation.

- There are many typos (including notations) and grammatic errors. The writing generally needs some improvement.

limitations: The limitations and potential negative societal impact are well addressed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the problem of composing skills from multiple offline datasets. Aspire learns multiple priors from different offline datasets and then learns a set of weights over the priors and a composite prior predictor during online training on new tasks that require application of multiple skills. The method outperforms BC, Multiplicative Compositional Policies (MCP), and SPiRL on a set of 3 Maze tasks that require the agent to compose pushing, avoiding, and navigating actions. 
strengths_and_weaknesses: - Strengths
  - Clarity: The abstract and introduction state the motivation and setup clearly. I found the method section up to 3.3 to be straightforward and concise.
  - Significance: The authors demonstrate that aspire can effectively learn composite policies from offline data of diverse and distinct skills. These results hold even in the presence of distractor skills. Work in this direction enables datasets to be leveraged for a broader set of target control tasks.
  - Quality: The evaluation is thorough: the authors test multiple relevant baselines (MCP, spirl) and show the effectiveness of the method in the presence of irrelevant skills.
  - Originality: To my knowledge, the algorithm and associated losses for a policy regularized by composed primitives is novel.
- Weaknesses
  - Clarity: The explanation of the approach could be made more straightforward.
    - It would be useful to clarify what is learned offline vs online in section 3, especially 3.3. 
    - The motivation behind the skill prior generator is difficult to understand because there is no reference to the skill prior generator in the introduction text of section 3 or in figure 2. Based on the previous sections, I expected the adaptive weight module to be a convex combination of the individually learned priors. 
  - Quality:
    - A baseline that composes the learned priors with a mixture of experts model (i.e., $G = \sum_{i=1}^Kp_a^i(z|s)$) would clarify the need for the skill prior generator.
    - Figure 6 shows that the push prior accounts for much of the success of the model and Figure 4 shows that, even for maze traversal, the push prior is contributes the most weight to the policy. This counterintuitive result is not explained.
  - Significance:
    - The baselines and prior work present results on more challenging control tasks. E.g., MCP studies tasks on biped, humanoid, and t-rex in addition to ant. SPiRL studies FrankaKitchen.
  - Originality: Section 3.2 could make reference to SVGD [1] and SVPG [2]. SVPG has a similar prior regularization, but for policy learning directly as opposed to skill primitives.

[1] https://arxiv.org/abs/1608.04471
[2] https://arxiv.org/abs/1704.02399
questions: - Is $G_\mu$ learned offline and $\omega_\sigma$ learned online?
- What $\omega$ is $G_\mu$ conditioned on in equation 6? Should it be $\omega_i$?
- Are all of the baselines trained with the same number of parameters?
- Lines 305-307 state that the push agent fails to complete the setting requiring traversing the maze. But in figure 4 the weight of the push prior is highest and sometimes 1 while traversing the maze. What accounts for this counterintuitive behavior? Why is the push prior so much stronger than navigation in figure 6?
limitations: The authors describe that their algorithm relies on the presence of labeled skill data for offline training and depends on a set of specified skills in advance.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors present a method for 1) extracting skills from offline demonstration datasets and 2) using these learned skills for accelerated learning of downstream tasks, where the new task might often require composing two or more skills (either concurrently, or sequentially). For (1), the authors first train a VAE on action sequences from the offline dataset to learn a latent space of skills. They then extract primitive skills priors P^i_a(z|s_t) (one for each dataset D_i) by minimizing a KL divergence term (line 140 in the paper). Finally, when learning a new task, the policy that is being learned is constrained to stay close to a mixture of skill priors through a KL divergence term. The weights of this mixture (called the adaptive weight module) is learned so as to maximize the Q-values of the downstream task. 

The authors evaluate their method on three simulated navigation tasks. In two of the tasks, two skill priors need to be combined for learning a new task, while there are three skill priors in one of the tasks. The authors provide comparisons against SPiRL (the method on which they closely build), learning from scratch, and naively combining BC with RL, and show that their method significantly outperforms these other techniques. 
strengths_and_weaknesses: Strengths

- The paper tackles an important problem – how can we compose a diverse set of previously learned skills to solve new tasks quickly? 
- The paper shows strong performance when compared to the mostly closely related work - SPiRL. 

Weaknesses

- The proposed method has a lot of moving parts: skill encoder / decoders, skill priors (one per skill), downstream policy, Q-function, skill prior generator, adaptive weight module, and so on. All of these various components are closely dependent on each other, and makes me a little concerned about the overall training stability of the method and sensitivity of hyperparameters. 
- The experiments are fairly small-scale. The main point of the paper is that it allows combining multiple skill priors to solve the tasks, but two of the experiments involve combining only two skill priors, and one set of experiments involves combining three skill priors. The experiments are also all in navigation domains, so it is unclear if the method will work on other domains, say robot manipulation or Atari games. 


questions: - Since I have concerns regarding the overall training stability of the method (due to the large number of moving parts), can the authors share in detail what kind of hyperparameter sweeps had to be done? Generally speaking, a hyperparameter sensitivity analysis would be very insightful for this work. 
- Did you try running the method on domains other than navigation? What did you find? Even negative results can be useful to look at here. 
- As mentioned in the introduction, one of the relevant related work is PARROT, but there is no comparison provided to this method. Would it be possible for the authors to run this comparison? 

I am open to updating my rating if the authors can provide answers to these questions / make edits to the paper that address these concerns. 

limitations: I think the limitations section in the paper is very limited. The only limitation that the authors mention is the difficulty of collecting useful prior dataset, but I think the work has other limitation as well (mentioned in the weaknesses and questions section above) that would be worthwhile to discuss in more detail.
review_title: null

Meta-review: metareview: This paper proposes a method to adaptively combine and use skill priors for reinforcement learning. The setting is very practical and the method proposed is novel and effective empirically. The main concerns from the reviewers were around (1) the experimental setup being too narrow and simple, and (2) the clarity of the paper. The authors added in an additional robotics experiment which partly addressed (1) and provided clarifications in their response to address (2). Overall, this seems like a solid contribution idea-wise that will inspire more work in this direction. I highly encourage the authors to revise the paper in terms of clarity (based on their rebuttal responses) as well as to consider adding a comparison to PARROT (as suggested by reviewer iPXJ). 
confidence: Less certain
Decision: Accept