Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission addresses the problem of information asymmetry (IA) in hierarchical behavior priors as proposed in Tirumala et al. (2020). The role of IA is to limit the prior's access to observations or actions so that the behaviors that are captured are general rather than environment- or instance-specific (for example, restricting observations to proprioceptive sensing for simulated robots might result in the prior capturing general locomotion behavior). As such, the concrete choice of IA is a design decision usually performed by humans, and the main contribution of the paper here is to instead learn this asymmetry in a data-driven manner. The setup that is examined here is that of a high-level policy prior, and IA concerns the amount of past states presented to this prior. The proposed method employs soft attention on the entire history of states for learning IA.
strength_and_weaknesses: Strengths:
- Highly relevant topic with focus on a fundamental trade-off when distilling behavior priors.
- Well-motivated method that effectively automates this trade-off as part of policy training.
- Strong performance in the tasks considered.

Weaknesses:
- Information Asymmetry not only concerns the amount of history presented, but also the dimensions of the state that are relevant. It would have been nice to see a further application of attention across state dimensions, e.g., in a simple navigation task. This could then also involve environments for which the selected baselines have been originally developed.
- On a similar note, the considered environments are relatively simple, although the sparse reward structure renders them of course challenging for an RL algorithm. It would be great to see that the proposed method does not introduce regressions on tasks in which standard behavior priors work well.
- The presentation and writing is at times unclear and could be improved (see below).
clarity,_quality,_novelty_and_reproducibility: Clarity:
- The write-up makes liberal use of italics, which hurts readability as the reader's focus is constantly diverted. I'd suggest to use italics much more sparingly and reserve it for crucial results or insights (e.g., re-formulating the Theorems in 3.1. seems fine, but the highlights in the first paragraph of Section 4 are not helpful).
- I'm unclear on how $m$ is learned exactly; Section 4 could be improved in this regard. For example, from Eq. (3) it seems like the only thing that gets optimized is its entropy?
- The paper initially talks about sequential tasks (e.g., in 3.1), but the experiments are carried out in a transfer setting where experts for less complex task instantiations are available for initialization via BC.
- To prevent misunderstandings, it would be good to include the tube around the blocks in the environment rendering in Fig. 2(b)
- When defining losses, I would strongly suggest using a more widely recognized symbol, i.e., $\mathcal{L}$ instead of $\mathcal{O}$
- The introduction promises experiments over a "wide range of domains", yet only 3 tasks in 2 domains are considered.

Novelty:
- Highlighting and addressing a fundamental trade-off
- For the related work, consider Gehring et al., 2021 (https://arxiv.org/abs/2110.10809) which address a related trade-off in the context of unsupervised skill discovery and likewise propose a data-driven solution.
- The overall setup seems to largely follow Tirumala et al. (2020), with the addition of a softmax over past states

Reproducibility:
- Source code is not provided, but the authors provide their algorithm and hyper-parameters in the Appendix. For full reproducibility, more information about (or access to) the scripted expert policies would be required.
summary_of_the_review: This appears to be a good paper addressing a relevant topic. Due to issues wrt clarity and the limited experimental section, I'm leaning slightly in favor of rejection at this point.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an approach to the problem of transferring learned behaviors between tasks in a sequential setting, called Attentive Priors for Expressive and Transferable Skills (APES). They first present two theorems: a) with more input variables, the shift of the input distributions in terms of KL divergence gets bigger and b) conditioning on more variables leads to better expected knowledge distribution in terms of the KL divergence between output distributions, which together suggest an expressivity-transferability trade-off. The authors then suggest APES, which learns soft masks for input to each module in the KL-regularized hierarchical policies. They test and perform analyses of their approach in the CorridorMaze and Stack environments to show the empirical effectiveness of the approach.
strength_and_weaknesses: Strengths

- I like the presentation of the expressivity-transferability trade-off using the theorems in terms of KL divergence, and the attempt to connect them with the proposed method.
- They analyze the experimental results in multiple ways, which could help understanding the empirical effectiveness of the proposed approach.

Weaknesses

- I'm concerned that by using the soft masks to allow them to be trained, the method could lose most of its theoretical connection with Thm.3.2 and thus the expressivity-transferability trade-off. It is because, unless some mask is strictly zero (or near-zero in practice) the information the corresponding variable carries does not go away. This is probably supported by Fig.5, where the minimum mask value is $\approx \exp(-3.1) \approx 0.045$. Their corresponding variables may not play big roles in the prediction, but it's hard to say that they are completely ignored.
- The scales of the soft masks can largely be affected by the scales of the input variables. I'm curious if they are properly normalized in the experiments.
- The writing could be improved in some ways. In Eq.3, $\pi$ and $\pi_0$ are not conditioned on the masks and $\mathcal{O}_{\text{APES}}$ does not take the masks as its input. The boldfacing of the inputs in the text and Thm.3.2 do not match. The second sentence of Sec.2 is missing an "and".
clarity,_quality,_novelty_and_reproducibility: - The presentation is basically clear, but there are some points for improvement (please see my comment above).
- The quality could be improved by strengthening the theoretical or empirical connection between the presented theorems and the proposed method.
- While the masking of input variables is not uncommon, the proposition of the masking method for the KL-regularized RL based on the theorems with KL divergence has some novelty.
summary_of_the_review: I find the connection between the theorems on the expressivity-transferability trade-off and the proposed method somewhat interesting, but my major concern is that the connection may not hold in the case of the soft mask learning.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of efficient transfer by combining two methods in the literature: hierarchical reinforcement learning (HRL) and KL-regularized RL. The paper argues that a key element to the transferability of the policies is by considering information asymmetry between them. To achieve this goal, the authors propose to employ soft attention on the inputs and learn this attention mask through an entropy objective on its weights (equation 9). To motivate information asymmetry, the authors two theorems on covariate shift and expectation distillation. The paper performs experiments on two domains, one a toy task and a harder stacking task, and report improved performance compared to other baselines.
strength_and_weaknesses: # Strengths 
- The paper tackles an important problem by combining two existing approaches in an interesting fashion
- The paper compares with a large set of baselines
- The theorems presented could be of interest to the community

# Weaknesses
- The claims do not seem necessarily true. In particular, the authors claim that successful transfer relies on information asymmetry, which may or may not be true, depending on the method one considers for transfer learning.
- The relation between the theorems and the ability to transfer is not necessarily direct. Some nuances would be important here
- The empirical setup is quite complex and the number of seeds is very low, bringing into question the reliability of the results.
clarity,_quality,_novelty_and_reproducibility: The paper could gain a lot in terms of clarity. The idea of a expressivity-transferability trade-off is mentioned many times, before we finally understand what the authors are referring to. The theorems present the idea of covariate shift and combine it with the idea of transferability. I am not convinced that the two are equal, as transferability may depend on many more things than covariate shift and may be reflected in different difficulties that the agent has to overcome. The authors give the example of self driving cars to motivate information asymmetry, but this is only one example. How would IA be considered when transferring between two opponents in a game of Go? Is covariate shift the right way to look at transferability here? And how would covariate shift be related to IA in this case? Those are important nuances that the paper does not consider: it is written as though we assume that IA is necessary. Is the IA implemented solely through the masking in equation 9? It seems like there would be many works in the literature that use similar ideas, but are not compared to or clearly referred. Similarly, the idea of an initiation set and more recently interest function in HRL is an important parallel idea that is not mentioned. 

In the experiments, why do start from expert trajectories? It seems like the proposed approach is straightforward and could be implemented in a straightforward setting, so that we clearly understand the benefits. For example, the proposed method is compared with methods that do not leverage expert trajectories. Even if they are trained longer, this seems quite unfair. Moreover, the number of seeds is only 4, which only brings additional questions as to what we can get out of the experiments. 
summary_of_the_review: The paper proposes an interesting combination between HRL and KL-regularized RL and motivate the idea of information asymmetry as a way to better transfer across tasks. The paper relates the idea of information asymmetry to transferability, which is not always true. Moreover, transferability is related to covariate, which once again is not always true. More nuances are needed. For the experiments, the setting seems more complex than it should and is only evaluated with a few seeds.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper targets the problem of skill-transfer in reinforcement learning by focusing on learning information asymmetry from data. The key contribution is formalizing the notion of information asymmetry as a unifying perspective on development of skills autonomously, and providing a neat knob that can be turned to tradeoff expressivity of skills with transferability. Experiments on simulated robotics tasks like block stacking and maze navigation show the utility of the proposed framework in skill emergence and transfer. 
strength_and_weaknesses: Strengths

1. I found the unifying perspective of Information Asymmetry to be very interesting in connecting skill-learning algorithms that condition on different aspects of the observation space, and in analyzing them under a common umbrella. 

2. The experimental settings of maze navigation and block stacking are insightful because the skills in these case are intuitive. Also, the tasks are non-trivial to solve which makes the comparisons with baselines clear - in that the baselines are not able to solve some of the tasks which the proposed algorithm is able to. 

Weakness

1. The theoretical sections are a bit hard to follow through. In particular it is not clear whether the expressivity-transferrability tradeoff emerges due to the specific sequential nature of the problem being considered and whether the tradeoff would still exist in the usual multi-task style setup described in section 3.1

2. It is unclear what Figure 6 is trying to show: it seems like skill-level exploration, but without the corresponding visualizations of robot behavior it is unclear whether these diverse trajectories are "meaningfully diverse" 

3. Some related works in model-based skill-transfer are not dicusssed (see A and B below)

A) Xie, Kevin, Homanga Bharadhwaj, Danijar Hafner, Animesh Garg, and Florian Shkurti. "Latent Skill Planning for Exploration and Transfer." In International Conference on Learning Representations. 2020.

B) Shi, Lucy Xiaoyang, Joseph J. Lim, and Youngwoon Lee. "Skill-based model-based reinforcement learning." arXiv preprint arXiv:2207.07560 (2022).
clarity,_quality,_novelty_and_reproducibility: The contributions are novel to the best of my understanding. The paper is clear, easy to follow, and of high quality. 
summary_of_the_review: My main concerns with the paper are minor and relate to proper explanation of some theoretical and empirical results, and some discussions of related works. These can be easily incorporated in the revisions, and as such I am recommending that the paper be accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the sequential transfer in RL and proposes to combine two approaches for better transfer: two-level hierarchical RL and KL-regularized RL with a learned prior. It argues that the use of Information Asymmetry, in which different modules of the method have access to different information about the problem, is a key to learn transferrable hierarchy and priors. As opposed to previous work, it learns how to create the Information Asymmetry by learning a masking function.

The paper has some theoretical justification showing a tradeoff between expressivity and transferability. It also performs empirical studies.

Three out of four reviewers are positive about this paper, and one is on the weak negative side. They found the unifying framework and the expressivity-transferability trade-off interesting (Reviewer ia3Q, RbDM), the automating the trade-off well-motivated (Reviewer Dj5g), and the experimental results convincing (Reviewers ia3QH, Dj5g).

There are some concerns though such as the unclear presentation at points (especially in the theory part), not a strong connection between the theoretical parts and the empirical ones, and the use of small number of seeds (4) in the empirical part.

Several of these concerns have been addressed in the revised version of the paper, and most reviewers are satisfied. There were long discussions between Reviewer pBgA and the authors, which did not end in a final agreement. 

After reading the paper myself, I have some suggestions:

- The number of seeds is four (4). This is small. Although this choice is aligned with the current practice of a subset of the deep RL community, I agree with Reviewer pBgA that bad practice of others is not a good justification for its continuation. The paper does not provide a statistical significant test on the results.

Looking at Table 2, I see that there is relatively good gap between most results, considering the (inaccurate estimate) of the standard deviations, perhaps except for APES vs APES-H1 in 4 block domain. Therefore, I am not too worried about the validity of the results, but one cannot be completely confident given this small number of seeds and the lack of a formal hypothesis test.

I strongly recommend the authors to re-run their experiments with more seeds. My request should be interpreted as that I trust authors to do this change in their final version. If this was a journal paper, I would recommend a major revision with this as one of the main requests.

- APES happens to find a masking that makes it similar to APES-H1 in the tested domains, as shown in Figure 5. The authors rightfully mention that this is domain-dependent and APES' benefit is that it automates the process of finding the right masking.

Is there any domain that APES finds a less trivial masking that doesn't focus on the most recent past? Having such a result would make the paper's argument on the importance of automating the process of finding the mask more convincing. This is a suggestion to make the paper stronger, and is not a requirement.

- The experiments in Table 2 look at APES-Hk with k being in the set of 1, 10, and 20 (as well as APES-S, which seems to be equivalent to k=0). It would be insightful to have a wider range of ks and present them in a graph. This provides a clear visualization of the tradeoff compared to showing only 4 points.

Overall,  this is a reasonably good paper. As most reviewers agree that this should be published, I would also recommend its acceptance with the understanding that the authors would improve their experiments.
justification_for_why_not_higher_score: Not enough enthusiasms from reviewers (two 6 and one 8; one 5).
justification_for_why_not_lower_score: It has novel contributions that can benefit the community.
Decision: Accept: poster