Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method, IRM, that finds appropriate skills for a given target task by matching the unsupervised-learned intrinsic reward and the extrinsic reward of the target task.
The matching procedures don’t require environment samples as in conventional fine-tuning approaches but assume black-box access to the ground truth reward function of the target task.
EPIC loss is used to correctly measure the behavior similarity between the intrinsic reward and the extrinsic reward.
In their evaluation, IRM shows the same order of zero-shot and fine-tuning performance with the baselines.
strength_and_weaknesses: ### Strength
- The proposed method can be a powerful tool that can facilitate zero-shot deployment of unsupervised trained skill policy into actual target tasks.
- The idea of matching intrinsic reward and extrinsic reward seems novel and the use of EPIC loss is well-motivated. The analysis supports that EPIC is effectively matching the intrinsic reward and the target task reward.

### Weakness
- Missing baseline
    - Given access to the extrinsic reward function, one can use it to relabel pre-training time buffer data. Then, the skill that is relabeled with the highest reward can be selected. I think this relabeling-based skill selection is the most convincing apples-to-apples comparison because it doesn’t use the idea of intrinsic and extrinsic reward matching but does use the extrinsic reward function.
    - HRL method deserves to be a baseline for long-horizon skill sequencing
- Empirical support is limited. 
    - Most of the comparison is not showing any consistent tendency. It is rather showing that the proposed skill selection can work worse than the grid search method or sampling-based skill selection. (table 1 all rows except Fetch Push, Figure3 first row, Figure 7). 
    - Skill sequencing experiments only show results of IRM random. Considering the inconsistency over IRM optimization methods in Table 1, the result should be presented along with other IRM optimization methods, or a reasonable model selection criteria should be given, to conclude IRM is better.


clarity,_quality,_novelty_and_reproducibility: ### Clarity, Quality
The presentation of the paper was mostly clear. Few unclear parts :
- Section 3.3 is very difficult to follow without more formal description or example.
- Figure 3 starts plotting from the middle only to denote the number of samples needed for skill selection, but have done only for Env Rollout CEM. In my understanding, Env Rollout and Grid Search also need to be started from the middle.

A few unimportant flaws are found :
- Multiple emphasizing in Table 1 "Walker Stand" row
- Inconsistent decimal format in Table 2

### Reproducibility
Code is not provided, but implementation details and hyperparameters are described.


summary_of_the_review: The proposed method is very convincing and supported by analysis results.  
However, provided empirical results less support the effectiveness of the core idea.  
Current results are enough to show that IRM is a working idea,
but the benefit of IRM, over any naive approach leveraging extrinsic reward function, is not clearly shown.  
Thus, I vote to reject this submission for now as I think the important comparison is missing,
but still willing to raise my score according to the rebuttal discussion.

### Post rebuttal
I appreciate the authors' efforts in the response.
The rebuttal response resolved my main concern about the experimental results.
I'm convinced that IRM is the only method that can zero-shot select skills without accessing pre-training data.
Still, given relabeling baseline provides a good intuition of the benefit of accessing the extrinsic reward function and how well EPIC is leveraging it.
Thus, I raised my rating as I agree with sharing the idea of EPIC-based skill matching with the community and leaving the study of consistently working skill matching methods to future work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: There is many works that propose methods for unsupervised skill discovery, e.g. via behavior diversification. This work builds on top of these and proposes an approach for offline skill selection when using the unsupervisedly pre-trained agent for downstream tasks. Specifically, assuming access to a downstream task reward function, they find a skill that aligns the intrinsic reward used during unsupervised pre-training with the downstream task reward under a robust reward metric, EPIC. In experimental evaluations on the URLB benchmark they show that this skill selection is competitive with prior works that require online environment interactions.
strength_and_weaknesses: # Strengths

- the problem of skill selection is important for effective fine-tuning of pre-trained agents —> the paper presents a novel approach that can work fully offline and leads to competitive results with prior works that require online interactions —> this is a nice contribution

- the use of the EPIC metric to compare rewards and *optimize* the skill choice is a novel idea and potentially useful even beyond the discussed approach, e.g. similar problems of downstream task reward alignment occur in unsupervised meta-RL approaches and aligning rewards with this robust metric could be an interesting direction to explore there too

- the submission has comprehensive analysis experiments that visualize the EPIC loss for different trajectories and give a good intuition that it correlates well with downstream task success. I also appreciate the ablation studies on other, more naive reward comparison metrics that nicely justifies the added complexity of the EPIC loss


# Weaknesses

The submission’s main weaknesses are in terms of the writing and the experimental evaluation:

(A) **writing hard to follow, too many details in appendix**: the description of the method as well as the experimental evaluation is a bit hard to follow at times. One reason is that many details are pushed to the appendix, which makes It hard to understand the paper from following the main text. For example, the very beginning of the method section should first give a rough intuition for how unsupervised skill discovery methods with intrinsic rewards work and introduce how they train a discriminator to diversify skills. Instead, the paper assumes that readers are already familiar with these concepts. Similarly, in the experimental section, it should be more explicitly stated what the standard skill selection method from prior work is — I had to go through the URLB paper (Laskin’21) to figure out that grid search over skills with the first 4k environment steps is the standard approach — that should be mentioned prominently in the submission. Also baselines, like the SeqEnv are only explained in the appendix, which makes it very hard to understand the comparisons in the experimental evaluation.

(B) **experimental results not very strong**: the presented experimental results show that the proposed method merely matches prior works and on most of the URLB tasks a random skill selection baseline is performing similarly. While prior works do require some environment interactions for skill selection, they use only 4k interaction steps, which is a small fraction of the 100k steps that are used for fine-tuning. Thus, the effective fine-tuning efficiency gains in Fig 3 are marginal.

(C) **discussion of assumptions and limitations not explicit enough**: the paper does not clearly state the implicit assumptions of their method in comparison to prior works. In contrast to prior works, the proposed method requires access to the downstream reward function (as a trade-off vs not requiring online interactions), it assumes that we can sample a representative distribution of states (this would be very hard for image-based observations for example) and for the sequential task evaluations assumes access to per-subtask reward functions. Its effectiveness seems limited in sparse reward cases and, in contrast to prior work, cannot straightforwardly be applied to image-based environments. While it is okay to have tradeoffs like this with respect to prior work, they should be more explicitly discussed to make readers aware of them when choosing what method to use.



# Questions

- How does the proposed approach perform with sparse rewards? This needs some more discussion (see point on limitations above)

- How does the method perform with shorter offline pre-training? URLB shows performance with 100k, 500k, 1M pertaining steps too — would be good to understand the tradeoffs of the proposed approach vs prior skill selection approaches in this regime too.

- Do you see a way to extend the proposed approach to environments with image-based observations?

- What are the assumptions in the sequential task case? Do you assume access to a reward function for every sub-task?


clarity,_quality,_novelty_and_reproducibility: The writing is at times hard to follow (see weaknesses above), but the introduced approach is novel and the experimental evaluation has a comprehensive set of analysis experiments.
summary_of_the_review: The paper proposes a novel and interesting approach for a relevant problem. The introduced idea of using the EPIC loss to optimize for reward alignment potentially is useful even outside the proposed application, so I think the paper can be an interesting contribution for the community. I do think that the writing could be substantially improved to make it easier to follow and the experimental results don’t show a substantial improvement over prior work. Thus, in conclusion, I am supporting acceptance but it’s only a weak accept and I am open to change my mind based on the other reviewers’ comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In the field of unsupervised skill discovery, the authors focus on that the skill discriminator is often used for generating intrinsic reward signals and propose to leverage the skill discriminator to match the intrinsic rewards with extrinsic rewards to solve downstream tasks. They use the EPIC pseudometric to measure distances between intrinsic and extrinsic reward functions, and test a number of methods for minimizing the distances and choosing optimal skills. The empirical evaluation and analysis are done in URLB and the Fetch environment.
strength_and_weaknesses: Strengths

- The idea of using distances between intrinsic reward functions induced by the skill discriminator and extrinsic reward functions in downstream tasks to choose one of the pre-trained skills is novel to some degree.
- Experimentation with the proposed approach is thoroughly done, which provides the empirical evaluation and analysis of the method in multiple aspects.

Weaknesses

- If I'm not mistaken, one important weakness of this work is that it requires more information about the environment and downstream tasks, compared to most prior skill discovery approaches, which makes the underlying problem settings different. The computation of the EPIC loss requires the ranges of state dimensions (or the state distribution) and the extrinsic (i.e., downstream task) reward functions, which, especially the latter, are usually not assumed by prior approaches.
- In the writing of the paper, the authors should be clear about the distinction in the problem settings. They mention that this approach can "determine the optimal skill for an unseen task without environment samples" and is sample-efficient (in the abstract, Sec.3.2, Sec.4.2, etc.), but determining the extrinsic reward function without given knowledge requires or even may not be easily possible (e.g., in sparse-reward environments) with samples from environment interactions.
- The writing could be improved in some minor ways. For instance, I believe "invariant on an equivalence class of reward functions that always induce the same optimal policy" in Sec.2.2 is quoted from the EPIC paper (Gleave et al. (2020)) as-is. Both "fine-tune" and "finetuning" appear in the same Algorithm 1.
clarity,_quality,_novelty_and_reproducibility: - The presentation is basically clear in general, but please check out my comment above.
- The overall quality of the paper is fine.
- I think their approach to matching intrinsic and extrinsic reward functions using the skill discriminator and the pseudometric for reward functions for skill selection is somewhat novel.
summary_of_the_review: I like the idea of exploiting the skill discriminator and the reward pseudometric to select optimal skills for downstream tasks and the experiments and analyses presented in the paper. On the other hand, as I mentioned in the Strength And Weaknesses section, to my understanding, it requires the additional information. I believe it would be fairer to state such additional assumptions or difference in the problem settings, because the knowledge of extrinsic reward functions on new tasks is not necessarily cheaper than environment interactions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method that learns to choose among a set of skills to solve a downstream task. The key idea presented is that some skill learning methods often learn a discriminator which can be thought of as an intrinsic reward function for the underlying skill. Using an equivalent policy-invariant metric called EPIC, the authors suggest comparing the ‘fitness’ of skills using the corresponding intrinsic rewards to the external task-specific environment reward. This presents a zero-shot method to find a good skill for an unseen task. This method is tested on a number of benchmarks including one that involves composing skills sequentially and the intrinsic reward and EPIC loss are also analyzed.
strength_and_weaknesses: I think the idea and method being proposed in this paper are somewhat interesting and, to my knowledge, novel. However, apart from some methodological issues I have with the idea, my main concern is the quality of writing and communication in the paper. There were multiple instances when reading the paper which made me think that the work was quite rushed and for which I had to spend a lot of time looking through the Appendices in order to make sure I understood the details. While I am willing to ignore some of these, when they repeat so often it makes the paper much less enjoyable to read and, more pertinent to the context of reviewing, much harder to parse and understand. 

I will first list out some questions I have regarding the method and will then follow with a list of ways in which I think the communication could be improved, including areas where I think claims need to be backed up with more references to the literature. I’d also like to note that if the writing is improved, I would be more than happy to reconsider my score. 

Methodological questions:
1. In the Appendix, the choices of D_S and D_A are ablated in Table 6. There seems to be quite a stark difference when switching from the ‘Uniform’ distribution for the Pearson distribution to the ‘Full Random’. My understanding was the ‘Full Random’ samples uniformly but with the upper and lower bound for each state fixed. It was not clear to me why changing the bounds would affect the sampling distribution so much unless it was quite a strong assumption. Could the authors clarify what these two distributions mean perhaps with an example?

2. Unfortunately I’m not convinced the method works particularly well given the results of Table 1 and Figure 3. In Table 1, for the JACO task the Env CEM baseline seems to sometimes get a reasonably higher performance (9-10 on Jaco Top and Bottom Right). In the Walker setting a ‘Random’ skill works so well that I am genuinely a bit perplexed as to why other methods are so terrible - for instance in Walker Walk. In the finetuning results of Fetch Reach as well I would not say that IRM is that much more sample efficient. Perhaps understanding this a bit better - for instance why the method does not work so well on Walker could lead to algorithmic improvements that make it more robust. As things stand I don’t think the empirical evidence is compelling enough for this to be used in practice. 
The Jaco tasks also bring up an interesting point regarding sparse reward tasks. Sparse rewards are easier to define but as the authors note, would be much harder to use as a way to pick skills via matching. In general perhaps the method is sensitive to the nature of the reward landscape? In particular how does the analysis of Figure 5 of the EPIC landscape change for different tasks? Currently I’m not certain if the performance is limited because R_int is not being estimated accurately enough or if the loss landscape itself is not smooth on some tasks and understanding this would be quite interesting I think.

3. In Figure 3, why would we expect the finetuning performance to be different under IRM?  My understanding is that the method would try to pick the ‘most suitable skill’ in terms of how it matches with the task reward. This says nothing about the skills ability to finetune on a new task per se, does it? 

4. In Section 7, the EPIC loss is described as a contribution of this work. My understanding was that this loss was used, albeit in a different context, in prior work?

Areas in which presentation can be improved and clarified:
1. There are multiple places (Section 1, 2.1) with sentences in the vein of - “the intrinsic reward function learned during skill pre-training can..”. First, this notion that when using a particular formulation to learn skills, we can interpret the skill as optimizing an intrinsic reward function is somewhat new. This idea needs to be introduced and at least some context given to the reader to understand what it means. Also I don’t think the claims can be made generically across *all* skill learning methods. At least a gentle introduction of what skill learning methods this applies to would make this part much more readable. Often these sentences do not have references and citations to specific works to back them up as well which is an issue in scientific writing.

2. The equations relating to the EPIC losses - Equations 2 and 3 are hard to parse and not strictly correct I think. Equation 2 includes S, A, S’ on the right-hand-side which should be input to the function. The definition in Equation 3 is also inconsistent with that of (2) and trajectories are defined as (s, s’) which means the action is not used. I understand that the reward functions being considered here do not require actions but this should be specified explicitly in the main text and in either case the mathematical definitions should be consistent.
 
3. The second point is particularly important in the context of Algorithm 1 since these equations are the only way to understand it. In general, as far as possible, algorithm boxes should be self-contained and express the main jist of the approach. Currently in the box variables N_FT, S_p, A_p, S’_p, S_c, A_c, S’_c are defined but never used and variables s and s’ are used but not properly introduced. I think if the algorithm box just explicitly wrote out the EPIC loss from Equation 2 it would make it much more self contained and easier to parse. 

List of typos and other less minor issues in writing:
1. Paragraph 3, Introduction: paramterized instead of parameterized.

2. Section 2.1 after Equation (1): ‘is’ is missing after ‘where VLB<= I(\Tau, z)’.

3. Equation (3) and (4) - it equation does not include over what random variables the expectation is being taken.

4. Section 3.1 paragraph 1: ‘specified our intrinsic’ should be ‘specified by our intrinsic’ and ‘As is such’ should be ‘As such’. 

5. Section 3.1, paragraph 2: ‘which our pretrained’ should be ‘which of our pretrained’.

6. Section 3.2, paragraph 3: ‘leads improved’ should be ‘leads to improved’. 

7. Table 6 v/s 6, Appendix 1 v/s 1 etc. are used interchangeably throughout. It helps when the relevant sections and appendices are explicitly and consistently labeled  throughout the text. As a specific example, the ablation for P_s and P_A at the end of section 3.2 refer to ‘6’ which is unclear.

8. Section 4.1, paragraph 1: ‘is either comparable to our outperforms..’ - ‘our’ should be ‘or’.

9. Section 4.3: ‘We compare an extended versions’ - no ‘on’. Sentence ends with ‘...environment rollout method identifying’. Figure 4.3 Row 2 should be Figure 3?


EDIT:
After looking through the changes made during the rebuttal period I think the paper reads much much more clearly and is stronger now. There are still some nagging issues in the presentation (predominantly of Section 3) but I appreciate that the authors also ran experiments for other baselines and it is challenging to satisfy all constraints within a short time limit. I have increased my scores to reflect the updated paper and would like to thank the authors for the great work in addressing concerns from the rebuttal period.

clarity,_quality,_novelty_and_reproducibility: As described above, while I like the idea and think it is an interesting view of some skill based learning methods I think the clarity and quality of the paper would need to be improved for it to meet the bar for this venue. 
summary_of_the_review: As things stand, I think the paper has too many flaws for it to be published at ICLR. I am happy to reconsider though based on the rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces the EPIC-loss as a way of choosing which pre-trained skill is optimal, given a downstream reward function. Reviewers all agreed that the proposed idea was interesting, of interest to the community and that the clarity of the paper had greatly improved throughout the discussion phase (with scores improving as a result). Authors should also be commended for the number of ablations provided.

However, most reviewers and AC agreed that despite this, empirical results were poor and did not convincingly support the claims made in the paper, both in the zero-shot domain (Table 1) and fine-tuning regime. Fig 7 in particular is problematic as it shows that zero-shot gains are neither consistent (e.g. on par with random skills on Walker {Run,Stand}) nor significant compared to the benefits of fine-tuning. Given the claimed contribution of the EPIC loss (in contrast to the EPIC reward comparison metric), I also personally expected bigger improvements compared to IRM-Rand, especially for larger dimensional skill spaces where sampling based would underperform vs optimization based approaches (Fig 6).

While we acknowledge that these results could come down to sub-optimal skill pre-training algorithms, it does not follow that results would necessarily improve with better algorithms, nor can we base our decision on this hypothetical scenario. Ultimately, the authors chose the problem setting and it is their responsibility to show that the method works well in this setting. Had the EPIC framework been first proposed in this paper, poor empirical results could perhaps come second to the idea, however as an application of EPIC to the problem of skill selection empirical results must come first. And for that reason, I am recommending rejection at this stage but am looking forward to seeing future revisions of this work.

In a personal reviewing capacity (which I stress did not impact the above decision), I also believe that this paper would benefit from the broader context of rapid adaptation in RL, as “skill selection” is only one mechanism for adapting quickly to downstream rewards. Similar experiments could have been conducted in the successor features framework which makes me think this, or the related VISR method (if we do not want to assume known features) could have been valid baselines. Could the authors not also evaluate the EPIC loss in multi-goal conditioned RL (multi-task pre-training, with known reward functions per task), as a way to hedge against underperforming skill discovery algorithms?

justification_for_why_not_higher_score: The results in the paper does not convincingly show that the method is effective, combined with limited novelty.
Decision: Reject