Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an approach for learning policies that can enable better generalization across tasks. This is done using an ensemble of learners, each of which uses attention to focus on a different part of the feature space, and then using bandit-exploration to select a particular policy. Further, the authors include an ensemble of classifiers to account for initiation and terminations sets, and argue that the combination of the classifiers and the policies give rise to transferable options. Generalization results are included on ProcGen, and Montezuma's revenge (across multiple levels). 
strength_and_weaknesses: Strengths 
1. Generalization through learning diverse representations
Using an ensemble of policies each of which focus on different parts of the input, and then selecting among them using a bandit-style objective to find the most robust policy is an intriguing idea. There has been work that focuses on learning the right representations of the environment to enable better generalization [1], and this paper approaches the same problem by instead encouraging diverse representations for multiple learners (enforced using distance in the embedded space), and then selecting for the most robust one. 

Weaknesses - 

1. Complexity of approach, specificity of Montezuma experiments, relation to exploration algorithms

The overall approach has a large number of moving parts - the policy ensemble, the classifier ensemble, the markov classifier. For the experiments included in particular in the paper (Montezuma's revenge), the initiation and termination conditions need to be specified by hand for the classifiers. Even the options are hand-specified, and not learned for the task (the authors seem to indicate that there is further training for these options, but this isn't clear). Also evaluation is done on specific tasks within the game (rolling skull, spider etc). Contrast this to results in self-supervised exploration. Go-Explore [2] was able to enable an agent to learn how to play Montezuma's revenge without any of these constraints (hand specified option/initiation/termination conditions) and without any rewards, and keep generalizing across different levels also without any restriction to the particular task considered. 


2. Sensitivity of ensemble size for policy ensemble - 
 Even for the policy ensemble results on the ProcGen environment, the policy ensemble size needs to be tuned for each environment, and performance does not generally improve with more policies in the ensemble. The authors argue this is because 'too much time is spent exploring policies that do not generalize'. However, there is no analysis that shows this to be the case. It might be difficult for practitioners to adopt this approach if there is heavy reliance on tuning this hyper-parameter. The results on the ProcGen environments show that no single choice for ensemble size performs consistently (they each are close to the 1-member ensemble case in atleast one environment). 


[1]: Eysenbach, Ben, Russ R. Salakhutdinov, and Sergey Levine. "Robust predictable control." 
[2] Ecoffet, Adrien, et al. "Go-explore: a new approach for hard-exploration problems."
clarity,_quality,_novelty_and_reproducibility: The paper is quite easy to follow up until section3. However, I found section4 which brings together the entire approach with the ensemble of classifiers and markov classifier, as well as the various specific conditions used in the montezuma experiments to be more convoluted. 
Due to the complex nature (multiple moving parts) of the overall system, it might be difficult to reproduce.

summary_of_the_review: Overall, I am not in favor of acceptance since the full presented system is evaluated for particular choices of options/initial/final conditions and specific tasks, while we have have algorithms from the self-supervised learning literature like Go-explore which can keep exploring and generalizing (for the same environment considered in this paper). The idea of learning diverse features to find one that's robust to new tasks seems interesting but performance is very dependent on ensemble size as described previously, and this will hinder widespread adoption.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper tackles the problem of learning reusable option policies to achieve cross-task generalization using reinforcement learning. It describes a method for learning feature spaces that can be reused by an option-like policy for (1) determine the next action to take (2) whether it is executable and (3) whether it should terminate. The core technical component is an ensemble feature learner (Kim et al., 2018) that aims to discover a set of feature extractors that are both diverse and optimizes some objective. In this paper, the ensemble feature learner is used to discover features for an ensemble of RL-based option policies for optimizing task rewards. The same ensemble feature learner is also used to learn an ensemble of initiation set and termination set classifier. The core component of learning reusable policies using feature ensemble learner is evaluated on a set of Procgen environments. The full method (including the initiation and termination set classifier) is evaluated on the montezuma’s revenge domain, where different rooms are considered different environments. 

strength_and_weaknesses: Overall I like the premise of the paper. Options should be able to port to other environments or different parts of the state space, as opposed to only function in the state distribution that it’s trained on. I also applaud that the paper chooses to evaluate its method on a pixel-input domain as opposed to more “convenient” environments where each input dimension is already meaningful.

However, I have to admit that I have a hard time appreciating the method itself and the conclusion drawn by the evaluation. I also had a hard time understanding the method itself, despite multiple attempts at reading the section. Below I will describe my comments in detail.

Writing quality
The structure of the paper is clear, and the introduction is enjoyable to read. However, the method section is difficult to understand, especially section 4.2 and 4.3 and their connections wit the previous subsections. Here is a list of points that I do not fully understand
I understand why it’s difficult to evaluate the initiation condition, but if the method assumes a reward function is given, why can’t it use the reward function to determine the termination set?
The third paragraph of 4.2 says “If the initiation and termination set are both incorrect during the same option execution, the option will appear to succeed but will have an incorrect instantiation”. I understand false positives are likely, but the sentence appears to state that the option is guaranteed to succeed if both classifiers are wrong. Why is that?
I’m generally confused about the role of the markov classifier. The paragraph says the markov classifier “learns the initiation or termination set of a new option instantions”, What does that mean?
How does markov classifier allow the option to “return to a specific instantiation”? Does it reset the simulator status?
In the 5th paragraph, the “classification loss” is used to determine policy success. The classification loss of which classifier? And why is loss used, not the prediction? And why is a true positive state “almost indistinguishable from the previously seen state the classifier deems positive” since the invariant feature should only be a subset of the entire space?
How is the portability estimate in 4.3 used by the classifier ensembles concretely?
In section 4.4, what exactly is the “true determination state” in eq 5? What is the “true” meant to contrast? 

My second major comment is regarding the conclusion drawn in the experiments in general, especially in Section 3.2. The paper seems to imply that the improved performance is solely due to the reusable feature set extracted via the ensemble feature learning scheme. However, another factor that may play a critical role is the ensemble policies themselves. It is widely known that ensemble could aid exploration. Thus to correctly attribute the performance gain, It would be critical to ablate this factor and evaluate the ensemble RL policy without the feature learner. Similar counter arguments can be made for all later experiments, as they all rely on the same ensemble RL strategy. 

Finally, the paper has many moving parts and little ablation studies. The ambiguous evaluation metrics that entirely relies on the learned classifier makes the experiment results less clear. I believe the paper will benefit tremendously from a set of experiments using domains that support option-level skill evaluation with ground truth initiation and termination set. For now, it is really difficult to understand the validity of the conclusion drawn from the experiments.  

clarity,_quality,_novelty_and_reproducibility: Clarity: the motivation is clear and enjoyable to read. The method section and the experiment section are really difficult to understand.
Quality: the motivation and the premise are clear. But I had a hard time evaluating the validity of the method and the evaluation. 
Originality: learning reusable feature set for options framework is important and novel. 
summary_of_the_review: Overall I really like the premise and the motivation of the paper. However, the writing / clarity is lacking and the experiment evaluation is insufficient.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an approach for learning portable options by training and ensemble of options and enforcing that the ensemble members learn feature representations that attend to different parts of the observation. The approach then learns to determine the feature set that generalizes best across environments to choose the most portable option. The paper demonstrates learning of portable policies in ProcGen and portable options in Montezuma’s revenge.
strength_and_weaknesses: # Strengths

- the problem of learning portable options is very important: the whole point of learning options is to “port” them to different tasks, but often options trained on observations from one task overfit to spurious features and then don’t generalize well to another — approaches like the one proposed here, that aim to address this problem, are of high relevance to the community

- the intuition of the proposed (learning ensembles with disjoint feature sets) is novel to my knowledge and intuitive

- I like the structure of the paper, in which the authors first explain their idea in the context of policy learning and only afterwards extend it to the more complex case of option learning (which additionally requires to learn initiation and termination sets) —> disentangling the complexity in this way makes it much easier to follow the description of the method


# Weaknesses

The main weakness of the submission is the experimental evaluation of the proposed approach. The paper proposes an approach for option learning, but never evaluates whether the learned options are actually useful for downstream learning. There are no comparisons to prior works and no ablation studies of the components of the proposed approach. There is no qualitative analysis of the learned feature representations, even though they are the main novelty proposed in the approach.

The experimental analysis will need substantial improvements to verify the usefulness of the approach.

(1) Evaluations on using the options for downstream task learning should be added. 

(2) A comparison to (a) prior option learning works and (b) the exact same setup as the proposed approach but without the ensemble learning part should be included.

(3) Detailed analysis experiments of the many moving parts of the proposed approach should be added: how do different number of ensemble elements or different regularizing coefficients influence the learned options?

(4) Qualitative results and visualizations should be added, to show what features the ensemble identifies as transferrable, in what situations transfer works well (compared to baselines), in what situations the proposed approach fails etc

The current experimental evaluation does not allow to properly judge the proposed approach and, due to a lack of qualitative results, it is hard to understand why some of the trained options fail or why their performance *degrades* over training.

-----
Apart from the experimental evaluation, there are a few additional weaknesses:

(A) The description of the extension of the ensemble approach to learning options (instead of policies) was hard for me to follow, particularly I had trouble understanding the second-to-last paragraph in section 4.2 — some intuitive examples for options and feature sets, e.g. from Montezuma’s revenge, and a figure visualizing the approach could help here.

(B) The proposed objective encourages the learned representations between ensemble elements to be disjoint — it seems to me that this can make optimization tricky since there are many local optima in which important features are “trapped” within disjoint feature sets of different ensemble members, making me wonder how hard it is to tune this approach on tasks that require the interplay of many features for policy learning.

(C) More generally, like many option papers, the proposed approach has many moving parts (learning ensembles of policies, initiation and termination classifiers, additional regularization terms with coefficients that require tuning, bandit-optimization of which policy from the ensemble to pick for execution). This suggests that it will be hard / require a lot of tuning to apply this approach to complex problems. Again, this is generally true for many option learning approaches, but the paper introduces additional complexities that go beyond conventional option learning and exacerbate the problem.

(D) The related work section misses some references to relevant work on learning state abstractions for generalization in RL. See e.g. Amy Zhang’s McGill PhD thesis from 2021 for an overview of relevant works.


# Questions


- Can we learn a single embedding space shared between initiation and termination classifiers and policy? This could substantially reduce the number of learned components and required tuning.

- Fig 3 shows that the number of ensemble elements needs to be tuned separately for each task. Is there a good way to choose the number of ensemble elements without just trying all of them?

clarity,_quality,_novelty_and_reproducibility: The writing is clear in introduction and the first part of the approach section. The writing in the later part of the approach section (that explains extension of the idea to the option setting) as well as in the experimental evaluation however is hard to follow and lacks visualizations. 

The quality of the experimental evaluation is severely lacking. 

The approach is novel to the best of my knowledge and an interesting idea.
summary_of_the_review: I am a bit torn about this submission. I really like the tackled problem and the approach seems interesting to me. There are some doubts about the empirical performance of the approach which I have outlined in “weaknesses” above. These would require a solid experimental evaluation to address. However, the experimental evaluation of the current submission is severely lacking. Thus I do not recommend acceptance of the paper in it’s current form and suggest the authors extend the experimental evaluation and resubmit their paper at ICLR or another fitting venue.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper focuses on the problem of learning generalizable skills in multi-task reinforcement learning settings. The authors propose an ensemble approach, which is empirically validated on the procgen benchmark and the Atari game Montezuma's revenge.

Reviewers appreciated the importance of the problem tackled here - a key argument behind options is generalization, yet this is a hard and underexplored area. The paper proposes a promising idea, and reviewers find the approach intuitively appealing. They also commend the fact that evaluation was performed on challenging domains with visual observations.

At the same time, initial reviews noted several concerns, some of which have been addressed by the authors during the rebuttal phase. The algorithm was found to be quite complex, and the clarity of describing the approach needed improvement. Reviewers raised questions about the hand-specified components (initiation, termination as specified on screen 1 of MR), and need for hyperparameter tuning (ensemble size). Additional concerns were raised regarding a lack of ablations to confirm the role of individual components of the approach, comparison to prior work, and a demonstration of the usefulness of the learned generalizable options to downstream tasks.

The authors responded to several of these concerns, clarifying the approach in section 4, and providing additional experiments that covered the requested ablations. The paper is assessed to be in much stronger shape than before. At the same time, concerns remain. First, the additional experiments are a substantial and necessary addition to the paper, and should be re-evaluated in a thorough set of reviews. Second, empirical validations such as the usefulness for downstream tasks have not been added. In sum, the paper has improved during the rebuttal phase, but is not yet ready for publication at the current stage.
justification_for_why_not_higher_score: The paper was missing important empirical validation in the initial submission. Based on the initial reviews, it has become stronger, but would need to be re-reviewed in detail. In addition, the authors have not shown the usefulness of the learned options for downstream tasks.
Decision: Reject