Learning Portable Skills by Identifying Generalizing Features with an Attention-Based Ensemble
Authors: Anita Taosheng De Mello Koch
Zhiyuan Zhou
Akhil Bagaria
Haotian Fu
Cameron Allen
George Konidaris
Venue: Submitted to ICLR 2023 (2023)
Keywords: Hierarchical reinforcement learning
Skill transfer
Ensembling

Abstract: The ability to rapidly generalize is crucial for reinforcement learning to be practical in real-world tasks. However, generalization is complicated by the fact that, in many settings, some state features reliably support generalization while others do not. We consider the problem of learning generalizable policies and skills (in the form of options) by identifying feature sets that generalize across instances. We propose an attention-ensemble approach, where a collection of minimally overlapping feature masks is learned, each of which individually maximizes performance on the source instance. Subsequent tasks are instantiated using the ensemble, and transfer performance is used to update the estimated probability that each feature set will generalize in the future. We show that our approach leads to fast policy generalization for eight tasks in the Procgen benchmark. We then show its use in learning portable options in Montezuma's Revenge, where it is able to generalize skills learned in the first screen to the remainder of the game. 

Under review as a conference paper at ICLR 2023
LEARNING PORTABLE SKILLS BY IDENTIFYING GEN- ERALIZING FEATURES WITH AN ATTENTION-BASED ENSEMBLE
Anonymous authors Paper under double-blind review
ABSTRACT
The ability to rapidly generalize is crucial for reinforcement learning to be practi- cal in real-world tasks. However, generalization is complicated by the fact that, in many settings, some state features reliably support generalization while others do not. We consider the problem of learning generalizable policies and skills (in the form of options) by identifying feature sets that generalize across instances. We propose attention-ensemble approach, where collection of minimally overlap- ping feature masks is learned, each of which individually maximizes performance the source instance. Subsequent tasks are instantiated using the ensemble, and transfer performance is used to update the estimated probability that each feature set will generalize in the future. We show that our approach leads to fast policy generalization for eight tasks in the Procgen benchmark. We then show its use in learning portable options in Montezuma' Revenge, where is able generalize skills learned in the first screen to the remainder of the game.
INTRODUCTION
In recent years reinforcement learning has outperformed humans in many Atari games (Mnih al. 2015). learned to play world champion level Go (Silver et al.) 2017) and mastered many robot manipulation tasks (Levine et al., | 2216 (2018). While these achievements are undeniably impressive, they are in simulated and controlled environments stripped of many of the complexities humans face in everyday life. For reinforcement learning to be viable real-world applications, the ability to scale to large, high- nigh-dimensional environments is crucial. Hierarchical reinforcement learning (Barto) Mahadevan 2003) is a promising approach to achieve this scalability through the use high-level skills that abstract away the detail of low-level action. The most popular hierarchical RL framework the options framework (Sutton et al. 1999)), which models abstract actions as consisting of three components: set of states from which execution begin, a policy which specifies how the option executes, and a set of states where execution ceases.
To fully realize the promise of the options framework, learned options should ideally be easily reused, or ported, to new tasks and environments (Konidaris & Barto 2007). The core difficulty here is that, in practice, an option will be first learned in a small number of specific instantiations possibly just one withwithout foreknowledge of the circumstances under which it will be applied again in the future. In such cases there may be many state features over which the first instance(s) of the option could be successfully defined, but which will not support reuse. For example, a single option to open door might be equally well-deffned using features describing the door's location a global map, or features describing the location of its handle relative to the agent, but only the latter will generalize to new doors. This problem exacerbated by the fact that all three components of the option must simultaneously function in new instantiations, or the option will fail.
We therefore propose to learn portable options by identifying sets state features that support generalization. We adopt the transfer learning setting (Taylor & Stone, 2009). where the goal is to learn on number of source tasks and perform well on target tasks with minimum re- re-training. We introduce method where agent leamn a collection of diverse feature sets that each individually maximize performance on the source task. Subsequent option instantiations evaluated for success, and the results used update
an attention-basee ensemble (Kim al., 2018) to

---

Under review as a conference paper at ICLR 2023
the agent's s estimate of the probability that each feature set will generalize future tasks. These probabilities in turn govern which feature sets are used in new option instantations.
We begin by showing how to learn a portable policy using an ensemble, and demonstrate that it leads to fast learning on cight games from the Proegen generalization benchmark (Cobbe et al. 2020). The ensemble is then extended learn portable classifiers that represent initiation and termination sets. We combine the resulting portable policy, initiation classifier, and termination classifier methods to learn portable options in Montezuma's Revenge, where our method enables an agent to generalize skills learned in the first room to all the others.
2 BACKGROUND AND RELATED WORK
We consider the episodic reinforcement learning setting. An agent operates in a Markov Decision Process (MDP) with state space S and action space A. The transition probability p(*++1|*8 1 az) is the probability of transitioning from state St state 8++1 with action dz- At each time step, the agent receives a scalar reward defined by the reward function r(st, at). The goal of the reinforcement learning agent to maximize the cumulative discounted reward over an episode by finding a policy π (als) that selects actions at each step.
The options framework by Sutton et al. 1999) extends the MDP framework by creating temporally extended abstract actions known as options. An option o is defined by a three-tuple (Lo, To. 8o). The initiation set, In : S + {0, 13. is the set of states in which option o can initiate. The termination set, B. S - (0, 1) is the set of states in which option successfully terminates. The option policy To : > A is controller that transitions the agent from states in To to states in f%.
Options in their most basic formulation offer no guarantees of being portable. However, Konidaris & Barto 2007) argue that if the inputs to an option retain the same semantics (introducing the notion of an agent-space) across option instances, these options will be reusable. There are many works focused on building derived input spaces with transferable semantics. Konidaris & Barto| (2007) showed that an agent-centric representation, analogous to the egocentric space (the space surround- ing the agent (Klatzky 1998), would be sufficient for many tasks, especially in the robotics domain. The successor features (SF) and generalized policy improvement (GPI) framework proposed by Bar- reto et al. (2018) creates a space derived from successor features leading to reusable skills. However, these skills only adapt changes in the reward function, not the transition dynamics. [Gupta et al. (2177) learn skills in an invariant feature space which enables generalization across morphollogi- cally different robots. However, the invariant feature space cannot generalize across tasks. |Hausman eet (2018) propose learning separate embedding space which can be used to parameterize discov- skills. They show this successfully generalizes these parameterized skills across tasks but make the assumption that the agent has access to a collection of different tasks which are used to learn the embedding space. This embedding space results in portable policies, but does not address the main issue of over-fitting to the few instances option is defined over. Topin et al. (2015) transfers options between different object-oriented MDPs, but are confined to discrete-domains. Our work instead focuses building portable options in high-dimensional and continuous domains. We also differ from these approaches by using diversity to build portable options. We propose to learn a diverse set of state features-each of which maximizes rewards-to both define the option policy and identify the portable initiation and termination sets.
One way of maintaining set of features is by building an ensemble. Ensemble methods train mul- tiple learners on the same task, resulting in a combined model which performs better than each individual model. The quality of the ensemble depends on the quality of each individual and the diversity among them. The attention-based ensemble proposed by Kim et al. (2018) is deep learn- ing framework, originally intended for deep metric learning, that encourages diversity in the feature embeddings learned by each member in the ensemble. The objective for the ensemble is to minimize a combination of the training objective loss and a divergence loss:
L({x;)) = Σ Lusinn((m) (1z+)) + Laiv(()))))
(1)
m
2

---

Under review as a conference paper at ICLR 2023
Here, {ag} is a set of training samples and Lurin is the training loss of the learning objective for the m-th learner. The divergence loss, Lavv, is weighted by Advv and defined as
Laiv (av((xx))) ΣΣ max(0, mai dy (B B,()))))
(2)
P.
where dy is distance metric in the embedded space, midie is margin, and Bp (ar)); (24) is the embedding output of learners p and q. Optimizing the total loss L({x;}) incentivizes learners to in- dividually perform well on the training task while learning maximally different feature embeddings.
PORTABLE POLICIES WITH AN ATTENTION-BASE ENSEMBLE
We propose to achieve option portability through ensembles. By learning multiple diverse feature sets, we increase our chances of finding a portable one. We then condition the option's 's policy, initiation classifier, and termination classifier on the portable features, such that the resulting option generalizes to subsequent tasks.
In this section, we focus on learning a portable policy. The agent learns to solve task in the flat reinforcement learning setting, where only the policy needs to learned, without the need for initiation and termination sets. We later extend this framework to options in Section 4
3.1 ENSEMBLE OF POLICIES
The main challenge of learning an ensemble of policies is learning an ensemble of feature sets and identifying portable one. We use attention-based ensemble architecture to learn N distinct feature embeddings, which in turn are input to N individual reinforcement learners. Figure I shows an architecture diagram for ensemble with N worker policies, each attending a different set of features.
Each feature-learner consists of three components: a spatial feature extractor, attention module, and a global feature extractor. The spatial and global feature extractor are shared among all the learners, but each learner has an individual attention module. The attention module encourages each learner to attend to a different aspect of the input, thereby promoting diversity. The input passed through the shared spatial feature extractor first to obtain the spatial feature embedding 6. This embedding is then passed to each attention module to obtain the attention mask, which multiplied with the spatial feature embedding to form an attention embedding ÿ)- Finally, each attention embedding is passed through the global feature extractor to get the final feature embedding u. Each individual worker policy is conditioned on the features derived from the corresponding member of the ensemble.
The ensemble minimizes the loss defined in Equation I with Atvain defined by the objective loss of the reinforcement learning algorithm. Minimizing this loss encourages the agent to attempt to learn a good policy using each of the fcature sets, while keeping feature sets different from one Proximal Policy Optimization (PPO) (Schulman et al. (017) the
another. In this section, reinforcement learner, but our method is compatible with any reinforcement learning algorithm.
Attention Mask (Com/)
7
Hi
Policy
action
Sparial Feature Extrader (Cone Poot
+
1/2
Hg
(Com/)
Extrador
(Conv
Inout Image observation
Flutte))
Adontion Mask
(Comg)
TJN
J.N.
Polly N
acton N
Figure 1: Architecture for learning an ensemble of policies using Intention-Based Ensemble.
Each episode, a single ensemble member, known as the leader, is selected to govern action selection and all members trained on the experiences collected. The choice of leader has a big impact on
3

---

Under review as a conference paper at ICLR 2023
the ensemble performance when transferring to new tasks. If the leader is portable policy, then the ensemble will still able to perform well during transfer. Otherwise, the ensemble will likely fail. Since repeatedly select new leader every episode, it is natural frame this an N-armed bandit problem (Sutton & Barto, 2018). We can view each choice of leader as a bandit-action and the cumulative reward earned by the leader that episode the handit-reward. We choose the leader using an Upper Confidence Bound (LAI] 1985) method to balance exploration and exploitation. At timestep 6, we select leader L, follows:
Int (1)
L arg max
P +
(i (i) is the cumulative reward of ensemble member i at time t, (i) denotes the number of times that has been elected leader prior to time and constant controls the degree of exploration. The first part of Eq. 3 can be understood as exploiting good arms, and the second part exploring others.
3.2 PROCGEN RESULTS
We showcase the effect of the attention- ensemble on Proegen (Cobbe et al. 2020). a popular benchmark for evaluating generalization in reinforcement learning. Procgen is suite of procedu- rally generated game environments, each offering many levels. Each level is a unique instance of the game, differing in appearance- for example background colour object shape- but represents the same underlying game mechanism. We evaluated our approach on 8 games: Bigfish, Coinrun, Dodgeball, Heist, Jumper, Leaper, Maze and Ninja Following Cobbe al. (2020), choose PPO (Schulman et al. 2017) as the reinforcement learning algorithm. The training loss Ltrain is the PPO loss detailed in Appendix B.2
To provide intuition on the effect of the attention modules, we visualize the attention masks at single time-step of 3-policy ensemble trained on the game Coinrun in Figure Areas of high attention (yellow) are accentuated by the attention module, while areas with low attention (dark purple) have little effect on the final feature embedding space. Each ensemble member attends to different aspects of the image, each of which represents different set of features.
Figure 2: Visualization of the attention masks of a 3-policy ensemble trained on Coinrun, with intensity represented by color. Each attention mask focuses different of features.
We adapt the training scheme of Procgen to better align with the transfer learning paradigm. For each game, we randomly select 20 training levels. The ensemble trains for 500k steps on each level sequentially, maintaining knowledge obtained in previous levels. To evaluate, we compare the average within-level performance across the 20 levels. In Figure we plot the discounted episodic reward achieved on each game, comparing 4 ensemble agents with different ensemble sizes 1, 2, 3, and 5. The baseline agent with only one policy in the ensemble is equivalent to a PPO agent, because the divergence loss in Equation 2 zero and the attention-based ensemble architecture is equivalent to the feature extraction layers in the PPO. For six out of the eight games, multi-member ensembles outperform the 1-member ensemble, showing better skill transfer across levels. This demonstrates that portable policies can be learned with an ensemble.
To confirm that the improved performance results from learning portable features instead of having an ensemble of policies, which can bring benefits such as better exploration (Osband et al. 2016). we repeat the experiment with the same ensemble of policy approach but with attention-ensemble
In consideration of computing resources, the difficulty level easy
4

---

Under review as a conference paper at ICLR 2023
Training Curve Averaged Across Levels
Bigfish
Dodgeeball
Epleeds Rewwed
1.15
351
20
t
0.09
jumper
Leaper
Maz8
2.4
Eptodd: laveed
40
Ensemt ble
Ersemble
Ensemble
Ensemble 5
Figure 3: Learning curves of different ensembles of policies on 8 Procgen environments. The different agents have an ensemble of I, 2, 3, and 5 policies, respectively. On six of the environments, multi-member ensemble agents are able outperform the baseline agent. (10 random seeds; error bars denote standard deviation.)
Festure Leamnr: Training Curve Averaged Across Levels
Bglahh
Dodgebal
Eohoddo Recard
0.5
Leaper
Maze
Cptock: (agoo)
2.
23
Ersemble 1
Ersembiee
Ensemble
Ensemble 5
Figure 4: Ablation study of ensemble of policies without attention-ensemble feature learner on Procgen environments. The 4 different agents have an ensemble of 1, 2, 3, and 5 policies, respec- tively, but all policies share the same learned feature. (6 random seeds; error bars denote standard deviation.)
feature learner. All policies in the ensemble will share single learned feature. The results in Figure show that performance decreases without learning portable features for five out of eight games.
We note that the optimal ensemble size varies for each game. This an expected result because of the differing complexities of the environments, which result in different numbers of possible useful feature sets. When the ensemble size is too small, the learned feature representation may not be fine- grained enough to generalize. On the other hand, excessive number of ensemble members can also degrade performance by spending too much time exploring policies that do not generalize, and not enough time exploiting ones that do. We also draw attention to the importance of the exploration constant c in the UCB algorithm (Equation If c is too small, there is not enough exploration when learning to choose the leader, essentially collapsing the ensemble agent 1-member ensemble.
5

---

Under review as a conference paper at ICLR 2023
4 PORTABLE OPTIONS WITH AN ATTENTION-BASED ENSEMBLE
The results from Section 3.2 show that agent policies can be generalized with an attention-based ensemble. However, portable option also requires portable initiation and termination set. We now extend the attention-based ensemble to an ensemble classifiers that can be combined with the ensemble of policies to learn portable option. We first go over the construction of the ensemble of classifiers Section[4.] [ and then explain how to combine it with an ensemble of policies to build the portable options in Section 4.2
4.1 ENSEMBLE OF CLASSIFIERS
Similar to the intuition in Section 3 we learn portable classifiers with an ensemble. We use the attention-based ensemble to learn ensemble of N classifiers, each trained a unique set of features. This increases the number of feature sets considered when learning the initiation and termination sets, some of which will generalize.
Each classifier in the ensemble learns the set of states that make up the initiation and termination set. The attention-base ensemble learns N diverse embedding spaces which are input to the classifiers. The ensemble is trained using the objective function described by Equation 1 where the training loss Ltrain is the classification loss when predicting whether a state in the initiation or termination set (here, binary cross entropy), with labels from the instantiating option's 's classifier. In this paper we use a two layer perceptron as the classifier, and train an ensemble of N = 8 classifiers.
Because not all classifiers in the ensemble are portable, we weight each classifier's prediction by the agent's ' belief of its portability. In Section 3 we modeled this problem as an N-armed bandit problem because only one policy can control action-selection during execution and so successful learning requires exploration. However, when querying the classifiers, we receive predictions from all ensemble members, and there is need to explore. As such bandit model with a simpler Bayesian update rule
replace the N-armed
P.).D.) B(k+ann-k+ - (1 -.)
where given dataset D,, h out of n classifications are positive. 0, E [0, 1] is the prediction from classifier and a and B are selected to represent initial belief over portability. A full derivation of this update rule can be found in Appendix A
4.2 PORTABLE OPTIONS
We build a portable option using an ensemble of classifiers for the initiation and termination sets and an ensemble of policies for the option policy. The training scheme of portable options is shown in Figure 5 The portable option is first trained on the initial task instance (Figure green box). The agent then explores the environment for possible
instantiations of the portable option (Figure 5 right red box). Finally, the new option instantiation is used to update the portable option after being validated (Figure 5 left blue box). Instance discovery and validation of different instantiations can occur concurrently as the agent continues to interact with the environment.
During discovery the agent uses the portable initiation set identify new states in which the option can be used, or instantiated. Once new possible instantiation is identified, the portable option policy executed. If the option execution is successful (as shown in Figure 5), new instantiation of the option is created. Note that there is no supervision signal for successful executions of an option when it is ported to new tasks because there no guarantee that the reward function will align with the option termination. As such, it is possible to incorrectly identify new option instantiations if either both the initiation and termination sets fire on false positive states or, due to stochasticity in the environment, the successful option execution cannot be reliably replicated. If the portable option is updated using these incorrectly identified instantiations the option will degrade over time.
Therefore, use the validation stage to refine discovered instantiations before updating the portable option. During this stage the agent learns Markov classifiers that characterize the initi- ation and termination sets of this single option instantiation, and is not portable. These Markov classifiers remove the need for the agent to rely on the possibly unstable portable initiation and ter- mination sets by learning the true initiation and termination sets of one task instance. Unreliable

---

Under review as a conference paper at ICLR 2023
Initial training
Discovery of new option instantiations
Portable initation set triggers
Validation of discovered
Improve portable option
Portable Option
option instantiation
Well-trainet Markop classifiers
Excrute portable polisy
Refinement
Portable temmination set triggers
Oplion execution considered sucecssfll
Create Markow classifiuns this option instantiation
Figure 5: After portable option is learned the agent continues to explore the environment, finding new possible instantiations of the option. If valid instantiation is found, this data is used further improve the portable option. This figures shows this process for single successful option instanti- ation.
option instantiations will "disappear" as the Markov classifiers learn that the initiation and termina- tion null sets and the instantiation is not used to update the portable option. This ensures that new instantiations are replicable. Finally, the portability belief over classifier ensemble members for the initiation and termination sets is updated.
Before updating the portable option, we impose a check to ensure that the Markov classifiers are "well-trained".. We expect low classification loss for true positive states and, therefore, do not instantiations with a classification loss above a chosen threshold to update the portable option. This because, if feature that of previously seen states the classifier deemed positive.
portable, positive states should be almost indistinguishable from
After the validation phase, valid option instantiations are used improve the portable option. This is done by further training the portable initiation and termination sets samples from the Markov classifiers. The portable policy, however, continues train during the validation stage.
4.3 MONTEZUMA RESULTS
In this section we validate portable option in the Atari game Montezuma's Revenge, in which the player must navigate character through multiple rooms filled with enemies, obstacles, and trea- sures. We define three options that can be utilized in multiple rooms across the game in Table The initiation and termination sets for these options hand-labeled and only used for initial training (see Figure 5) and evaluation but could principle be learned by any skill discovery method (e.g. Mannor al. (2004); Şimsek & Barto (2004); McGovern & Barto (2001)).
The portable option is initially trained on the first room of Montezuma's Revenge, then trained to generalize subsequent rooms. The portable option receives stack of four 56 x 40 images around

---

Under review as a conference paper at ICLR 2023
OPTION
INITIATION
TERMINATION
DESCRIPTION
Climb
The agent is on or above ladder.
The agent on a platform.
The agent climbs down the ladder to the next valid platform.
Down
Ladder
Walk Left
The agent can move left without dying.
The agent has reached the left edge of a platform or ladder.
The agent moves left until the platform ends ladder is found.
Move Left of Enemy
The agent to the right of an en- emy.
The agent is to the left of an enemy.
The agent must navigate the left side of the enemy without dying. We de- fine a separate option for each enemy because they require different policies.
Table 1: A description of options used in Montezuma's 's Revenge. These options are selected because they should be generalizable to many instances in the game.
the player as input. During initial training, we allow the ensemble of classifiers and ensemble of policies to train until convergence, using the hand-designed initiation and termination sets to provide labels for the classifiers. We then generalize the learned option by training relevant task instances multiple other rooms across the first level of the game, without hand- designed information. For example, after initially training to descend the ladders of the first room, we place the agent at the top of each ladder in subsequent rooms, and continue training the option on the new ladders. The portable option trains on each new task 100 times through creating Markov classifiers, updating its beliefs, and training the ensemble of policies.
a collection of other
While after initial training the agent is required determine when an option execution was correct without guidance from the environment, we still need way to quantify option generalization to validate our approach. We therefore use hand-designed initiation and termination sets for validation and record the final distance from the true termination after option execution, normalized by the full option distance. perfect execution would result in a distance of O. If the agent does not attempt the option due to the initiation set not triggering, the distance is 1. If the agent dies during the option execution, ending the episode, the recorded distance again
We record the option performance after initial training and after each subsequent training task (see Figure 6 This is done by running the portable option on all defined tasks and reporting the per- formance of the option on previously seen tasks and unseen tasks separately. The performance on previously seen tasks is an indicator of whether the option has deteriorated while performance on unseen tasks shows the option's 's ability to zero-shot generalize. During evaluation, the portable op- tion attempts each task 50 times and cannot make updates to its portability beliefs or ensemble of policies and classifiers. Additionally, no Markov classifiers are created. We perform this experiment with multi-head ensemble which has 8 attention heads for the initiation and termination sets and attention heads for the option policy. The experiment repeated on a single-head ensemble which uses one head for all option components and so does not learn diverse features.
Figures 6a and 6b show that the multi-head ensemble outperforms the single-head ensemble. Addi- tionally, as more tasks are seen, the multi-head ensemble's 's ability to zero-shot generalize improves, indicating that the diverse features learned by the portable option helps generalization.
For Climb Down Ladder, task 4 and all tasks after task 6 require the agent to learn to avoid an enemy, adding complexity to the initial training task as making contact with an enemy considered failed execution. While the ensemble is able to complete these tasks before they are seen during training, it clear that training on these tasks heavily deteriorates the original option. After all tasks are seen, the option fails on all previous tasks. This is because the ensemble loses confidence in all classifier ensemble members as the agent fails to execute the option. However, while all tasks require the agent to descend a ladder, it can be argued that ladders with enemies may require different option. Therefore, we likely need a mechanism that allows the ensemble to decide when a new option instantiation should become new option, which we leave for future work.
8

---

Under review as a conference paper at ICLR 2023
2.75
nelilecd seen meBlbece-saae
miBileed- -2tsee
sigletees-
segleeeet -poser
30-
3.25
08
08
0.50
0.25
00
Ceandee ircoudeed
0.00
Nambor
Swan
(a) Mean normalized distance from true termination for the Climb Down Ladder option in Montezuma's Revenge averaged over 5 seeds.
(b) Mean normalized distance from true termination for the Walk Left option in Montezuma's Revenge eraged over 5 seeds.
3.--
Stageoingle hood
single
Saare single booe Jesect
single Fwed -3558
Spicer mull-hee
Spicer single heed Seen
32
Spice single hood Jrocse
10-
14
08-
LG
4.5
20 4.2
100 250
1.0
2000
200
2.75
200
(c) Mean normalized distance from true termination for the Move Left of Enemy option for spiders and snakes in Montezuma's Revenge averaged over 5 seeds.
(d) Mean normalized distance from true termination for the Move Left of Enemy option for rolling skulls in Montezuma's 's Revenge averaged over 5 seeds.
Figure 6: Mean normalized distance from true termination for all Montezuma's revenge experi- ments.
We a similar trend in the Walk Left option, where task 2 takes place in room with disappearing lasers. Due to the four frame stack, this is a partially observable hazard and so appears unpredictable to the agent. We do see, however, that the option is able to recover from this task, with overall performance on all seen tasks decreasing over time.
Avoiding the spider and snake enemies is harder than avoiding the rolling skull enemy. This is evident in Figures 6cc and 6d where the multi-head ensembles perform better on the rolling skull compared to the snakes and spiders. However, we see again that the multi-head ensembles deteri- orate over time, occasionally performing worse than the single-head ensembles. Similar to Climb Down Ladder, the multi-hean ensembles lose confidence in all classifiers in the ensemble if the agent fails the option. This shows that the ability to identify valid option executions is critical to obtain portable options.
5 CONCLUSION
In this work, we considered the problem of learning portable options. We proposed the use of an caattion-bbsed ensemble that learns multiple, minimally-overlappinn feature sets to increase the probability of learning a portable feature set. We used this ensemble learn a portable policy which we tested on the Procgen benchmark. On six out of eight games we found that the ensemble of policies improved performance, leading to faster generalization across levels. We then extended the attention-based based ensemble to classifiers, learning portable initiation and termination sets. We combined the ensemble of policies and ensemble classifiers to form a portable option which we tested on the Montezuma'ss 's Revenge Atari game. We found that portable options were able reuse learned option in unseen rooms, For the portable option to improve over subsequent task executions it is critical to identify valid option executions.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Picter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pp. 501-510. PMLR, 2018.
Andrew G Barto and Sridhar Mahadevan, Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(1):41-77, 2003.
Tony Chan, Gene Golub, and Randall Veque. Updating formulae and pairwise algorithm for computing sample variances. In COMPSTAT 1982 5th Symposium held at Toulouse 1982, pp- 30-41. Springer, 1982.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 2014.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference machine learning, pp. 2048- 2056. PMLR, 2020.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint urXiv:1703.02945 029499999 2017.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learn- ing Representations, 2018.
Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, and Keunjoo Kwon. Attention-based ensemble for deep metric learning. In Proceedings of the European conference on computer vision (ECCV), pp. 736-751, 2018.
Roberta Klatzky. Allocentric and egocentric spatial representations: Definitions, distinctions, and interconmections. In Spatial cognition, pp. 1-17. Springer, 1998.
George Dimitri Konidaris and Andrew G Barto. Building portable options: Skill transfer in rein- forcement learning. In IJCAI, volume 7, pp. 895-900, 2007.
TL LAI. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6: 4-22, 1985.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-137, 2016.
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand- eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421 436, 2018.
Shie Mannor, Ishai Menache, Amit Hoze, and Uri Klein. Dynamic abstraction in reinforcement learning via clustering. In Proceedings of the twenty-first international conference on Machine learning, pp. 71, 2004.
Amy McGovern and Andrew Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. 2001.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51877540)559--333 2015.
10

---

Under review as a conference paper at ICLR 2023
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Özgür Şimşek and Andrew G Barto. Using relative novelty to identify useful temporal abstractions in reinforcement learning, In Proceedings of the twenty-first international conference on Machine learning, pp. 95, 2004.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, (12(1-)) 211. 1999.
Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: survey. Journal of Machine Learning Research, 10(7), 2009.
Nicholay Topin, Nicholas Haltmeyer, Shawn Squire, John Winder, James MacGlashan, al. Portable option discovery for automated learning transfer in object-oriented markov decision pro- cesses. In Twenty-Fourth International Joint Conference Artificial Intelligence, 2015.
A ENSEMBLE BELIEF UPDATE DERIVATION
Because the output of a classifier 0g, is value between 0 and 1, we use a Beta distribution for the prior P(O:)
P(O.) = B(a, B)
00 (1 8-1
where and 3 are selected to reflect our initial beliefs. Given data D; - if i made correct prediction the likelihood classifier successful modelled binomial distribution
12 0% 0,)
P(D.).))
Using Bayes rule, update belief in each ensemble member using
P.O.)D.) = B(k a,n k + ()) k - 0
(5)
B ARCHITECTURE AND HYPERPARAMETERS
B. ATTENTION-BASEE ENSEMBLE ARCHITECTURE
The attention-based ensemble has three components, a spatial feature extractor, global feature ex- tractor and attention module. The spatial feature extractor consists of convolutional layer with 32 channels, kernel size of 3 and stride 1 followed by max pooling layer with a kernel size of 2. Each attention module a convolutional layer with 32 channels and has kernel size of and stride We do not include bias in this layer. Furthermore, the attention outputs are normalized to between 0 and with min-max normalization. The global feature extractor consists a convolutional layer, with 64 channels, kernel size of 3, and stride 2, followed by max pooling layer with kernel size of 2. The output will also be flattened into a mie-dmmniion feature embedding.
For experiments Montezuma's Revenge, the global feature extractor has appended Gated Recurrent Unit (Cho et al.] 2014) layer with a hidden size 128. We found having this layer to improve generalization performance.
11

---

Under review as a conference paper at ICLR 2023
B.2 PPO
Our PPO policy network consists of two linear layers, where the output feature size is 256 and 15, respectively. The weights of the second layer are initialized with (semi) orthogonal matrix, and the bias to O. The output of the policy is fed into the value network to predict the value. The value network is simply a linear layer with output feature size
The PPO agents are optimized according to the loss function proposed by Schulman et al. At time step i, the loss of the network 0 is the combination of three parts
2017
1t
C.III+V (0) = R,, I. T.C (0) - eq (0) + (=)))
The clipped objective is
CLIP (0) = ()))n(rr(O)AA clip(r.(0, - e, e))
where Ae is an advantage estimator. The value function loss is
D (Vo(se) )2
Finally, S denotes an entropy bonus.
B.3 Q NETWORK
For experiments on Montezuma's Revenge, the ensemble of policies has the same structure of an attention-based ensemble followed by policy networks. Each policy network is a function, parame- terized by neural network with two linear layers. The first layer outputs embeddings of dimension 64, and the second layer outputs q values of dimension 18. The policy selects actions using an epsilon-grees scheme, where the epsilon parameter is being decayed linearly through time.
The updates the q function uses samples collected and stored in replay buffer. We weight each sample with Hindsight Experience Replay (Andrychowicz et al. 2017).
C ENVIRONMENT SETTINGS
C.I PROCGEN ENVIRONMENT
For all the experiments in Procgen, we use parallel environments to speed up learning. The dis- count factor is set to 7 0.999. We always center the image observations on the game character.
normalize the reward function using the Parallel Algorithm Chan et al. (982).
The results in Figure 4 report the discounted episodic return for every time step. Furthermore, the episodic rewards are smoothed over a sliding window of 100 episodes.
C.2 MONTEZUMA'S REVENGE ENVIRONMENT
We use frame stack of four previous time steps, where each frame is a crop of the game screen with the rectangular area around the player sprite, as input to the portable option. The size of the agent space 56 40 pixels centered around the player. During option training in the first room, we ignore the environment reward, and instead only reward +1 when an option succeeds, and otherwise.
While the portable option is trained using these cropped images, the Markov classifiers are trained on the player's position the game.
D EXPERIMENT HYPERPARAMETER
12

---

Under review as a conference paper at ICLR 2023
Table 2: PPO ensemble of policies hyperparameters.
HYPERPARAMETER
VALUE
C1
0.5
C2
0.01
Clip (likelihood ratio & value function)
0.2
Maximum L2 norm for gradient clipping
0.5
Agent update period
256
Batch size
1024
Number epochs per gradient update
Learning rate
x 10-4
X-return
0.95
Replay buffer size
10
Bandit exploration weight Embedding distance function
500
Euclidean distance
mydse
1000
Table 3: Q-function ensemble of policies hyperparameters.
HYPERPARAMETER
VALUE
Hidden layer output size
128
Discount rate
0.9
Target update interval
10
Exploration algorithm
Linear decay epsilon greedy
Initial epsilon
1.0
Final epsilon
0.01
Epsilon decay steps
106
Ensemble size
3
Replay buffer size
10a
Batch size
32
Learning rate
2.5 x 10-4
Bandit exploration weight
100
Embedding distance function
Euclidean distance
midio
Table Ensemble classifiers hyperparameters.
HYPERPARAMETER
VALUE
Ensemble learning rate
10-1
Classifier learning rate
10-2
100
B
10
Maximum loss
3x training loss
Initial embedding training epoch
100
Initial classifier training epoch
50
Batch size
32
Markov classifier interaction minimum
100
Maximum time steps before option timeout
50
Embedding distance function
Euclidean distance
Markov Classifier Architecture
Two-class support vector machine
Mdie
13


Reviews:
[{"review_id": "mjj_rlC39jZ", "reviewer": "Reviewer_rc41", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach for learning policies that can enable better generalization across tasks. This is done using an ensemble of learners, each of which uses attention to focus on a different part of the feature space, and then using bandit-exploration to select a particular policy. Further, the authors include an ensemble of classifiers to account for initiation and terminations sets, and argue that the combination of the classifiers and the policies give rise to transferable options. Generalization results are included on ProcGen, and Montezuma's revenge (across multiple levels). ", "summary_of_the_review": "Overall, I am not in favor of acceptance since the full presented system is evaluated for particular choices of options/initial/final conditions and specific tasks, while we have have algorithms from the self-supervised learning literature like Go-explore which can keep exploring and generalizing (for the same environment considered in this paper). The idea of learning diverse features to find one that's robust to new tasks seems interesting but performance is very dependent on ensemble size as described previously, and this will hinder widespread adoption. ", "strength_and_weaknesses": "Strengths \n1. Generalization through learning diverse representations\nUsing an ensemble of policies each of which focus on different parts of the input, and then selecting among them using a bandit-style objective to find the most robust policy is an intriguing idea. There has been work that focuses on learning the right representations of the environment to enable better generalization [1], and this paper approaches the same problem by instead encouraging diverse representations for multiple learners (enforced using distance in the embedded space), and then selecting for the most robust one. \n\nWeaknesses - \n\n1. Complexity of approach, specificity of Montezuma experiments, relation to exploration algorithms\n\nThe overall approach has a large number of moving parts - the policy ensemble, the classifier ensemble, the markov classifier. For the experiments included in particular in the paper (Montezuma's revenge), the initiation and termination conditions need to be specified by hand for the classifiers. Even the options are hand-specified, and not learned for the task (the authors seem to indicate that there is further training for these options, but this isn't clear). Also evaluation is done on specific tasks within the game (rolling skull, spider etc). Contrast this to results in self-supervised exploration. Go-Explore [2] was able to enable an agent to learn how to play Montezuma's revenge without any of these constraints (hand specified option/initiation/termination conditions) and without any rewards, and keep generalizing across different levels also without any restriction to the particular task considered. \n\n\n2. Sensitivity of ensemble size for policy ensemble - \n Even for the policy ensemble results on the ProcGen environment, the policy ensemble size needs to be tuned for each environment, and performance does not generally improve with more policies in the ensemble. The authors argue this is because 'too much time is spent exploring policies that do not generalize'. However, there is no analysis that shows this to be the case. It might be difficult for practitioners to adopt this approach if there is heavy reliance on tuning this hyper-parameter. The results on the ProcGen environments show that no single choice for ensemble size performs consistently (they each are close to the 1-member ensemble case in atleast one environment). \n\n\n[1]: Eysenbach, Ben, Russ R. Salakhutdinov, and Sergey Levine. \"Robust predictable control.\" \n[2] Ecoffet, Adrien, et al. \"Go-explore: a new approach for hard-exploration problems.\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "g6L8JNkCBl", "reviewer": "Reviewer_Q9hT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach for learning portable options by training and ensemble of options and enforcing that the ensemble members learn feature representations that attend to different parts of the observation. The approach then learns to determine the feature set that generalizes best across environments to choose the most portable option. The paper demonstrates learning of portable policies in ProcGen and portable options in Montezuma’s revenge.", "summary_of_the_review": "I am a bit torn about this submission. I really like the tackled problem and the approach seems interesting to me. There are some doubts about the empirical performance of the approach which I have outlined in “weaknesses” above. These would require a solid experimental evaluation to address. However, the experimental evaluation of the current submission is severely lacking. Thus I do not recommend acceptance of the paper in it’s current form and suggest the authors extend the experimental evaluation and resubmit their paper at ICLR or another fitting venue.", "strength_and_weaknesses": "# Strengths\n\n- the problem of learning portable options is very important: the whole point of learning options is to “port” them to different tasks, but often options trained on observations from one task overfit to spurious features and then don’t generalize well to another — approaches like the one proposed here, that aim to address this problem, are of high relevance to the community\n\n- the intuition of the proposed (learning ensembles with disjoint feature sets) is novel to my knowledge and intuitive\n\n- I like the structure of the paper, in which the authors first explain their idea in the context of policy learning and only afterwards extend it to the more complex case of option learning (which additionally requires to learn initiation and termination sets) —> disentangling the complexity in this way makes it much easier to follow the description of the method\n\n\n# Weaknesses\n\nThe main weakness of the submission is the experimental evaluation of the proposed approach. The paper proposes an approach for option learning, but never evaluates whether the learned options are actually useful for downstream learning. There are no comparisons to prior works and no ablation studies of the components of the proposed approach. There is no qualitative analysis of the learned feature representations, even though they are the main novelty proposed in the approach.\n\nThe experimental analysis will need substantial improvements to verify the usefulness of the approach.\n\n(1) Evaluations on using the options for downstream task learning should be added. \n\n(2) A comparison to (a) prior option learning works and (b) the exact same setup as the proposed approach but without the ensemble learning part should be included.\n\n(3) Detailed analysis experiments of the many moving parts of the proposed approach should be added: how do different number of ensemble elements or different regularizing coefficients influence the learned options?\n\n(4) Qualitative results and visualizations should be added, to show what features the ensemble identifies as transferrable, in what situations transfer works well (compared to baselines), in what situations the proposed approach fails etc\n\nThe current experimental evaluation does not allow to properly judge the proposed approach and, due to a lack of qualitative results, it is hard to understand why some of the trained options fail or why their performance *degrades* over training.\n\n-----\nApart from the experimental evaluation, there are a few additional weaknesses:\n\n(A) The description of the extension of the ensemble approach to learning options (instead of policies) was hard for me to follow, particularly I had trouble understanding the second-to-last paragraph in section 4.2 — some intuitive examples for options and feature sets, e.g. from Montezuma’s revenge, and a figure visualizing the approach could help here.\n\n(B) The proposed objective encourages the learned representations between ensemble elements to be disjoint — it seems to me that this can make optimization tricky since there are many local optima in which important features are “trapped” within disjoint feature sets of different ensemble members, making me wonder how hard it is to tune this approach on tasks that require the interplay of many features for policy learning.\n\n(C) More generally, like many option papers, the proposed approach has many moving parts (learning ensembles of policies, initiation and termination classifiers, additional regularization terms with coefficients that require tuning, bandit-optimization of which policy from the ensemble to pick for execution). This suggests that it will be hard / require a lot of tuning to apply this approach to complex problems. Again, this is generally true for many option learning approaches, but the paper introduces additional complexities that go beyond conventional option learning and exacerbate the problem.\n\n(D) The related work section misses some references to relevant work on learning state abstractions for generalization in RL. See e.g. Amy Zhang’s McGill PhD thesis from 2021 for an overview of relevant works.\n\n\n# Questions\n\n\n- Can we learn a single embedding space shared between initiation and termination classifiers and policy? This could substantially reduce the number of learned components and required tuning.\n\n- Fig 3 shows that the number of ensemble elements needs to be tuned separately for each task. Is there a good way to choose the number of ensemble elements without just trying all of them?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PnnfxyJDew", "reviewer": "Reviewer_8Rro", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of learning reusable option policies to achieve cross-task generalization using reinforcement learning. It describes a method for learning feature spaces that can be reused by an option-like policy for (1) determine the next action to take (2) whether it is executable and (3) whether it should terminate. The core technical component is an ensemble feature learner (Kim et al., 2018) that aims to discover a set of feature extractors that are both diverse and optimizes some objective. In this paper, the ensemble feature learner is used to discover features for an ensemble of RL-based option policies for optimizing task rewards. The same ensemble feature learner is also used to learn an ensemble of initiation set and termination set classifier. The core component of learning reusable policies using feature ensemble learner is evaluated on a set of Procgen environments. The full method (including the initiation and termination set classifier) is evaluated on the montezuma’s revenge domain, where different rooms are considered different environments. \n", "summary_of_the_review": "Overall I really like the premise and the motivation of the paper. However, the writing / clarity is lacking and the experiment evaluation is insufficient. ", "strength_and_weaknesses": "Overall I like the premise of the paper. Options should be able to port to other environments or different parts of the state space, as opposed to only function in the state distribution that it’s trained on. I also applaud that the paper chooses to evaluate its method on a pixel-input domain as opposed to more “convenient” environments where each input dimension is already meaningful.\n\nHowever, I have to admit that I have a hard time appreciating the method itself and the conclusion drawn by the evaluation. I also had a hard time understanding the method itself, despite multiple attempts at reading the section. Below I will describe my comments in detail.\n\nWriting quality\nThe structure of the paper is clear, and the introduction is enjoyable to read. However, the method section is difficult to understand, especially section 4.2 and 4.3 and their connections wit the previous subsections. Here is a list of points that I do not fully understand\nI understand why it’s difficult to evaluate the initiation condition, but if the method assumes a reward function is given, why can’t it use the reward function to determine the termination set?\nThe third paragraph of 4.2 says “If the initiation and termination set are both incorrect during the same option execution, the option will appear to succeed but will have an incorrect instantiation”. I understand false positives are likely, but the sentence appears to state that the option is guaranteed to succeed if both classifiers are wrong. Why is that?\nI’m generally confused about the role of the markov classifier. The paragraph says the markov classifier “learns the initiation or termination set of a new option instantions”, What does that mean?\nHow does markov classifier allow the option to “return to a specific instantiation”? Does it reset the simulator status?\nIn the 5th paragraph, the “classification loss” is used to determine policy success. The classification loss of which classifier? And why is loss used, not the prediction? And why is a true positive state “almost indistinguishable from the previously seen state the classifier deems positive” since the invariant feature should only be a subset of the entire space?\nHow is the portability estimate in 4.3 used by the classifier ensembles concretely?\nIn section 4.4, what exactly is the “true determination state” in eq 5? What is the “true” meant to contrast? \n\nMy second major comment is regarding the conclusion drawn in the experiments in general, especially in Section 3.2. The paper seems to imply that the improved performance is solely due to the reusable feature set extracted via the ensemble feature learning scheme. However, another factor that may play a critical role is the ensemble policies themselves. It is widely known that ensemble could aid exploration. Thus to correctly attribute the performance gain, It would be critical to ablate this factor and evaluate the ensemble RL policy without the feature learner. Similar counter arguments can be made for all later experiments, as they all rely on the same ensemble RL strategy. \n\nFinally, the paper has many moving parts and little ablation studies. The ambiguous evaluation metrics that entirely relies on the learned classifier makes the experiment results less clear. I believe the paper will benefit tremendously from a set of experiments using domains that support option-level skill evaluation with ground truth initiation and termination set. For now, it is really difficult to understand the validity of the conclusion drawn from the experiments.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject