The Skill-Action Architecture: Learning Abstract Action Embeddings for Reinforcement Learning

Abstract:   The option framework, one of the most promising Hierarchical Reinforcement Learning (HRL) frameworks, is developed based on the Semi-Markov Decision Problem (SMDP) and employs a triple formulation of the option (i.e., an action policy, a termination probability, and an initiation set). These design choices, however, mean that the option framework: 1) has low sample efficiency, 2) cannot use more stable Markov Decision Problem (MDP) based learning algorithms, 3) represents abstract actions implicitly, and 4) is expensive to scale up. To overcome these problems, here we propose a simple yet effective MDP implementation of the option framework: the Skill-Action (SA) architecture. Derived from a novel discovery that the SMDP option framework has an MDP equivalence, SA hierarchically extracts skills (abstract actions) from primary actions and explicitly encodes these knowledge into skill context vectors (embedding vectors). Although SA is MDP formulated, skills can still be temporally extended by applying the attention mechanism to skill context vectors. Unlike the option framework, which requires M action policies for M skills, SA's action policy only needs one decoder to decode skill context vectors into primary actions. Under this formulation, SA can be optimized with any MDP based policy gradient algorithm. Moreover, it is sample efficient, cheap to scale up, and theoretically proven to have lower variance. Our empirical studies on challenging infinite horizon robot simulation environments demonstrate that SA not only outperforms all baselines by a large margin, but also exhibits smaller variance, faster convergence, and good interpretability. On transfer learning tasks, SA also outperforms the other models and shows its advantage on reusing knowledge across tasks. A potential impact of SA is to pave the way for a large scale pre-training architecture in the reinforcement learning area.

Reviews and author responses:

We thank reviewer1 very much for his feedback. Reviewer1 clearly read our paper very closely and has a deep understanding of SA. We thank him very much for fairly acknowledging our contributions and providing his very constructive feedback. Other than this response we have also provided a general response to common questions from all reviewers. Please refer to our top review for more details.

Strengths
- The new value function having lower variance while being an unbiased estimate of the traditional value function is a useful result 

  We thank reviwer1 very much for picking this up. This is actually one of our most important contributions. However, we focus this paper on proposing SA and omitted its broad impacts in our first version to avoid over-claim. We have added those discussions back in Appendix A.1. Please refer to it should you be interested.

Weaknesses

- The choice of experiments

  Due to computation resource limitations, we can only finish transfer learning at the current stage. This paper now reproduces all DAC's experiments and SA ranks the first on 5 out of 6 transfer learning environments. Should this paper be accepted, we would love to provide robot manipulation experiments in future months.

- The skill activation pattern in Figure 3 seems to suggest that only one or two skills are being learned effectively

  This is our favorite feedback. Reviewer1 has a deep insight into SA and also identified a very important connection between SA and the causal reinforcement learning framework. When $M=1$, SA actually falls back to GPH-MDPs, which is used to encode environment-action causal relationship. With $M>1$, SA also encodes temporal causal relationships. SA is the first framework concerning such causal relations. However, this topic is beyond the scope of this paper and we provide a brief discussion in Appendix A.1.3.

  As for the failure of coordinating multiple skills, this is also an innate deficiency of SA which can be solved by extending SA into a deep (more hierarchies) wide (higher-order dependencies) architecture. We provide a detailed explanation in Appendix A.1. Please refer to it should you be interested.

- The explanation provided for lack of improvement on finite horizon tasks seems weak.
  
  Indeed it was. We were trying to avoid over-claiming and omitted some important content of SA. This problem is actually because  SA is unable to learn fine-grained skills using conventional value functions. We give a detailed explanation in Appendix A.1.1

- Footnote 4 mentions the Reacher environment and states that SA performs 38% better than

  We really appreciate reviwer1 give our paper a close read. The difference on Reacher is hard to observe because it has negative reward and the highest reward is 0. So all algorithms collapsed together in our figure. However, SA is roughly -4.7 v.s. the second best AHP is roughly -6. So this is not a typo, it is just not easy to be observed in figure.

- Feedbacks on typos:
  We thank reviwer1 very much for picking these up. Those are clearly our typos and have been changed accordingly.

Thank you again for your time and patience. Have a nice day!



We thank reviewer3 for his feedback. We really appreciate Reviewer3's deep understanding of SA, especially SA's deficiencies and potential solutions. We thank him very much for his constructive feedback. Other than this response we have also provided a general response to common questions from all reviewers. Please refer to our top review for more details.

- More related works

  References provided here are closely related works to SA. Our original scope was to discuss SA under the option framework. We had to omit these due to the page limitation. In this revision, we provide more discussions in Appendix A.2

- I think HO2 (Wulfmeier et al., 2020b) is a quite related work that appears to outperform SA. 

  We apparently missed HO2. Many thanks for referring us to it. It will be very interesting to see how SA performs under HO2's training framework. 

  HO2 is a very closely related work to SA. HO2 provides a novel off-policy optimization framework and performance improvements of HO2 are very impressive. We have cited HO2 in both Section 2 and Appendix A.2. However, SA and HO2 complements the option framework from different aspects: SA is proposed to improve option's sample efficiency from an SMDP to MDP perspective. Another important contribution of SA is to improve option's scalability. HO2 improves option from an off-policy optimization perspective. As we mentioned in our paper, SA and HO2 are beneficial to each other: HO2 provides an efficient off-policy algorithm which significantly outperforms on-policy algorithms that SA suffers from. On the other hand, due to the SMDP formulation, HO2 has to introduce the maximum number of switches as a hyper-parameter and marginalize over it to ensure temporal consistency. With SA's MDP formulation and attention mechanism, HO2 can improve temporal consistency from a much more efficient and data-driven manner.

- Q: I think the paper would be stronger if there were more environments in slightly different domains (perhaps in a manipulation setting); Q: Similarly, learning skills/options seem particularly promising in a transfer setting. I think transfer experiments would strengthen the paper.

  A: Due to computation resource limitations, we can only finish transfer learning at the current stage. This paper now reproduces all DAC's experiments and SA ranks the first on 5 out of 6 transfer learning environments. Should this paper be accepted, we would love to provide robot manipulation experiments in future months.

- The SA architecture as presented seems unlikely to learn multiple useful skills...

  This is one of our favorite comments. The reviewer has a deep understanding of such frameworks and proposed a very practical solution to this problem and we share the same perspective. We provide a detailed answer in Appendix A.1 for your reference.

  Briefly speaking, "the dominant skill problem" (term defined in DAC) is an innate design deficiency caused by the conventional value functions. Even though SA is not enough to solve this problem, a Deep Wide SA (DWSA) which employs SA's "deep (more hierarchies) wide (higher-order dependencies) value function" could be a solution to this problem. As we explained in Appendix A.3, SA has a strong connection with causal reinforcement learning. Another important solution is to add a causal reward in the objective function. SA is carefully designed to make the most out of these improvements. Advantages of SA from this perspective is further explained in Appendix A.1.2.

  As for disentangled skills, easy to ensuring disentangled skills is one advantage of SA. It is as easy as adding a distance regularizer between skill vectors in the objective function. We actually have implemented this in our submitted code. However, SA actually does not suffer an entangled skill problem. Adding this regularizer actually decreases the performance.  Therefore, we did not mention this in our paper. In experiments, we found that skills distance is diverse throughout episodes. The real problem is with conventional value functions, SA is unable to learn fine-grained skills as well as coordinate skills.

  To show this we have visualized two episodes of HalfCheetah: https://www.youtube.com/watch?v=QiLVZvI6NJU and HumanoidStandup: https://www.youtube.com/watch?v=xrfxbI3duBM. In which SA does learn disentangled skills (HalfCheetah: run forward; re-balance when fall) (HumanoidStandup: sit-up; persist sitting position). The main problem is SA only able to learn coarse skills as explained in Appendix A.1.1 the last paragraph.

- The multi-head attention aspect seems like an architectural choice,

  Indeed multi-head attention is a design choice. However, we consider this as our main contribution and explain this in Appendix A.1.2.

- Thank you very much for picking this up. We have changed all games to environments accordingly.

Thank you again for your time and patience. Have a nice day!

We thank reviewer2 for his detailed and constructive feedback. Other than this response we have also provided a general response to common questions from all reviewers. Please refer to our top review for more details.

- It is mentioned that SA has faster learning speed because it only has one action policy decoder, unlike DAC and AHP. Do all of the methods have the same number of learnable parameters for fair comparison?

  Yes. SA uses the same FFN (both size and architecture) action policy as DAC. There are two differences in SA's implementation: (1) SA's action policy takes more inputs than DAC because of the skill embedding vector. (2) SA only has one such action policy neural network v.s. DAC employs 4 of them to use.

- SA outperforms baselines on infinite-horizon tasks, but achieves similar performance on finite-horizon tasks. 

  This is an innate deficiency of the option framework. On other implementations such as DAC, there were no improvements observed in single task environments. Even though SA only has improvements on infinite-horizon tasks, as we explained in Appendix A.1, an SA-style **deep wide value function** (proposition 3.4 and 3.5) provides a promising solution to this kind of problem.

- I think more experiments on finite vs. infinite horizon tasks would be useful to the research community, e.g., compare SA vs. baselines on the same task as you increase the time horizon.

  Due to computation resource limitations, we can only finish transfer learning at the current stage. This paper now reproduces all DAC's experiments and SA ranks the first on 5 out of 6 transfer learning environments. Should this paper be accepted, we would love to provide robot manipulation experiments in future months.

- Insufficient related work

  Our original scope was to discuss SA under the option framework. Indeed our work is closely related to references especially PEARL. Due to page limitation, we provide more discussions in Appendix A.2

-  more difficult long-horizon continuous control task 

  As explained above, we are not able to finish these experiments at the current stage. We will definitely run those experiments in future work.

- Minor typos

  We thank our reviewer very much for picking these up and have changed accordingly.

Thank you again for your time and patience. Have a nice day!

We thank the reviewers for their detailed and constructive feedback. Below we comment on common points and highlight some changes that we have made as a response.

This paper devises SA as a candidate for a general large-scale pre-training framework in reinforcement learning. Main efforts are spent on proving the equivalence between MDP and SMDP, extending the conventional value function to ``skill value upon arrival function'', and deriving policy gradient theorems. Our experiments show that SA achieves significant sample efficiency improvements on infinite horizon single task environments.

Considering that all reviewers expressed common interests in the transfer learning setting. In this revision, we have added Section 4.4 to address SA's advantages of transfer learning tasks. We have reported all experiments from DAC. SA significantly outperforms DAC on 4 out of 10 single task environments and ranks the first on 5 out of 6 transfer learning environments.

Reviewers also wanted to understand how SA reduces the following two limitations of the option framework: (1) failure to improve the performance and the sample efficiency on finite horizon environments (section 4.1); (2) “the dominant skill problem” (section 4.2). Both of them are actually innate limitations of the option framework and have also been reported in the previous option implementations, such as DAC (Zhang & Whiteson, 2019).

Although SA empirically shares these two innate limitations, in Appendix A.1 we conceptually show that SA-style wide (higher-order dependencies) value functions could be a solution to both limitations. In Appendix A.1.1, we explained how both problems are connected to the innate deficiency of the value function employed by the option framework and why SA's "deep (more hierarchies) wide (higher-order dependencies) value function" could be a solution to this problem. In A.1.2, we give thorough explanations of motivations and advantages of design choices behind SA and how could they contribute to solving this problem. In A.1.3 we further explain SA's close relationship to causal reinforcement learning and how a causal reward can be devised to facilitate learning multi-levels granularity of skills.

In the first version, due to page limitation, we omitted several closely related works that are beyond the scope of this paper. As reviewers have suggested, they are beneficial to be discussed for readers' reference. Therefore, we include them in Appendix A.2.

We thank the reviewers again for their help in strengthening the paper and hope that all questions have been answered in the individual sections.

### Summary
The paper proposes the "Skill-Action" architecture with an MDP formulation, inspired by their novel discovery of an equivalence of Semi-MDPs and MDPs under some assumptions. The benefit of the MDP formulation is that a single skill-policy (i.e. policy that selects what skill to follow at a time step) and a single action-policy (for decoding primitive actions given skills) is learnt as opposed to learning an individual policy for each skill (as is the case in prior work on options framework, option-critic, etc) and no explicit termination function is required. The paper defines a new value and Q function to accomodate the skill context vector at each time step and derives the recursive bellman equations and policy-gradient update steps for the proposed formulation, stating that the new value function has lower variance than the conventional value function. Empirical evidence shows better convergence and overall performance on infinite horizon tasks but not for the finite horizon case. Interpretation of skill context vectors shows that the skill embeddings correspond to distinct behaviors in the HalfCheetah environment.

### Strengths
- The paper is the first to propose a concrete formulation of MDP-based skill learning with a single {skill policy, action policy} pair and with policy-gradient updates. The equivalence between semi-MDPs and MDPs is also novel.
- The new value function having lower variance while being an unbiased estimate of the traditional value function is a useful result (I have not verified the proof given in the appendix).
- Performance on three infinite horizon MuJoCo environments is particularly strong (HalfCheetah, Swimmer, HumanoidStandup), though the choice of comparing results at 1 million steps as opposed to the 2 millions steps used in prior works hides the overall picture of whether the baselines eventually reach the same performance or not. Nevertheless, convergence (sample efficiency) of the proposed method is higher on these environments.

### Weaknesses
- The choice of experiments seem to be slightly cherry picked. Only 4 infinite horizon MuJoCo envs are chosen out of which 3 show good performance whereas all of the finite horizon environments do not show any improvements. Given that no significant gains were seen in the finite-horizon setting, I am surprised to not see more experiments demonstrating the efficacy of the proposed method on more infinite-horizon environments.
- The skill activation pattern in Figure 3 seems to suggest that only one or two skills are being learned effectively by the proposed algorithm among the 4 possible skills to be learnt (the authors interpret this as the forward movement skill in HalfCheetah and the recovery skill for balancing itself). Firstly, the low number of skills to start out with seems quite limiting -- why not try any larger values? Secondly, given that just one skill is mainly used, this setting effectively reduces to the PPO baseline with a single policy, yet the proposed SA+PPO algorithm performs significantly better than all baselines. I suspect that the reason for increased performance may not necessarily come from having multiple skills and this can be verified by adding another baseline to all plots in Figure 2 -- the proposes SA + PPO algorithm with just 1 skill instead of 4 skills. If this baseline is same as PPO, then the conclusion can be drawn that SA + PPO effectively coordinates the skills learnt and this coordination is the reason for improved performance.
- The explanation provided for lack of improvement on finite horizon tasks seems weak. I would argue that a good skill-learning algorithm should be able to pick up temporally abstracted skills for completing episodic tasks. For environments like HalfCheetah, if one skill is responsible for making the cheetah move forward, then it is basically equivalent to learning a single policy (PPO baseline for example) to move forward, which is the only way to achieve positive reward. Rather than showing improved performance on environments such as HalfCheetah or Swimmer, some more carefully chosen environments would have been better to demonstrate a diverse set of skills which are essential for solving the task (e.g. HumanoidStandup or a non-MuJoCo environment with a discrete state space).

### Other issues/comments
- Footnote 4 mentions the Reacher environment and states that SA performs 38% better than the second best on this env. However, the footnote link in the main text talks about finite horizon envs (HumanoidStandup in particular) and not Reacher. Further, the plot in Fugure 2 does not show the 38% better performance on Reacher. This may be typo, but correct me if I am wrong.

### Feedback to authors
- $Q_{O}[\hat{o}_{t},s_t]$ are $Q_{O}[s_t,\hat{o}_{t}]$ used interchangeably in Eqns 5, 6. It would be good to have a consistent notation for position of state and context vector.
- Section 3.1 para 2, last line: This sentence does not make sense.

This paper introduces the skill-action (SA) architecture, an instantiation of the options framework that can be optimized with standard MDP algorithms (such as PPO). The SA architecture uses a latent one-hot vector to specify individual skills. The action decoder is conditioned on learned embedding vectors corresponding to the skills. Skills can be temporally extended through a skill policy with an attention mechanism. In the empirical evaluation the SA architecture trained with PPO outperforms baselines from the DAC codebase in infinite horizon environments and performs comparable to the baselines on finite horizon tasks.

The paper is generally clearly written and, as far as I can tell, technically correct. The results on the OpenAI Gym are encouraging and improvements in HRL would certainly be significant and of interest to the ICLR community. 
However, I have some concerns about the paper in its current form:
- Firstly, I think that the formulation is close to works in the broader HRL literature that use latent variables to learn skills without the options interpretation. Some (not particularly representative) examples include (but are certainly not limited to) Hausman et al. (2018), Tirumala et al. (2019), Wulfmeier et al. (2020a). This literature should be discussed in the related works section.
- I think HO2 (Wulfmeier et al., 2020b) is a quite related work that appears to outperform SA. I recognise that it is a recent paper but it should at least be discussed in the paper.
- I think the paper would be stronger if there were more environments in slightly different domains (perhaps in a manipulation setting).
- Similarly, learning skills/options seem particularly promising in a transfer setting. I think transfer experiments would strengthen the paper.
- The SA architecture as presented seems unlikely to learn multiple useful skills since there is nothing in the objective that encourages using different skills or diversity among skills. Indeed it seems that the method typically collapses to using one skill almost exclusively (figure 3). I am willing to change my mind on this point if the authors exhibited a scenario where the SA architecture reliably discovers and uses several skills.
- The multi-head attention aspect seems like an architectural choice, rather than something required by the framework. I would like to see an ablation of this architecture.

For these reasons, I think the paper should be rejected in its current form.
Minor comments:
I would recommend changing 'games' to 'environments' throughout the paper.
References:
- Hausman et al., Learning an Embedding Space for Transferable Robot Skills, ICLR 2018, https://openreview.net/forum?id=rk07ZXZRb
- Tirumala et al., Exploiting Hierarchy for Learning and Transfer in KL-regularized RL, https://arxiv.org/abs/1903.07438
- Wulfmeier et al., Compositional Transfer in Hierarchical Reinforcement Learning,  RSS 2020,  https://arxiv.org/abs/1906.11228
- Wulfmeier et al., Data-efficient Hindsight Off-policy Option Learning, https://arxiv.org/abs/2007.15588

This paper shows that the semi-MDP option framework has an MDP equivalence, by adding extra dependencies into the master policy. The proposed method (“Skill-Action” architecture) trains a skill policy which marginalizes those dependencies, allowing the master policy to be updated at each time step. SA encodes primary actions to skill context vectors, and applies the attention mechanism to these vectors to temporally extend the skills. 

Pros:
a. The paper is well-written and well-motivated.
b. The proposed method outperforms baselines on infinite-horizon mujoco locomotion tasks, and achieves comparable performance as baselines on finite-horizon mujoco tasks.
c. I think the empirical analyses of the learned skills, especially Figure 4, is interesting and valuable to the RL community.

Cons / Questions:

1. It is mentioned that SA has faster learning speed because it only has one action policy decoder, unlike DAC and AHP. Do all of the methods have the same number of learnable parameters for fair comparison?

2. SA outperforms baselines on infinite-horizon tasks, but achieves similar performance on finite-horizon tasks. The authors “conjecture that finite horizon games contain less significant temporal relationships than infinite horizon games”, but I would have expected the opposite (?). I think more experiments on finite vs. infinite horizon tasks would be useful to the research community, e.g., compare SA vs. baselines on the same task as you increase the time horizon.

3. Insufficient related work: There is a plethora of prior work on skill context vectors for RL (e.g., [1,2,3]) but they are not discussed or compared to:
[1] PEARL - Rakelly et al. 2019 https://arxiv.org/abs/1903.08254
[2] InfoGAIL - Li et al. 2017 https://arxiv.org/pdf/1703.08840.pdf
[3] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning - Gupta et al. 2019 https://relay-policy-learning.github.io/

4. Since this is a work on hierarchical RL, the paper could be made stronger with experiments on a more difficult long-horizon continuous control task with distinct skills, like robotic manipulation, or Ant locomotion around a small maze.

Minor typos:
p.2: “while improves their functionalities” -> “while improving their functionalities”
p.2: “Unlike the option framework requires M action policies” -> “Unlike the option framework, which requires M action policies”


Meta-review: This paper proposes the Skill-Action (SA) architecture, based on the insight that semi-MDPs in the option framework can be posed as an equivalent MDP. The paper presents interesting theoretical results and very promising empirical results.
We thank the reviewers for their revisions, which provided more insights into the method. Of particular interest was the discussion on the "dominant skill problem".
We still feel that the paper would benefit from additional experiments, as discussed in detail in all of the reviews. I believe with this inclusion this will be an impactful paper.

Decision: withdrawn-rejected-submissions