Reinforcement Learning with Competitive  Ensembles of Information-Constrained Primitives
Authors: Anirudh Goyal
Shagun Sodhani
Jonathan Binas
Xue Bin Peng
Sergey Levine
Yoshua Bengio
Venue:  (2020)
Keywords: Reinforcement Learning
Variational Information Bottleneck
Learning primitives

Abstract: Reinforcement learning agents that operate in diverse and complex environments can benefit from the structured decomposition of their behavior. Often, this is addressed in the context of hierarchical reinforcement learning, where the aim is to decompose a policy into lower-level primitives or options, and a higher-level meta-policy that triggers the appropriate behaviors for a given situation. However, the meta-policy must still produce appropriate decisions in all states.
In this work, we propose a policy design that decomposes into primitives, similarly to hierarchical reinforcement learning, but without a high-level meta-policy. Instead, each primitive can decide for themselves whether they wish to act in the current state.
We use an information-theoretic mechanism for enabling this decentralized decision: each primitive chooses how much information it needs about the current state to make a decision and the primitive that requests the most information about the current state acts in the world. The primitives are regularized to use as little information as possible, which leads to natural competition and specialization. We experimentally demonstrate that this policy architecture improves over both flat and hierarchical policies in terms of generalization. 

Published as a conference paper at ICLR 2020
REINFORCEMENT LEARNING WITH COMPETITIVE ENSEMBLES OF INFORMATION-CONSTTAANNDDD PRIMITIVES
Anirudh Goyal', Shagun Sodhani?, Jonathan Binas', Xue Bin Peng', Sergey Levine Yoshua Bengio*
ABSTRACT
Reinforcement learning agents that operate in diverse and complex environments can benefit from the structured decomposition of their behavior. Often, this is addressed in the context of hierarchical reinforcement learning, where the aim is decompose a policy into lower-level primitives or options, and higher-level meta-policy that triggers the appropriate behaviors for a given situation. However, the meta-policy must still produce appropriate decisions in all the states. In this work, we propose policy design that decomposes into primitives, similarly to hierarchical reinforcement learning, but without an explicit high-level meta-policy. Instead, each primitive can decide for themselves whether they wish to act in the current state. We use an ceemmatioo-thhoorete mechanism for enabling this decentralized decision: each primitive chooses how much information it needs about the current state to make a decision and the primitive that requests the most information about the current state acts in the environment. Regularizing the primitives to use as little information as possible leads to natural competition and specialization. We experimentally demonstrate that this policy architecture improves over both flat and hierarchical policies in terms of generalization.
INTRODUCTION
Learning policies that generalize to new environments or tasks is a fundamental challenge in re- inforcement learning. While deep reinforcement learning has enabled training powerful policies, which outperform humans on specific, well-defined tasks (Mnih et al. 2015), their performance often diminishes when the properties of the environment the task change to regimes not encountered during training.
This is in stark contrast to how humans learn, plan, and act: humans scamlessly switch between different aspects of task, transfer knowledge to new tasks from remotely related but essentially distinct prior experience, and combine primitives (or skills) used for distinct aspects of different tasks in meaningful ways to solve new problems. A hypothesis hinting at the reasons for this discrepancy is that the world is inherently compositional, such that its features of small sets of primitive mechanisms (Parascandolo et al., 2017). Since humans seem to benefit from learning skills and learning combine skills, might be useful inductive bias for the learning models as well.
be described by compositions
This is addressed to some extent by the hicrarchical reinforcement learning (HRL) methods, which focus on learning representations at multiple spatial and temporal scales, thus enabling better explo- ration strategies and improved generalization performance (Dayan Hinton 1993 Sutton al. 1999b Dietterich] 2000 Kulkarni et al.) 2016). However, hierarchical approaches rely on some form of learned high-level controller, which decides when to activate different components in the hierarchy. While low-level sub-policies can specialize to smaller portions of the state space, the top-level controller (or master policy) needs to know how to deal with any given state. That is, should provide optimal behavior for the entire accessible state space. As the master policy is trained
Mila, University of Montreal; Facehook AT Research; work done while the author at Mila, University of Montreal; University of California, Berkeley. an rudhgoya 1 91 9m0ma 1 cco

---

Published as a conference paper at ICLR 2020
on particular state distribution, learning it in way that generalizes to new environments effectively becomes the bottleneck for such approaches (Sasha Vezhnevets et al. 2017 Andreas et al. 2017).
selection p(k) & efuu
(C Cx)
DRL(IN
IL
rechasiin
(Z) ZK)
Tig
decoder
Goegettioral
encoder
state
action a
Figure 1: Illustration of our model (Left): An intrinsic competition mechanism, based on the amount of information each primitive requests, is used to select primitive to be active for given input. Each primitive focuses on distinct features of the environment: in this case, one policy focuses on boxes, a second one gates, and the third one spheres. Right: The primitive-selection mechanism of our model. The primitive with most information acts in the environment and gets the reward.
We argue, and empirically show, that in order achieve better generalization, the interaction between the low-level primitives and the selection thereof should itself be performed without requiring single centralized network that understands the entire state space. We, therefore, propose decentralized approach as alternative to standard HRL, where only learn set of low-level primitives without learning an explicit high-level controller. In particular, we construct a factorized representation of the policy by learning simple primitive policies, which focus distinct regions of the state space. Rather than being gated by single meta-policy, the primitives directly compete with one another to determine which should be active any given time, based on the degree to which their state encoders "recognize" the current state input. While, technically, the competition between primitives implicitly realizes global selection mechanism, consider our model decentralized in the sense that individual primitives can function on their own, and can be combined in new ways, without relying on an explicit high-level controller.
We frame the problem as one of information transfer between the current state and a dynamically selected primitive policy. Each policy can, by itself, decide request information about the current state, and the amount of information requested is used to determine which primitive acts in the current state. Since the amount of state information that a single primitive can access is limited, each primitive is encouraged to use its resources wisely. Constraining the amount of accessible information this way naturally leads to a decentralized competition and decision mechanism where individual primitives specialize in smaller regions of the state space. We formalize this information-drie driven objective based on the variational information bottleneck. The resulting set of competing primitives achieves both meaningful factorization of the policy and effective decision mechanism for which primitives to use. Importantly, not relying on a centralized meta-policy enables the individual primitive mechanisms can be recombined in plug-and-play fashion, and the primitives can be transferred seamlessly to new environments.
Contributions:
In summary, the contributions of our work are follows: (1) We propose method
for learning and operating set of functional primitives in decentralized way. without requiring an explicit high-level meta-controller to select the active primitives (see Fig. for illustration). (2) We introduce an information-thherree the specialization of individual primitives to distinct regions the state space, and b) enables competition mechanism, which is used to select active primitives in a decentralized manner. (3) We demonstrate the superior transfer learning performance of our model, which is due to the flexibility of the proposed framework regarding the dynamic addition, removal, and recombination of primitives. Decentralized primitives can be successfully transferred to larger or previously unseen environments, and outperform models with explicit meta-controller for primitive selection.
objective, the effects of which are twofold: a) it leads to
2

---

Published as a conference paper at ICLR 2020
2 PRELIMINARIES
where the state space S and the action space A may be discrete continuous. The environment emits bounded reward r : S x A + [mmin, Tmoa]] on each transition and 7 E [0. 1) is the discount factor. π л(.ss) denotes policy over the actions given the current state s. R(=)
We consider a Markov decision process (MDP) defined by the tuple (S, P,r, P.r,))),,
Ex Ex ((((ss denotes the expected total return when an agent follows the policy т. The standard objective in reinforcement learning is to maximize the expected total return R(T). We use the concept of the information bottleneck (Tishby et al., 2000) to learn compressed representations. The information bottleneck objective formalized as minimizing the mutual information of bottlenec representation layer with the input while maximizing its mutual information with the corresponding output. This type of input compression has been shown to improve generalization (Achille & Soatto 2016 Alemi et al. 2016).
3
INFORMATION-
LEARNING OF DISTINCT PRIMITIVES
Our goal is to learn policy, composed of multiple primitive sub-policies, to maximize the expected reward over -step interactions for distribution of tasks. Simple primitives which focus on solving part of the given task (and not the complete task) should generalize more effectively, as they can be applied to similar aspects of different tasks (subtasks) even if the overall objective of the tasks drastically different. Learning primitives in this way also be viewed as learning a factorized representation of a policy, which is composed of several independent policies.
Our proposed approach consists of three mechanisms: 1) mechanism for restricting particular primitive to a subset of the state space; 2) a competition mechanism between primitives to select the most effective primitive for given state; 3) regularization mechanism to improve the generalization performance of the policy as a whole. We consider experiments with both fixed and variable sets of primitives and show that method allows for primitives to be added removed during training, recombined in new ways. Each primitive is represented by a differentiable, parameterized function approximator, such as neural network.
3.1 PRIMITIVES WITH AN INFORMATION BOTTLENECK
To encourage each primitive encode information from a particular part of state space, we limit the amount of information each primitive can access from the state. In particular, each primitive has an information bottleneck with respect to the input state, preventing it from using all the information from the state.
We define the overall policy mixture of primitives,
Taa s) = E. cx s),
where =*(a s) denotes the 4thh primitive and C₁ des for k' ~ p(k' s). We denote the probability of selecting the th primitive as as(s) := [ s).
Rather than learning an explicit model for p(k: | 5), however, we impose an information-bace mechanism for selecting primitives, wherein we limit the amount of information each primitive can contain and select the ones that request the most information about the state. To implement an information bottleneck, we design each of the K primitives to be composed of an encoder Pene and decoder Paee (a zx), together forming the primitive policy,
(a s) f- Penc (2k s) Paee (a 2xx dz
The encoder output zạ is meant to represent the information about the current state s that an individual primitive l: believes is important access in order perform well. The decoder takes this encoded information and produces a distribution over the actions a. Following the variational information bottleneck objective (Alemi et al. 2016)) we penalize the KL divergence of Peno(zkl5 (s)s) and prior p(z).
L DKL (pone (Zx ss)[p(z)))
In practice, we estimate the marginalization over using single sample throughout our experiments.
3

---

Published as a conference paper at ICLR 2020
In other words, primitive pays an "information cost" proportional to L3 for accessing the information about the current state.
learn the In the experiments below, fix the prior to be unit Gaussian. In the general case, prior as well and include its parameters in 0. The information bottleneck encourages each primitive to limit its knowledge about the current state, but it will not prevent multiple primitives from specializing to similar parts of the state space. To mitigate this redundancy, and to make individual primitives focus on different regions of the state space, introduce an information-base competition mechanism to encourage diversity among the primitives.
3.2 COMPETING INFORMATION IPPORMATION-CONSTRIINNE PRIMITIVES
We can use the information measure from equation | to define selection mechanism for the primitives without having to learn centralized meta-policy. The intuition is that the information content of an individual primitive encodes its effectiveness in given state such that the primitive with the highest value Lg should be activated that particular state.
In particular, we set ag z-1 exp(3Ca) to obtain distribution over & as function of the information content, activating the primitives with the highest information content. Here, X = Σn cxp( is normalization constant. This mechanism enables competition between primitives, leading them to focus on parts of the state space that they "understand" well and letting others act in other parts.
Trading reward and information. To perform proper credit assignment, the environment reward is distributed to primitives according to their participation in the global decision, i.e. the reward rk given to the Ath primitive is weighted by its selection coefficient, such that rx ogr, with Σn Hence, primitive can potentially get higher reward when deciding to act, but it also pays higher price for accessing more information about the current state. The information bottleneck and the competition mechanism, when combined with the overall reward maximization objective, will lead to specialization of individual primitives to distinct regions in the state space. That is, each primitive should specialize in part of the state space that it can reliably associate rewards with. Since the entire ensemble still needs to understand all the state space for the given task, different primitives need to encode and focus on different parts of the state space.
3.3 REGULARIZING PRIMITIVE SELECTION
The objective described above will optimize the expected return while minimizing the information content of individual primitives. This not sufficient, however, as it might lead to highly unbalanced outcomes: some primitives might be active initially and learn to become more active, completely disabling other primitives.
Thus, addition to minimizing each primitive's absolute information content, we need normalize their activity with respect to each other. To do so, we penalize their information content in proportion their activation by adding a regularization term of the form
freg Ex Ci
Note that this can rewritten (see Appendix A as Lreg = -H( + LSE( Lx), where ((a) is the entropy of a, and LSE is the LogSunExp function, LSE(x) log(E, e²)) Thus, minimizing Freg increases the entropy of o, leading to diverse set of primitive selections, in turn, ensuring that different combinations of the primitives are used. Similarly, LSE approximates the maximum of its arguments, LSE(x) R maxj Zj. and, therefore, penalizes the dominating Ca terms, thus equalizing their magnitudes.
3.4 OBJECTIVE AND ALGORITHM SUMMARY
Our overall objective function consists of 3 terms,
1. The expected return from the standard RL objective, R(T) which is distributed to the primitives according their participation,
4

---

Published as a conference paper at ICLR 2020
2. The individual bottleneck terms leading the individual primitives to focus on specific parts of the state space, C3 for k = 1....,,, K,
3. The regularization term applied to the combined model, Leg.
The overall objective for the hth primitive thus takes the form:
Jx (0) Ex [Tw] - Bra Cx - Breg
(3)
where Em, denotes an expectation over the state trajectories generated by the agent's policy, Tk- = or the reward given the kth primitive, and Sidd. Prog are the parameters controlling the impact of the respective terms.
Implementation: In our experiments, the encoders Pene (zn s) and decoders pase (a zx) (see. Fig. I are represented by neural networks, the parameters of which we denote by 8. Actions are sampled through each primitive every step. While our approach is compatible with any RL method, we maximize J (0) computed on-policy from the sampled trajectories using score function estimator (Williams 1992, Sutton et al. (999a) specifically A2C (Mnih et al. 2016) (unless otherwise noted). Every experimenaa result reported has been averaged over 5 random seeds. Our model introduces 2 8ind. Sreg.
extra
4 RELATED WORK
There a wide variety of hierarchical reinforcement learning approaches (Sutton al. 1998 Dayan & Hinton 1993 Dietterich 2000). One of the most widely applied HRL framework is the Options framework Sutton et al., 1999b)) An option can be thought of as an action that extends over multiple timesteps, thus providing the notion of temporal abstraction or subroutines in an MDP. Each option has its own policy (which followed the option is selected) and the termination condition (to stop the execution of that option). Many strategies are proposed for discovering options using task-specific hierarchies, such as pre-defined sub-goals (Heess et al. 2017)) hand-designed features snvensity-promotine
(Florensa et al., 2017), These approaches do not generalize well to new tasks. Bacon et al. (2017) proposed an approach to learn options in an end-to-end manner by parameterizing the intra-option policy well as the
priors (Daniel et al., 2012 Eysenbach al.) 2018).
policy and termination condition for all the options. Figen-options (Machado et al., (017) use the eigenvalues of the Laplacian (for the transition graph induced by the MDP) derive intrinsic reward for discovering options as well as learning an intra-option policy.
In this work, we consider a sparse reward setup with high dimensional action spaces. In such a scenario, performing unsupervised pretraining or using auxiliary rewards leads to much better performance (Frans al. 2017; Florensa al., 2017 Heess al., 2017). Auxiliary tasks such as motion imitation have been applied to learn motor primitives that are capable of performing variety of sophisticated skills (Liu & Hodgins 2017 Peng et al. 2017 Merel et al. 2019bb a). Our work also related to the Neural Module Network family of architectures (Andreas et al., 2017 Johnson al., 2017 Roosenbamm et al., (219)) where the idea is to learn modules that can perform some useful computation like solving a subtask and controller that can learn to combine these modules for solving novel tasks. More recently, Wu et al. (019) proposed a framework for using diverse suboptimal world models learn primitive policies. The key difference between our approach and all the works mentioned above is that we learn functional primitives without requiring any explicit high-level meta-controller or master policy.
5 EXPERIMENTAL RESULTS
In this section, we briefly outline the tasks that we used to evaluate our proposed method and direct the reader to the appendix for the complete details of each task along with the hyperparameters used for the model. We designed experiments to address the following questions: a) Learning primitives
- Can an ensemble of primitives be learned over a distribution tasks? b) Transfer Learning using primitives Can the learned primitives be transferred to unseen/unsolvable sparse environments? e) Comparison to centralized methods How does our method compare to approaches where the primitives are trained using an explicit meta-controller, in centralized way?
5

---

Published as a conference paper at ICLR 2020
Run
Fronthip
Figure 2: Snapshots of motions learned by the policy. Top: Reference motion clip. Middle: Simulated character imitating the reference motion. Bottom: Probability of selecting each primitive.
Baselines. We compare our proposed method to the following baselines: a) Option Critic (Bacon
et al. 2017) We extended the author's implementatiorf] of the Option Critic architecture and experimented with multiple variations in terms of hyperparameters and state/goal encoding. None of these yielded reasonable performance in partially observed tasks, so omit it from the results. b) MLSH (Meta-Learning Shared Hierarchy) (Frans et al. 2017) This method uses meta-learning to learn sub-policies that are shared across tasks along with learning task-specific high-level master. It also requires phase-wise training schedule between the master and the sub-policies to stabilize training. We use the MLSH implementation provided by the authors] e) Transfer A2C: In this method, we first learn a single policy on the one task and then transfer the policy to another task, followed by fine-tuning in the second task.
5.1 LEARNING ENSEMBLES OF FUNCTIONAL PRIMITIVES
We evaluate our approach on a number of RL environments demonstrate that we can indeed learn sets of primitive policies focusing on different aspects of a task and collectively solving it.
0.5
2.5
04
An
www m
04
0.0
Mumber
Rusmber
200
Figure 3: Convergence of four primitives Four Room Maze: Left: We trained four primitives
on the Four Room Maze task, where the goal was sampled from one of the two fixed goals. We see that the proposed algorithm is able leam four primitives. Right: We transfer the learned primitives to the scenario where the goal is sampled from one the four possible goals. The checkpointed model is ran on 100 different episodes (after fixed number of steps/updates) and the normalized frequency of activation of the different primitives is plotted.
We consider the Four-rooms gridworld environment (Sutton et al., 1999c) where the agent has to navigate its way through grid of four interconnected rooms to reach goal position within the grid. We consider the scenario where the starting position of the agent fixed, but the goal sampled from discrete set. Fig. 3 shows that the proposed algorithm can learn four primitives. Refer to Appendix F for more details.
Four Room Maze:
Motion Imitation. To evaluate the proposed method in terms of scalability, we present a series of tasks from the motion imitation domain, showing that can use set of distinct primitives for imitation learning. In these tasks, we train simulated 2D biped character to perform a variety of highly dynamic skills by imitating motion capture clips recorded from human actors. Each mocap
https///ittubb.om/jaaaaaat critic hutp://gttuuh.co//ppnaiii

---

Published as a conference paper at ICLR 2020
clip is represented by a target state trajectory r' ( si, ..., s where 3t denotes the target state
at timestep t. The input to the policy is augmented with a goal 9: = 1: f which specifies the the target states for the next two timesteps. Both the state s, and goal 9r are then processed by the encoder (emc(z) |&t, 9t). The repertoire of skills consists of 8 clips depicting different types of walks, runs, jumps, and flips. The motion imitation approach closely follows |eng et al.) (2018). To analyze the specialization of the various primitives, we computed 2D embeddings of states and goals which each primitive is active in, and the actions proposed by the primitives. Fig. 3 illustrraees the embeddings computed with t-SNE (van der Maaten & Hinton, 2008). The embeddings show distinct clusters for the primitives, suggesting a degree of specialization of each primitive to certain states, goals, and actions.
5.2 MULKI-TASK TRAINING
We evaluate our model in a partially- observable 2D multi-task environment called Minigrid, similar to the one intro- duced in Sntevalier-Boisved et al.] 2018). The environment is a two-dimensional grid with a single agent, impassable walls, and many objects scattered in the environment. The agent is provided with a natural lan- guage string that specifies the task that the agent needs to complete. The setup is partially observable, and the agent only gets the small, egocentric view of the grid (along with the natural language task de- scription). We consider three tasks here: the Pickup task (A), where the agent is re- quired to pick up an object specified by the goal string, the Unlock task (B) where the agent needs to unlock the door (there could be multiple keys the environment, and the agent needs to use the key which matches the color of the door) and the UnlockPickup task (C), where the agent first needs to unlock a door that leads to another room. In this room, the agent needs find and pick up the object specified by the goal string. Additional implementation details of the environment are provided in appendix D Details on the agent model can be found in appendix D.3
S
Figure 4: Embeddings visualizing the states (S) and goals (G) which each primitive is active in, and the actions (A) proposed by the primitives for the motion imitation tasks. A total of four primitives are trained. The primitives produce distinct clusters.
We train agents with varying numbers of primitives on various tasks concurrently, as well as in transfer settings. The different experiments are summarized in Figs. [ 7 An advantage of the multi-task setting is that it allows for quantitative interpretability as to when and which primitives are being used. The results indicate that a system composed of multiple primitives generalizes more easily to new task, as compared to a single policy. We further demonstrate that several primitives can be combined dynamically and that the individual primitives respond to stimuli from new environments when trained related environments.
5.3 Do LEARNED PRIMITIVES HELP IN TRANSFER LEARNING?
We evaluate our approach in the settings where the adaptation to the changes in the task is vital. The argument in favor of modularity that it enables better knowledge transfer between related tasks. Naturally, the transfer is easier when the tasks are closely related, as the model will only need to learn how to compose the already-learned primitives. In general, it is difficult to determine how closely related two tasks are, however, and the inductive bias of modularity could even be harmful if the two tasks are very different. In such cases, we could add new primitives (which would need to be learned) and still obtain a sample-efficient transfer, as some part of the task structure would already have been captured by the pretrained primitives. This approach can be extended towards adding primitives during training, providing a seamless way to combine knowledge about different tasks solve more complex tasks. We investigate here the transfer properties of a primitive trained in one environment and transferred to a different one. Results are shown in Fig.
Continuous control for ant maze We evaluate the transfer performance of pretrained primitives on the cross maze environment (Haarnoja et al.) 2018). Here, a quadrupedal ant robot must walk to the different goals along the different paths (see Appendix G for details). The goal is randomly
7

---

Published as a conference paper at ICLR 2020
B
oo
co
00
keshhe komheeaaaay www.aaaarararrtttooooo ww
0.0
500
32
044
3.6
6.8
1.0
0.0
6.2
0.4
0.6
1.0
1.2
60
0.4
0.6
0.
A
B
A
B
cO
oo
o o
0o
O Croy
0o
retrain
0.
copy
combine
zero-shot generalisation
1:2
1:2
TraefariCC Our asprash
Trandl AZC Oction Crtic (MLS)
0.25
3,02
Mumber
humber fra
Figure 5: Multitask training. Each panel corresponds to different training setup. where different tasks are denoted A, B, C, .... and rectangle with circles corresponds n primitives trained on the respective tasks. Top row: activation primitives for agents trained on single tasks. Bottom row: Retrain: Two primitives are trained task A and transferred to task B. The results (success rates) indicate that the multi-primitive model is substantially more sample efficient than the baseline (transfer A2C). Copy and Combine: More primitives are added to the model over time in a plug-and-pla fashion (two primitives are trained on task A; the model is extended with a copy of the two primitives; the resulting four-pric nour-prmmitive model is trained on task B.) This is more sample efficient than other strong baselines, such as (Frans et al., 2017; Bacon al. 2017). Zero-Shot Generalization: A set of primitives trained on task C, and zero-shot generalization to task A and B is evaluated. The primitives learn a form of spatial decomposition which allows them to be active in both target tasks, A and B. The checkpointed model is ran on 100 different episodes, and the normalized frequency of activation of the different primitives is plotted.
agent composed of
chosen from set of available goals at the start of each episode. We pretrain policy (see model details in Appendix G.1) with motion reward in an environment which does not have any walls (similar to Haarnoja et al. (018)), and then transfer the policy to the second task where the ant has to navigate to a random goal chosen from one of the 3 (or 10) available goal options. For our model, we make four copies the pretrained policies and then finetune the model using the pretrained policies primitives. We compare to both MLSH (Frans et al. (017) and option-critic (Bacon et al. (017). All these baselines have been pretrained in the same manner. As evident from Fig. [ our method outperforms the other approaches. The fact that the initial policies successfully adapt to the transfer environment underlines the flexibility of our approach.
The purpose of this experiment to show that the model consisting of multiple primitives is somewhat able to decompose the task C into its subtasks, A and B. The better this decomposition is the better should the model transfer the individual subtasks. In order to test this, we trained set of 4 primitives on task C, and then evaluate them (without finetuning) on tasks and B. We note that the ensemble is able solve the transfer tasks, and B, successfully 72% of the time, while a monolithic policy's success rate is 38%. This further shows that the primitives learn meaningful decompositions.
Zero Shot Generalization:
Continual Learning: 4 Rooms Seneario. We consider continual learning scenario where train two primitives for two-goal positions ie the goal position is selected randomly from one of the two positions the start of the episode. The primitives are then transfer (and finetuned) on four-goal
8

---

Published as a conference paper at ICLR 2020
Relative frequency of activation
1.25
1e6
1.00
idx
of frames
Transfer Baseline
2.0
Our Approach
0.75
0.50
0.25
0.00
0.0
0.5
1.0
2.0
Number of frames
1e6
2 goals
4 goals
8 goals
Figure 6: Continual Learning Scenario: The plot on the left shows that the primitives remain activated. The solid green line shows the boundary between the tasks. The plot on the right shows the number of samples required by our model and the transfer baseline model across different tasks. We observe that the proposed model takes fewer steps than the baseline (an A2C policy trained in similar way), and the gap terms of the number of samples keeps increasing as tasks become harder. The checkpointed model is 100 different episodes (after fixed number of steps/updates) and the normalized frequency of activation of the different primitives is plotted.
Method
goals
10 goals
D
oooo0
OO0000o0
0.6
Flat Policy (PPO)
11 + %
4 4 %
Option critic
18 + 10 %
7 3 %
MLSH
32 0
5 %
B
Explicit high level policy
21
11
0.0
Proposed method
40 3%
Number frames
1c0
Figure 7: Left:: Multitask setup, where we show that we are able to train eight primitives when training mixture of four tasks in the Minigrid environment. Here, the x-axis denotes the number of frames (timesteps). Right: Success rates of the different methods on the Ant Maze tasks. Success rate measured the number of times the ant is able reach the goal (based on 500 sampled trajectories).
positions then transfer (and finetune) eight-goal positions. The results shown in fig. The proposed method achieves better sample efficiency as compared to training a single monolithic policy.
SUMMARY AND DISCUSSION
We present framework for learning an ensemble of primitive policies that can collectively solve centralized, learned tasks without learning explicit master policy. Rather than relying information-theorrtie meta-controller, the selection of active primitives is implemented through an mechanism. The learned primitives can be flexibly recombined solve more complex tasks. Our experiments show that, on partially observed "Minigrid" task and continuous control "Ant Maze" walking task. our method can enable better transfer than flat policies and hierarchical RL baselines. including the Meta- Meta-lcarning Shared Hicrarchies model and the Option-Criie Crinii framework. On Minigrid, we show how primitives trained with our method can transfer much more successfully new tasks. On the Ant Maze, we show that primitives initialized from a pretrained walking control can learn to walk to different goals stochastic, multi-modal environment with nearly twice the success rate of more conventional hierarchical RL approach, which uses the same pretraining but centralized high-level policy. The proposed framework could be very attractive for continual learning settings, where one could add more primitive policies over time. Thereby, the already learned primitives would keep their focus on particular aspects the task, and newly added ones could specialize on novel aspects.

---

Published as a conference paper at ICLR 2020
7 ACKNOWLEDGEMENTS
The authors acknowledge the important role played by their colleagues at Mila throughout the duration of this work. AG would like thank Greg Wayne, Mike Mozer, Matthew Botvinick, Bernhard Schölkopf for very useful discussions. The authors would also like to thank Nasim Rahaman, Samarth Sinha, Nithin Vasisth, Hugo Larochelle, Jordan Hoffman, Ankesh Anand, Michael Chang for feedback on the draft. The authors are grateful to NSERC, CIFAR, Google, Samsung, Nuance, IBM, Canada Research Chairs. Canada Graduate Scholarship Program, Nvidia for funding, and Compute Canada for computing resources. We are very grateful to Google for giving Google Cloud credits used in this project.
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: learning optimal representations through noise. CoRR, abs/1611.01355 2016. (RL/htt:://axx: org. baaegg/ab///eee abs//611.. 01353
Alexander A. Alemi, lan Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. CoRR, abs/1612.004400, 2016. URL htttp:://axxiv.orr///eeeee 11..
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning-Voluue Volume 70, pp. 165-175. JMLR. org. 2017.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI, pp. 1726-1734, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Maxime Bhevelier-Boiseert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: /ttp:::/gitttuu cwwm www.//mww.mc..a.mmmooooo.......e http://w..mmmmggmmmmmmaeaaeeeeee
gym gym-gmmiimiiiii 2018.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. 1078, 2014.
Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, pp. 273-281, 2012.
Peter Dayan and Geoffrey Hinton, Feudal reinforcement leammnn In Advances in neural information processing systems, pp. 271-278, 1993.
Thomas Dietterich. Hierarchical reinforcement learning with the maxq value function decomposi- tion. Journal of Artificial Intelligence Research, 13:227-303, 2000.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint wwXXiv:802.000 2018.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce- ment learning. arXiv preprint arXiv:1704.03012 2017.
K. Frans, J. Ho, X. Chen, P. Abbeel, and J. Schulman. Meta Learning Shared Hierarchies. arXiv e-prints, October 2017.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. arXiv preprint 1rXiv:1710.09767. 2017.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hicrarchical reinforcement learning. arXiv preprint arXiv:1804.02808 2018.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. Emergence locomotion behaviours in rich environments. arXiv preprint I707 02780. 2017.
10

---

Published as a conference paper at ICLR 2020
Justin Johnson, Bharath Hariharan, Laurens der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitmick, and Ross Girshick. Inferring and executing programs for visual reason- ing. In Proceedings of the IEEE International Conference on Computer Vision, pp- 2999. 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1441269980 2014.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675-3683, 2016.
Libin Liu and Jessica Hodgins. Learning to schedule control fragments for physics- characters using deep q-learning. ACM Transactions on Graphics, 36(3), 2017.
Marlos C Machado, Mare Bellemare, and Michael Bowling. laplacian framework for option discovery in reinforcement learning. arXiv preprint arXiv: 00956, 2017
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas
Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. In International Confer- ence on Learning Representations, 2019a. URL hhttps://oppereeeeeee
net/ forum?ide
BJfYvo09Y7
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. In International Conference on Learning Representations, 2019b. URL https; 7/openreview. net www.moorumirooooooe ptt.://orourRdddB3116........
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla, and Bernhard Schölkopf. Learning independent causal mechanisms. arXiv preprint arXiv:
2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic diffcrentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Xue Bin Peng, Glen Berseth, Kangkang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Trans. Graph., 36(4): htt::tt:://an
41:1-41:13, July 2017. ISSN 0730-0301, doi: 10.1145/3072959.8073622
acm. acm.org/10 145 3072959 0075593..73333 3073602
Xue Bin Peng, Picter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37 URLhtttp
(4):14311-143:1, July 2018. ISSN 0730-0301. doi: 10. 10.11145/11975173221111
//doi acm. or.. 10. 1145 114./111113779 31155/319957..222 3201311
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer. and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprint mexivv:9904 /2774, 2019.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning, arXiv preprint arXiv: /703.0 161, 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 1xXxv:1506. 02438, 2015.
11

---

Published as a conference paper at ICLR 2020
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press, 1998.
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS 99, pp. 1057-1063, Cambridge, MA, USA, a999a. MIT Press. URLbttp: acm. wwt.:aaa.....aaaaa..o..oo/ htt.:/ooo..mmmmm33000000031 3009806
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2)1811-21, 1999b.
Richard Sutton, Doina Precup. and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, (12(1-2):181-2111 1999c.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop, coursera: Neural networks for machine learning. University of Toronto, Technical Report, 2012.
Naftali Tishby, Fernando C. N. Pereira, and William Bialek. The information bottleneck method. CoRR, physics/0004057, 2000. /ttp://////rxxx /ttp://ass/pphsssc///000000
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. IEEE, 2012.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Ma- chine Learning Research, 9:2579-2605, 2008. URL http:://tww jmlr. r org/ arg/papers/v9
vandermaatenn8a.herr html
Ronald J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Re-
inforcement Learning. Machine Learning, 8(3-4):229-256, 1992. ISSN 0885-6125, doi: 10.1007/BB00992666
/
Bohan Wu, Jayesh K Gupta, and Mykel Kochenderfer. Model primitive hierarchical lifelong reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 34-2., International Foundation for Autonomous Agents and Multiagent Systems, 2019.
Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using cronecker-factoree approximation. Advances in neural information processing systems, pp. 5279-5288, 2017.
12

---

Published as a conference paper at ICLR 2020
A INTERPRETATION OF THE REGULARIZATION TERM
The regularization term given by
Lreg Eanls
k
where
0 = c+ Σ cl,
and thus
log ag = Lx - log E efs
La log on +
where LS Lx) = log E, e independent of Plugging this in, and using Σ ak = we get
Lreg Σ ag log ax LSE(CL...... = =x)==HH()) +LS
Information-theorrii interpretation Notably, Leg also represents an upper bound to the KL- divergence of mixture of the currently active primitives and prior,
Lrgg 2 DEL Σ
CrsPenn (2sp)m(224/))))((0111))
and thus can be regarded as term limiting the information content of the mixture of all active primitives. This arises from the convexity properties of the KL divergence, which directly lead to
DKL
(l S Σ ae DKL
B ADDITIONAL RESULTS
B.1 2D BANDITS ENVIRONMENT
In order to test if our approach can learn distinct primitives, we used the 2D moving bandits tasks (introduced Frans et al. 2017)). In this task, agent placed in 2D world and is shown the position of two randomly placed points. One of these points is the goal point but the agent does not know which. We use the sparse reward setup where the agent receives the reward certain distance of the goal point and 0 at all other times. Each episode lasts for 50 steps and to get the reward, the learning agent must reach near the goal point in those steps. The agent's action space consists of actions moving in one of the four cardinal directions (top, down, left, right) and staying still.
if is within
B.1.1 RESULTS FOR 2D BANDITS
We want to answer the following questions:
Can our proposed approach learn primitives which remain active throughout training?
2. Can our proposed approach learn primitives which can solve the task?
We train two primitives on the 2D Bandits tasks and evaluate the relative frequency of activation of the primitives throughout the training. It is important that both the primitives remain active. If only I primitive is acting most of the time, its effect would be the same as training a flat policy. We evaluate the effectiveness of our model by comparing the success rate with flat A2C baseline. Fig. g shows that not only do both the primitives remain active throughout training, our approach also outperforms the baseline approach.
13

---

Published as a conference paper at ICLR 2020
Percentage epioodes completed
Fiat Pollyy
Relative frequency activation
1.0
lak
80
Decentralized Policy
0.8
60
0.8
40
0.4
Myhul WW M /
20
0.2
0.0
2
Numbar frames
Numbar framos
1e6
Figure 8: Performance on the 2D bandits task. Left: The comparison of our model (blue curve- decentralized policy) with the baseline (red curve flat policy) in terms of success rate shows the effectiveness of our proposed approach. Right: Relative frequency of activation of the primitives (normalized to sum up to 1). Both primitives are utilized throughout the training.
B.2 FOUR-ROOMS ENVIRONMENT
We consider the Four-rooms gridworld environment (Sutton al. 1999c) where the agent has to navigate its way through a grid of four interconnected rooms to reach goal position within the grid. The agent can perform one of the following four actions: move up, move down, move left, move right. The environment is stochastic and with 1/3 probability. the agent's 's chosen action is ignored and a new action (randomly selected from the remaining actions) is executed ie the agent's 's selected action is executed with a probability of only 2/3 and the agent takes any of the 3 remaining actions with probability of 1/9 each.
B.2.1 TASK DISTRIBUTION FOR THE FOUR-ROOM ENVIRONMENT
In the Four-room environment, the agent has navigate to goal position which is randomly selected from a set of goal positions. We can use the size of this set of goal positions to define a curriculum of task distributions. Since the environment does provide any information about the goal state, the larger the goal set, harder is the task as the now goal could be any element from a larger set. The choice of the set of goal states and the choice of curriculum does not affect the environment dynamics. Specifically, we consider three tasks Fourroom-vO, Fourroom-vl vl and Fourroom-v2 with the set of 2, and goal positions respectively. The set of goal positions for each task is fixed but not known to the learning agent. We expect, and empirically verify, that the Fourroom-vd environment requires the least number of samples to be learned, followed by the Fourroom-vl and the Fourroom-v2 environment (Fig. 6 in the paper).
B.2.2 RESULTS FOR ROOMR ENVIRONMENT
We want to answer the following questions:
1. Can our proposed approach learn primitives that remain active when training the agent over sequence of tasks?
2. Can our proposed approach be used to improve the sample efficiency of the agent over a sequence of tasks?
To answer these questions, we consider two setups. In the baseline setup, we train a flat A2C policy on Fourrooms-vd till achieves 100 % success rate during evaluation. Then we transfer this policy to Fourrooms-v/ and continue to train till it achieves 100 % success rate during the evaluation on Fourrooms-v/. We transfer the policy one more time to Fourrooms-v2 and continue train the policy until it reaches 60% success rate. In the last (ask(Fourrooms-22) threshold as the models do not achieve 100% for training even after training for 10M frames. We use 60% the baseline models generally converge around this value.
do not use 100% the
In the second setup, we repeat this exercise of training on one task and transferring to the next task with our proposed model. Note that even though proposed model converges higher value than 60% in the last Fouk(Fooooooss2222 we compare the number of samples required to reach 60% success rate provide fair comparison with the baseline.
14

---

Published as a conference paper at ICLR 2020
C IMPLEMENTATION DETAILS
In this section, we describe the implementation details which are common for all the models. Other lask-specific details are covered in the respective task sections.
1. All the models (proposed well the baselines) implemented in Pytorch 1.1 unless stated otherwise. (Paszke et al. (017).
2. For Meta-Learning Shared Hierarchies (Frans et al. 2017) and Option-Critic (Bacon et al. 2017), we adapted the author's implementations]] Jor our environments.
3. During the evaluation, use 10 processes in parallel to run 500 episodes and compute the percentage of times the agent solves the task within the prescribed time limit. This metric is referred the "success rate".
4. The default time limit is 500 steps for all the tasks unless specified otherwise.
5. All the feedforward networks are initialized with the orthogonal initialization where the input tensor is filled with a (semi) orthogonal matrix.
6. For all the embedding layers, the weights are initialized using the unit-Gaussian distribution.
7. The weights and biases for all the GRU model are initialized using the uniform distribution from U--Vk, vk) where k hidden size'
8. During training, perform 64 rollouts in parallel to collect 5-step trajectories.
9. The Bind and Brey parameters are both selected from the set {0.001, 0.005, 0.009) by performing validation.
In section D.4.2] we explain all components of the model architecture along with the implemen- tation details in the context of the MiniGrid Environment. For the subsequent environments, describe only those components and implementation details which are different than their counterpart in the MiniGrid setup and do not deseribe the components which work identically.
D MINIGRID ENVIRONMENT
We use the MiniGrid environment CChevvlier-Boisvertt et al. 2018) which is an open-source, grid- world environment package|] It provides a family of customizable reinforcement learning environ- ments that are compatible with the OpenAl Gym framework (Brockman et al. 2016). Since the environments
be easily extended and modified, it is straightforward control the complexity of the task (eg controlling the size of the grid, the number rooms or the number of objects in the grid, etc). Such flexibility is very useful when experimenting with curriculum learning testing for generalization.
D.1 THE WORLD
In MiniGrid, the world (enviromment for the learning agent) is rectangular grid of size say MæN. Each tile in the grid contains either zero door, key, ball, box and goal. Each object has an associated string (which denote the object type) and an associated discrete color (could be red, green, blue, purple, yellow and grey). By default, walls always grey and goal squares are always green. Certain objects have special effects. For example, key can unlock door of the same color.
object. The possible object types
floor, lava,
D.1.1 REWARD FUNCTION
We consider the sparse reward setup where the agent gets reward (of 1) only if completes the task and 0 at all other time steps. We also apply time limit of 500 steps on all the tasks ie the agent must complete the task 500 steps. task is terminated either when the agent solves the task or when the time limit is reached whichever happens first.
https//ggthuub.oom/oppppettt http:://gghhi..cccmmmmmmmmmmmmgp
critic
15

---

Published as a conference paper at ICLR 2020
D.1.2 ACTION SPACE
The agent can perform one of the following seven actions per timestep: turn left, turn right, move forward, pick up an object, drop the object being carried, loggle, done (optional action).
The agent can the lurn left and directions (north, south, east, west). The move forward action makes the agent move from its current tile onto the tile in the direction it currently facing, provided there is nothing on that tile, or that the tile contains an open door. The toggle actions enable the agent to interact with other objects in the world. For example, the agent can use the toggle action open the door if they are right in front of and have the key of matching color.
right actions rotate around and face one of the 4 possible
D.1.3 OBSERVATION SPACE
The MiniGrid environment provides partial and egocentric observations. For all our experiments, the agent sees the view of square of 4x4 tiles in the direction it is facing. The view includes the tile on which the agent is standing. The observations are provided as tensor of shape 4x4x3. However, note that this tensor does not represent RGB images. The first two channels denote the view size and the third channel encodes three integer values. The first integer value describes the type of the object, the second value describes the color of the object and the third value describes if the doors are open closed. The benefit of using this encoding space-efficient and enables faster training. For human viewing, the fully observable, RGB image view of the enviromments is also provided and use that view as example in the paper.
the RGB encoding is that this encoding is more
Additionally, the environment also provides a natural language description the goal. An example of the goal description is: "Unlock the door and pick up the red ball". The learning agent and the environment use shared vocabulary where different words are assigned numbers and the environment provides number-encoded goal description along with each observation. Since different instructions can be of different lengths, the environment pads the goal description with <unk> tokens to ensure that the sequence length is the same. When encoding the instruction, the agent ignores the padded subbsequence in the instruction.
D.2 TASKS IN MINIGRID ENVIRONMENT
Figure 9: RGB view of the Pickup environment.
Figure 10: RGB view the Unlock environment.
16

---

Published as a conference paper at ICLR 2020
Figure 11: RGB view of the UnlockPickup environment.
We consider the following tasks in the MiniGrid environment:
1. Pickup: In the Pickup task, the agent spawns at arbitrary position in 8 8 grid (Fig. ). It is provided with natural language goal description of the form "go pickup a yellow box". The agent has navigate the object being referred to in the goal description and pick it up.
2. Unlock: In the Unlock task, the agent spawns at an arbitrary position in a two-room grid environment. Each room is 8 square (Fig. 10) ). It provided with natural language goal description the form "open the door". The agent has to find the key that corresponds the color of the door, navigate to that key and use that key to open the door.
3. UnlockPickup:
This task is basically union of the Unlock and the Pickup tasks. The agent spawns at an arbitrary position in a two-room grid environment. Each room is 8 x square (Fig. [T]. It is provided with natural language goal description of the form "open the door and pick the yellow box". The agent has to find the key that corresponds to the color of the door, navigate that key, use that key open the door, enter the other room and pick up the object mentioned in the goal description.
D.3 MODEL ARCHITECTURE
D.3.1 TRAINING SETUP
Consider agent training any task in the MiniGrid suite of environments. At the beginning of an episode, the learning agent spawns at a random position. each step, the environment provides observations in two modalities a 4 x 4 x 3 tensor (an egocentric view of the state of the environment) and a variable length goal description g. We describe the design of the learning agent in terms of encoder-decoder architecture.
D.3.2 ENCODER ARCHITECTURE
The agent's encoder network consists of two models a CNN+GRU based observation encoder and GRU (Cho al. 2014) based goal encoder
Observation Encoder:
It is a three layer CNN with the output channel sizes set 16, 16 and 32 respectively (with ReLU layers in between) and kernel size set to x 2 for all the layers. The output of the CÑN is flattened and fed to GRU model (referred to the observation-rmi with 228-dimensional hidden state. The output from the observation-rmn represents the encoding of the observation.
Goal Encoder:
comprises of an embedding layer followed by unidirectional GRU model. The dimension of the embedding layer and the hidden and the output layer of the GRU model all set to 128.
The concatenated output of the observation encoder and the goal encoder represents the output of the encoder.
17

---

Published as a conference paper at ICLR 2020
D.3.3 DECODER
The decoder network comprises the action network and the critic network both of which are implemented feedforward networks. We describe the design of these networks.
D.3.4 VALUE NETWORK
1. Two-layer feedforward network with the tanh non-linearity.
2. Input:
Concatenation of and the current hidden state of the observation-run.
3. Size of the input to the first layer and the second layer of the policy network are 320 and 64 respectively.
4. Produces a scalar output.
D.4 COMPONENTS SPECIFIC TO THE PROPOSED MODEL
The components that we described so far are used by both the baselines as well as our proposed model. We now describe the components that are specific our proposed model. Our proposed model consists of an ensemble of primitives and the components we describe apply to each of those primitives.
D.4.1 INFORMATION BOTTLENECK
Given that we want to control and regularize the amount of information that the encoder encodes, we compute the KL divergence between the output of the action-feature encoder network and a diagonal unit Gaussian distribution. More is the KL divergence, more is the information that is being encoded with respect to the Gaussian prior and vice-versa. Thus we regularize the primitives minimize the KL divergence.
D.4.2 HYPERPARAMETERS
Table listts the different hyperparameters for the MiniGrid tasks.
Parameter
Value
Learning Algorithm
A2C (Wu et al.
2017
Opitimizer
RMSProp Tieleman & Hinton
2012
learning rate
10
batch size
discount
0.99
lambda (for GAE (Schulman et al. 2015) entropy coefficient
0.95
10-2
loss coefficient
0.5
Maximum gradient norm
0.5
Table 1: Hyperparameters
E 2D BANDITS ENVIRONMENT
E. OBSERVATION SPACE
The 2D bandits task provides 6-dimensional flat observation. The first two dimensions correspond to the (x, y) coordinates of the current position of the agent and the remaining four dimensions correspond to the (x, coordinates of the two randomly chosen points.
18

---

Published as a conference paper at ICLR 2020
E.1 MODEL ARCHITECTURE
E.1.1 TRAINING SETUP
Consider an agent training on the 2D bandits tasks. The learning agent spawns at fixed position and randomly assigned two points. At each step, the environmental observation provides the current position of the agent well the position of the two points. We describe the design of the learning agent terms of an encoder-decoder architecture.
1.1.2 ENCODER ARCHITECTURE
The agent's encoder network consists of GRU-based recurrent model (referred the observation- nun) with a hidden state size of 128. The 6-dimensional observation from the environment is the imput the GRU model. The output from the observation-rii represents the encoding of the observation.
E.2 HYPERPARAMETERS
The implementation details for the 2D Bandits environment are the same as that for MiniGrid environment and are described detail section D.4.2 the table below, we list the values the task-specific hyperparameters.
Parameter
Value
Learning Algorithm epochs per update (PPO)
PPO (chhulmnn et al.
2017)
Optimizer
Adam Kingma & Ba
2014)
learning rate
3 10
B1
0.9
B2
0.999
batch size
64
discount
0.99
entropy coefficient
0
loss coefficient
1.0
Maximum gradient norm
0.5
Table 2: Hyperparameters
F FOUR-ROOMS ENVIRONMENT
F.1 THE WORLD
In the Four-rooms noor-roomsooo setup. the world (environment for the learning agent) is square grid of say 11 11. The grid is divided into rooms such that each room is connected with two other rooms via hallways. The layout of the rooms shown in Fig. 12 The agent spawns at random position and has to navigate to a goal position within 500 steps.
F.1.1 REWARD FUNCTION
We consider the sparse reward setup where the agent gets reward (of 1) only if it completes the task (and reaches the goal position) and 0 all other time steps. We also apply time limit of 300 steps on all the tasks ie the agent must complete the task in 300 steps. A task is terminate either when the agent solves the task or when the time limit is reached whichever happens first.
1.1 OBSERVATION SPACE
The environment is 11 11 grid divided into 4 interconnected rooms. As such, the environment has a total 104 states (or cells) that can occupied. These states are mapped to integer identifiers. At any time i, the environment observation is one-hot representation of the identifier corresponding
19

---

Published as a conference paper at ICLR 2020
Figure 12: View of the four-room environment
the state (or the cell) the agent is in right ie the environment returns vectors of zeros with only one entry being and the index this entry gives the current position of the agent. The environment does not return any information about the goal state.
F.2 MODEL ARCHITECTURE FOR ROOM ENVIRONMENT
F.2.1 TRAINING SETUP
Consider agent training any task in the Four-room suite of environments. At the beginning of an episode, the learning agent spawns at a random position and the environment selects a goal position for the agent. At each step, the environment provides one-hot representation of the agent's s current position (without including any information about the goal state). We describe the design of the learning agent terms of encoder-decoder architecture.
F.3 ENCODER ARCHITECTURE
The agent's encoder network consists of a GRU-based recurrent model (referred as the observation- with a hidden state size of 128. The 104-dimensional one-hot input from the environment is fed to the GRU model. The output from the observation rnn represents the encoding of the observation.
The implementation details for the Four-rooms environment are the same that for MiniGrid environment and are described in detail in section D.4.2
G ANT MAZE ENVIRONMENT
We the Mujoco- quadruple ant (Todorov et al. (0112) to evaluate the transfer performance of our approach on the cross maze environment (Haarnoja et al. 2018). The training happens in two phases. In the first phase, we train the ant walk on surface using motion reward and using just 1 primitive. In the second phase, we make 4 copies of this trained policy and train the agent to navigate to a goal position in a maze (Figure 13). The goal position chosen from a set 3 (or 10) goals. The environment is a continuous control environment and the agent can directly manipulate the movement of joints and limbs.
G.O.1 OBSERVATION SPACE
In the first phase (training the ant to walk), the observations from the environment correspond to the state-space representation a real-valued vector that describes the state of the ant in mechanical terms position, velocity, acceleration, angle, etc of the joints and limbs. In the second phase (training the ant to navigate the maze), the observation from the environment also contains the location the goal position along with the mechanical state of the ant.
20

---

Published as a conference paper at ICLR 2020
Figure 13: View the Ant Maze environment with 3 goals
G.1 MODEL ARCHITECTURE FOR ANT MAZE ENVIRONMENT
G.1.1 TRAINING SETUP
We describe the design of the learning agent in terms of an encoder-decoder architecture.
G.1.2 ENCODER ARCHITECTURE
The agent's encoder network consists of a GRU-based recurrent model (referred as the observation- with hidden state size of 128. The real-valued state vector from the environment is fed to the GRU model. The output from the observation-rmn represents the encoding of the observation. Note that in the case of phase vs phase 2, only the size of the input to the observation-rmu changes and the encoder architecture remains the same.
G.1.3 DECODER
The decoder network comprises the action network and the critic network. All these networks are implemented feedforward networks. The design of these networks is very similar to that of the decoder model for the MiniGrid environment as described in section D.3.3 with just one difference. In this case, the action space continuous the action-feature decoder network produces the mean and bog-standard-deviatioo for a diagonal Gaussian policy. This is used to sample real-valued action execute the environment.
21


Reviews:
[{"review_id": "ryxcJ3XpYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper draws upon the idea of information bottleneck to do task decomposition so as to learn policy primitives similar to hierarchical reinforcement learning that combine together in a competitive manner to specialize in different parts of the task's domain. These policy primitives don't need a higher-level meta-policy to stitch them together. Instead the decision is made in a decentralized manner balancing the cost of information acquisition with maximizing rewards.\nThe paper seems to build on the idea of decomposing the task primitives that specialize in different parts of the state space. A related recent paper [1] which used similar ideas, albeit with a central coordinator to do task composition is missing from related works.\n\nThere are other issues with the paper as well. \n- Decision making is not exactly \"decentralized\"? Computing $Z$ in $\\alpha_k$, still requires the values from other primitives?\n- Sec 5.1 seems tacked on. Motion Imitation is not RL. Experiment details about motion imitation unclear from description. \n- In Figure 5: Zero-shot generalization, it's unclear from the plot whether it actually generalized to solve the tasks A and B. Relative frequency of activation are meaningless without reporting the actual performance on these environments. Where do the 4 indices come from? From the plot above there were only 2 indices but plot below has 4.\n- Continuous control tasks required pretraining.\n\nOverall the experiments seem a little underwhelming. More details on transfer performance without pretraining would be quite helpful. I like the idea of a competitive ensemble figuring out a useful task decomposition and using an information bottleneck like approach makes sense.\n\n[1] https://dl.acm.org/citation.cfm?id=3331671\n\n*Edit*: Updated the score after the reading the revision and the authors' responses.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rke4PHfTFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about a policy design, where the policy is expressed as a mixture of policies called primitives.  Each primitive is made of an encoder and a decoder, mapping state to actions, rather than temporally extended actions (or options in RL).  The primitives compete with each other to be selected in each state and thus do away with the need for a meta-policy to select the primitives.  The selected primitive in each state trades between reward maximization and information content.  \n\nThe paper is well written and is enjoyable to read.  It is helpful for me to have equation (3) in mind before reading about the explanation on the tradeoff between the reward and information, but this is a minor point.  My concern is that by scaling the reward in proportion to L_k redistributes the rewards in a way that is not reflective of the underlying reward structure of the MDP.  If so, the constructed policy \\pi could place a high probability on the suboptimal actions.  How do we know if the action selected according to policy \\pi will indeed lead to high rewards?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgFlSj0Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\tThis paper takes a different approach for tackling the hierarchical RL problem. Their approach is to decompose the policy into a bunch of primitives. Each primitive acts according to its own interpretation of the state. All the primitives are competing with each other on a given state to take an action. It turns out that these primitive policies can be transferred to other tasks as they represent subtasks of a bigger task. The paper performs extensive experiments to show that this scheme improves over both flat and hierarchical policies in terms of generalization.\n\t\n\tThis paper is well-written. I enjoyed reading it. Almost all my questions and doubts are explained when I read through the paper. The framework about using primitive policy to solve a big task is novel and original. The experiments are nice and convincing.\n\t\n\tMinor comments:\n\t• I am understanding the methods as a decomposing the policy into components. Different components are combined together using a probability distribution. To balance the competition and overlap of different primitives, different regularization objective are used.  Did you think about other simple methods, e.g., decompose the policy using linear combination work? It may worthwhile to compare your method with this baseline?\n\t• About the prior p(z), how to choose it in a meaningful way? I do not see why a unit gaussian is a good prior. It seems you want p(z|s) to be as close to a unit Gaussian as possible?\n\t• Figure 4 does not show a clear cluster structure. Explain more?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In contrast to many current hierarchical reinforcement learning approaches, the authors present a decentralized method that learns low level policies that decide for themselves whether to act in the current state, rather than having a centralized higher level meta policy that chooses between low level policies.  The reviewers primarily had minor concerns about clarity, reward scaling, and several other issues that were clarified by the authors.  The only outstanding concern is that of whether transfer/pretraining is required for the experiments to work or not.  While this is an interesting question that I would encourage authors to address as much as possible, it does not seem like a dealbreaker in light of the reviewers' agreement on the core contribution.  Thus, I recommend this paper for acceptance.
Decision: Accept (Poster)