Under review as a conference paper at ICLR 2018

H IERARCHICAL AND I NTERPRETABLE S KILL ACQUI SITION IN M ULTI - TASK R EINFORCEMENT L EARNING
Tianmin Shu∗
University of California, Los Angeles
tianmin.shu@ucla.edu

Caiming Xiong†& Richard Socher
Salesforce Research
{cxiong, rsocher}@salesforce.com

arXiv:1712.07294v1 [] 20 Dec 2017

A BSTRACT
Learning policies for complex tasks that require multiple different skills is a major
challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-world scenarios. This paper proposes a novel framework for efficient
multi-task reinforcement learning. Our framework trains agents to employ hierarchical policies that decide when to use a previously learned policy and when to
learn a new skill. This enables agents to continually acquire new skills during different stages of training. Each learned task corresponds to a human language description. Because agents can only access previously learned skills through these
descriptions, the agent can always provide a human-interpretable description of
its choices. In order to help the agent learn the complex temporal dependencies
necessary for the hierarchical policy, we provide it with a stochastic temporal
grammar that modulates when to rely on previously learned skills and when to
execute new skills. We validate our approach on Minecraft games designed to
explicitly test the ability to reuse previously learned skills while simultaneously
learning new skills.

1

I NTRODUCTION

Deep reinforcement learning has demonstrated success in policy search for tasks in domains like
game playing (Mnih et al., 2015; Silver et al., 2016; 2017; Kempka et al., 2016; Mirowski et al.,
2017) and robotic control (Levine et al., 2016a;b; Pinto & Gupta, 2016). However, it is very difficult
to accumulate multiple skills using just one policy network Teh et al. (2017). Knowledge transfer
techniques like distillation (Bengio, 2012; Rusu et al., 2016; Parisotto et al., 2016; Teh et al., 2017)
have been applied to train a policy network both to learn new skills while preserving previously
learned skill as well as to combine single-task policies into a multi-task policy. Existing approaches
usually treat all tasks independently. This often prevents full exploration of the underlying relations
between different tasks. They also typically assume that all policies share the same state space and
action space. This precludes transfer of previously learned simple skills to a new policy defined over
a space with differing states or actions.
When humans learn new skills, we often take advantage of our existing skills and build new capacities by composing or combining simpler ones. For instance, learning multi-digit multiplication
relies on the knowledge of single-digit multiplication; learning how to properly prepare individual
ingredients facilitates cooking dishes based on complex recipes.
Inspired by this observation, we propose a hierarchical policy network which can reuse previously
learned skills alongside and as subcomponents of new skills. It achieves this by discovering the
underlying relations between skills.
To represent the skills and their relations in an interpretable way, we also encode all tasks using
human instructions such as “put down.” This allows the agent to communicate its policy and generate
plans using human language. Figure 1 illustrates an example: given the instruction “Stack blue,” our
hierarchical policy learns to compose instructions and take multiple actions through a multi-level
hierarchy in order to stack two blue blocks together. Steps from the top-level policy π3 (i.e., the red
∗
†

This work was done when the author was an intern at Salesforce Research.
Corresponding author

1

Under review as a conference paper at ICLR 2018

Task: “Stack blue”

Task accumulation

“Get blue”
Task set: {Find/Get/Put x}

“Find blue”

⇡2

“Get blue”

“Find blue”

⇡1

Task set: {Find/Get x}

⇡1

⇡0

Action:
move forward
Egocentric view

…

⇡2

“Find blue”

“Find blue”
Task set: {Find x}

“Put blue”

⇡2

Policy unfolding

⇡3

Task set: {Find/Get/Put/Stack x}

⇡0
Action:
turn right

Action:
pick up

Action:
turn left

…

…

Action:
move forward

Action:
put down

…

Item in hands

None

None

Blue block

Blue block

Blue block

None

Time

t=1

t=4

t=5

t=6

t=9

t = 10

Figure 1: Example of our multi-level hierarchical policy for a given task – stacking two blue blocks.
Each arrow represents one step generated by a certain policy and the colors of arrows indicate the
source policies. Note that at each step, a policy either utters an instruction for the lower-level policy
or directly takes an action.
branches) outline a learned high-level plan – “Get blue → Find blue → Put blue.” In addition, from
lower level policies, we may also clearly see composed plans for other tasks. Based on policy π2 ,
for instance, the task “Get blue” has two steps – “Find blue → action: turn left,” whereas “Put blue”
can be executed by a single action “put down” according to π3 . Through this hierarchical model, we
may i) accumulate tasks progressively from a terminal policy to a top-level policy and ii) unfold the
global policy from top-level to basic actions.
In order to better track temporal relationships between tasks, we train a stochastic temporal grammar
(STG) model on the sequence of policy selections (previously learned skill or new skill) for positive
episodes. The STG focuses on modeling priorities of tasks: for example, it is necessary to obtain an
object before putting it down. Integrating the STG into the hierarchical policy boosts efficiency and
accuracy by explicitly modeling such commonsense world knowledge.
We validated our approach by testing it on object manipulation tasks implemented in a Minecraft
world. Our experimental results demonstrate that this framework can (i) efficiently learn hierarchical policies and representations for multi-task RL; (ii) learn to utter human instructions to deploy
pretrained policies, improve their explainability and reuse skills; and (iii) learn a stochastic temporal
grammar via self-supervision to predict future actions.

2

R ELATED W ORK

Multi-task Reinforcement Learning. Previous work on multi-task reinforcement learning mainly
falls into two families: knowledge transfer through distillation (Rusu et al., 2016; Parisotto et al.,
2016; Teh et al., 2017; Tessler et al., 2017) or modular policy design through 2-layer hierarchical
policy (Andreas et al., 2017). Our multi-level policy is more similar to the latter approach. The
main differences between our model and the one in Andreas et al. (2017) are two-fold: i) we do
not assume that a global task can be executed by only performing predefined sub-tasks; ii) in our
multi-level policy, global tasks at a lower-level layer may also be used as sub-tasks by global tasks
carried out at higher-levels.
Hierarchical Reinforcement Learning. Complex policies often require the modeling of longer
temporal dependencies than what standard Markov decision processes (MDPs) can capture. To
combat this, hierarchical reinforcement learning was introduced to extend MDPs to semi-MDPs
(Sutton et al., 1999), where options (or macro actions) are introduced on top of primitive actions
to decompose the goal of a task into multiple subgoals. In hierarchical RL, two sets of policies are
trained: local policies that map states to primitive actions for achieving subgoals, and a global policy
that initiates suitable subgoals in a sequence to achieve the final goal of a task (Bacon & Precup,
2015; Kulkarni et al., 2016; Vezhnevets et al., 2016; Tessler et al., 2017; Andreas et al., 2017).
2

Under review as a conference paper at ICLR 2018

This two-layer hierarchical policy design significantly improves the ability of discovering complex
policies which can not be learned by flat policies. However, it also makes some strict assumptions
that limit its flexibility: i) a task’s global policy cannot use a simpler task’s policy as part of its base
policies; ii) a global policy is assumed to be executable by only using local policies over specific
options. In this work, we aim to learn a multi-level global policy which does not have these two
assumptions. In addition, previous work usually use a latent variable to represent a task. In our
work, we encode a task by a human instruction to learning task-oriented language grounding as well
as to improve the interpretability of plans composed by our hierarchical policies.
Language grounding via reinforcement learning. Recently, there has been work on grounding human language in 3D game environments (Hermann et al., 2017; Chaplot et al., 2017) or in text-based
games (Narasimhan et al., 2015) via reinforcement learning. In these games agents are instructed to
pick up an item described by a sentence. Besides visual grounding, Andreas et al. (2017) grounded
instructions (not necessarily using human language) to local policies in hierarchical reinforcement
learning. Our approach not only learns the language grounding for both visual knowledge and policies, but is also trained to utter human instructions as an explicit explanation of its decisions to
humans. To our knowledge, this is the first model that learns to compose plans for complex tasks
based on simpler ones which have human descriptions.

3

M ODEL

In this section, we discuss our multi-task RL setting, hierarchical policy, stochastic temporal grammar, and how interaction of these components can achieve plan composition.
3.1

M ULTITASK RL S ETTING

Let G be a task set, where each task g is uniquely described by a human instruction. For simplicity, we assume a two-word tuple template consisting of a skill and an item for such a phrase,
i.e., huskill , uitem i. Each tuple describes an object manipulation task. In this paper, we define
g = huskill , uitem i by default, thus tasks and instructions are treated as interchangeable concepts.
For each task, we define a Markov decision process (MDP) represented by states s ∈ S and primitive
actions a ∈ A. Rewards are specified for goals of different tasks, thus we use a function R(s, g) to
signal the reward when performing any given task g.
We assume that as a starting point, we have a terminal policy π0 (as shown in Figure 2a) trained for
a set of basic tasks (i.e., a terminal task set G0 ). The task set is then progressively increased as the
agent is instructed to do more tasks by humans at multiple stages, such that G0 ⊂ G1 ⊂ · · · ⊂ GK ,
which results in life-long learning of polices from π0 for G0 to πK for GK as illustrated by the
“task accumulation” direction in Figure 1. At stage k > 0, Gk−1 is defined as the base task set of
Gk . The tasks in Gk−1 are named as base tasks at this stage and πk−1 becomes the base policy of
πk . Here, we utilize weak supervision from humans to define what tasks shall be augmented to the
previous task set at each new stage. But in general, our model is suitable for arbitrary order of task
augmentation.
3.2

H IERARCHICAL P OLICY

One of our key ideas is that a new task in current task set Gk may be decomposed into several simpler
subtasks, some of which can be base tasks in Gk−1 executable by base policy πk−1 . Therefore,
instead of using a flat policy (Figure 2a) as π0 that directly maps state and human instruction to
a primitive action, we propose a hierarchical design (Figure 2b) with the ability to reuse the base
policy (i.e., πk−1 ) for performing base tasks as subtasks. Namely, at stage k, the global policy πk
is defined by a hierarchical policy. This hierarchy consists of four sub-policies: a base policy for
executing previously learned tasks, an instruction policy that manages communication between the
global policy and the base policy, an augmented flat policy which allows the global policy to directly
execute actions, and a switch policy that decides whether the global policy will primarily rely on the
base policy or the augmented flat policy.
The base policy is defined to be the global policy at the previous stage k − 1. The instruction policy
maps state s and task g ∈ Gk to a base task g 0 ∈ Gk−1 . The purpose of this policy is to inform base
3

Under review as a conference paper at ICLR 2018

“Get white”
“Get white”

Visual
Encoder

Visual
Encoder

Instruction
Encoder

Global Policy

⇡k

Fusion

Fusion

V (s, g)

Instruction
Policy
LSTM

Instruction
Encoder

LSTM

V (s, g)

V sw (s, e, g)

Base Policy

⇡k

Policy

⇡k

1

Switch
Policy

Augmented
Policy

Selector
action

action

Actions

Actions

(a) Flat policy.

(b) Hierarchical design of global policy.

Figure 2: Flat and hierarchical policy architectures.
policy πk−1 which base tasks it needs to execute. Since an instruction is represented by two words,
we define the instruction policy using two conditionally independent distributions, i.e., πkinst (g 0 =
aug
item
huskill , uitem i|s, g) = pskill
k (uskill |s, g)pk (uitem |s, g). An augmented flat policy, πk (a|s, g), maps
state s and task g to a primitive action a for ensuring that the global policy is able to perform novel
tasks in Gk that can not be achieved by only reusing the base policy. To determine whether to perform
a base task or directly perform a primitive action at each step, the global policy further includes a
switch policy, πksw (e|s, g), where e is a binary variable indicating the selection of the branches, πkinst
(e = 0) or πkaug (e = 1).
Note that the above description of the hierarchical policy does not account for an STG. The instruction policy and switch policy introduced here are simplified from the ones in the full model (see
Section 3.3).
At each time step, we first sample et from our switch policy πksw to decide whether the global policy
πk will rely on the base policy πk−1 or the augmented flat policy πkaug . We also sample a new
instruction gt0 from our instruction policy πkinst in order to sample actions from the base policy. This
can be summarized as:
et ∼ πksw (et |st , g),
(1)
gt0 ∼ πkinst (gt0 |st , g),

(2)

at ∼ πk (at |st , g) = πk−1 (at |st , gt0 )(1−et ) πkaug (at |st , g)et ,

(3)

and finally
where πk and πk−1 are the global policies at stage k and k − 1 respectively. After each step, we will
also obtain a reward rt = R(st , g).
3.3

S TOCHASTIC T EMPORAL G RAMMAR

Different tasks may have temporal relations. For instance, to move an object, one needs to first find
and pick up that object. There has been previous research (Si et al., 2011; Pirsiavash & Ramanan,
2014) using stochastic grammar models to capture such temporal relations. Inspired by this, we
summarize temporal transitions between various tasks with an stochastic temporal grammar (STG).
In our full model, the STG interacts with the hierarchical policy described above through modified
switch policy and instruction policy by using the STG as a prior. This amounts to treating the past
history of switches and instructions in positive episodes as a guidance on whether the hierarchical
policy should defer to the base policy to execute a specific base task or employ its own augmented
flat policy to take a primitive action.
4

Under review as a conference paper at ICLR 2018

In an episode, the temporal sequence of et and gt0 , i.e., {het , gt0 i; t ≥ 0}, can be seen as a finite
state Markov chain (Baum & Petrie, 1966). Note that the state here is referred to the tuple het , gt0 i,
which is not the state of the game st ∈ S defined in Section 3.1. Consequently, at each level k > 0,
0
we may define an STG of a task g by i) transition probabilities, ρk (et , gt0 |et−1 , gt−1
, g), and ii) the
distribution of he0 , g00 i, qk (e0 , g00 |g).
With the estimated probabilities, we sample et and gt0 in an episode at level k > 0 w.r.t. to reshaped
0
0
policies πksw and πkinst respectively:
• If t = 0,
e0 ∼ πksw 0 (e0 |st , g) ∝ πksw (e0 |st , g)

X
g 0 ∈G

qk (e0 , g 0 |g),

(4)

k−1

0

g00 ∼ πkinst (g00 |st , g) ∝ πkinst (g00 |st , g)qk (e0 = 0, g00 |g);

(5)

• Otherwise,
0
et ∼ πksw 0 (et |et−1 , gt−1
, st , g) ∝ πksw (et |st , g)

X

0
ρk (et , g 0 |et−1 , gt−1
, g),

(6)

g 0 ∈Gk−1
0

0
0
gt0 ∼ πkinst (gt0 |et−1 , gt−1
, st , g) ∝ πkinst (gt0 |st , g)ρk (et = 0, gt0 |et−1 , gt−1
, g).

(7)

Note that primitive action sampling is not affected by the STG.
3.4

P LAN C OMPOSITION

Combined with our hierarchical policy and STG defined above, we are able to run an episode to
compose a plan for a task specified by a human instruction. Algorithm 1 in Appendix A summarized
this procedure with respect to the policy and STG at level k. Note that to fully utilize the base policy,
we assume that once triggered, a base policy will play to the end before the global policy considers
the next move.

4

L EARNING

The learning algorithm is outlined in Algorithm 2 in Appendix A. We learn our final hierarchical
policy through k stages of skill acquisition. Each of these stages is broken down into a base skill
acquisition phase and a novel skill acquisition phase in a 2-phase curriculum learning.
In the base skill acquisition phase, we only sample tasks from the base task set Gk−1 . This ensures
that the global policy learns how to use previously learned skills by issuing instructions to the base
policy. In other words, this phase teaches the agent how to connect its instruction policy to its base
policy. Once the average reward for all base tasks exceeds a certain threshold, we proceed to the
next phase.
In the novel skill acquisition phase, we sample tasks from the full task set, Gk , for the k-th stage of
skill acquisition. It is in this phase that the agent can learn when to rely on the base policy and when
to rely on the augmented flat policy for executing novel tasks.
In each of these phases, all policies are trained with advantage actor-critic (A2C) (Section 4.1) and
distributions in the STG are estimated based on accumulated positive episodes (Section 4.2).
4.1

P OLICY O PTIMIZATION BY A DVANTAGE ACTOR -C RITIC

We use advantage actor-critic (A2C) for policy optimization with off-policy learning (Su et al.,
2017). Here, we only consider the gradient for global policies (i.e., k > 0) as we assume the
terminal policy has been trained as initial condition. Let Vk (st , g) be a value function indicating the
expected return given state st and task g. To reflect the nature of the branch switching in our model,
we introduce another value function Vksw (st , et , g) to represent the expected return given state st ,
task g and current branch selection et .
5

Under review as a conference paper at ICLR 2018

aug
inst
Thus, given a trajectory Γ = {hst , et , gt0 , at , rt , µsw
k (·|st ), µk (·|st , g), µk (·|st , g), gi : t =
aug
inst
0, 1, · · · , T } generated by old policies µsw
k (·|st ), µk (·|st , g), and µk (·|st , g), the policy gradient reweighted by importance sampling can be formulated as
ωtsw ∇θsw log πksw (et |st , g)A(st , g, et )
|
{z
}
1st term: switch policy gradient

+

(1 − et )ωtinst ∇θinst log πkinst (gt0 |st , g)A(st , g, et , gt0 )
|
{z
}

+ et ωtaug ∇θaug
|

{z

3rd term: augmented policy gradient

π sw (e |s ,g)

(8)

2nd term: instruction policy gradient
log πkaug (at |st , g)A(st , g, et , at ),

}
π aug (a |s ,g)

π inst (g 0 |s ,g)

where ωtsw = µksw (ett |stt ,g) , ωtinst = µkinst (gt0 |stt ,g) , and ωtaug = µkaug (att |stt ,g) are importance sampling
k
t
k
k
weights for the three terms respectively; A(st , g, et ), A(st , g, et , gt0 ), and A(st , g, et , at ) are estimates of advantage functions, which have multiple possible definitions. In this paper, we define
them by the difference between empirical return and value function estimation:
P
P∞ τA(st , g, et ) =
∞
τ
0
γ
R(s
,
g)
−
V
(s
,
g),
A(s
,
g,
e
,
g
)
=
A(s
,
g,
e
,
a
)
=
t+τ
k
t
t
t
t
t
t
t
τ =0
τ =0 γ R(st+τ , g) −
Vksw (st , g, et ), where γ is the discounted coefficient.
Finally, the value functions can be updated using the following gradient:
"∞
#2
"∞
#2
1 X τ
1 X τ
sw
∇θ v
γ R(st+τ , g) − Vk (st , g) + ∇θvsw
γ R(st+τ , g) − Vk (st , et , g) .
2 τ =0
2 τ =0

(9)

To increase the episode efficiency, after running an episode, we conduct n mini-batch updates where
n is sampled from a Poisson distribution with λ = 4, similar to Wang et al. (2017). Note that one
can also apply other common policy optimization methods, e.g., A3C (Mnih et al., 2016), to our
model. We leave this as future work to evaluate the efficiency of different methods when using our
model.
Optimizing all three sub-policies together leads to unstable learning. To avoid this, we apply a
simple alternating update procedure. For each set of M iterations, we keep two of the sub-policies
fixed and train only the single policy that remains. When we reach M iterations, we switch the
policy that is trained. For all experiments in this paper, we use M = 500. This alternating update
procedure is used within both phases of curriculum learning.
4.2

L EARNING AN STG

If at any point in the aforementioned training process the agent receives a positive reward after an
episode, we update the stochastic temporal grammar. ρk and qk of the STG are both initialized to
be uniform distributions. Since the STG is a finite state Markov chain over tuples het , gt0 i, we use
maximum likelihood estimation (MLE) to update the distributions (Baum & Petrie, 1966). As the
training progresses, the STG starts to guide the exploration.
To avoid falling into local minima in the early stages of training, it is important to encourage random
exploration in early episodes. Based on our experiments, we find that using -greedy suffices.

5
5.1

E XPERIMENTS
G AME E NVIRONMENT AND TASK S PECIFICATIONS

Figure 3 shows the two room environment in Minecraft that we created using the Malmo platform
(Johnson et al., 2016). In each episode, an arbitrary number of blocks with different colors (totally
6 colors in our experiments) are randomly placed in one of the two rooms. The agent is initially
placed in the same room with the items. We consider four sets of tasks: i) G (0) = {“Find x”},
walking to the front of a block with color x, ii) G (1) = {“Get x”}, picking up a block with color x,
iii) G (2) = {“Put x”}, putting down a block with color x, and iv) G (3) = {“Stack x”}, stacking two
blocks with color x together. In total, there are 24 tasks. An agent can perform the following actions:
“move forward,” “move backward,” “move left,” “move right,” “turn left,” “turn right,” “pick up,”
“put down.”
6

Under review as a conference paper at ICLR 2018

Without loss of generality, we assume the following skill acquisition order: Gk =
∪kκ=1 G (κ) , ∀k = 0, 1, 2, 3, which is a natural way to increase skill sets. One
may also alter the order, and the main conclusions shall still hold. This results in
policies {πk : k = 0, 1, 2, 3} for these four task sets.

Room 1

Room 2

We adopt a sparse reward function: when reaching the goal of a task, the agent gets
a +1 reward; when generating an instruction g 0 that is not executable in current
3:
game (e.g., trying to find an object that does not exist in the environment), we give Figure
a −0.5 reward; otherwise, no reward will be given. Whenever a non-zero reward Room layout.
is given, the game terminates.
5.2

I MPLEMENTATION D ETAILS

We specify the architecture of the modules in our model in Appendix B, where the visual and instruction encoding modules have the same architectures as the ones in Hermann et al. (2017). We
train the network with RMSProp (Tieleman & Hinto, 2012) with a learning rate of 0.0001. We set
the batch size to be 36 and clip the gradient to a unit norm. For all tasks, the discounted coefficient
is γ = 0.95. For the 2-phase curriculum learning, we set the average reward threshold to be 0.9
(average rewards are estimated from the most recent 200 episodes of each task).

1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2

0.8

0

ours
ours w/o grammar
ours w/o alternating
ours w/o V sw(s, e, g)
flat policy
10000 15000 20000
Episodes

5000

Reward

Reward

To encourage random exploration, we apply -greedy to the decision sampling for the global policy
(i.e., only at the top level k at each stage k > 0), where  gradually decreases from 0.1 to 0.

0.6
0.4
0.2
0

(a) Learning curves for π1 .

5000

ours
ours w/o grammar
ours w/o alternating
ours w/o V sw(s, e, g)
10000 15000 20000 25000
Episodes

(b) Learning phase 2 for π3 .

1.0

1.0

0.8

0.8
Reward

Reward

Figure 4: Comparison of learning efficiency on two task sets: (a) G1 for global policy π1 and (b) G3
for global policy π3 respectively.

0.6
0.4
0.2
0

5000

ours
ours w/o curriculum
10000 15000 20000
Episodes

(a) Learning curves for π1 .

0.6
0.4
0.2
0

ours
ours w/o curriculum
5000 10000 15000 20000 25000 30000
Episodes

(b) Learning curves for π3 .

Figure 5: Effect of our 2-phase curriculum learning.
5.3

L EARNING E FFICIENCY

To evaluate the learning efficiency, we compare our full model with 1) a flat policy (Figure 2a)
as in Hermann et al. (2017) fine-tuned on the terminal policy π0 and variants of our approach: 2)
ours without STG, 3) ours without alternating policy optimization, and 4) ours without Vksw (s, e, g)
(replaced by Vk (s, g) instead). Note that all the rewards have been converted to the same range, i.e.,
[0, 1] for the sake of fair comparison.
7

Under review as a conference paper at ICLR 2018

Table 1: Success rates in different game environments. All policies are trained in a small room.
Method
Full model
Flat policy

Find x
0.995
0.980

Small room
Get x Put x
0.970 1.00
0.965 1.00

Stack x
0.955
-

Find x
0.723
0.515

Big room
Get x Put x
0.648 1.00
0.450 1.00

Stack x
0.613
-

In Figure 4a, we use various methods to train policy π1 for the task set G1 based on the same base
policy π0 . The large dip in the reward indicates that the curriculum learning switches from phase
1 to phase 2. From Figure 4a, we may clearly see that our full model and variants can all converge
within 22,000 episodes, whereas the average reward of the flat policy is still below 0.8 given the
same amount of episodes. In addition, our full model finishes phase 1 significantly faster than other
methods and its curve of average reward maintains notably higher than the remaining ones.
To further examine the learning efficiency during phase 2 when new tasks are added into the training
process, we first pretrain π3 using our full model following our definition of phase 1 in the curriculum learning. We then proceed to learning phase 2 using different approaches all based on this
pretrained policy. As shown in Figure 4b, our full model has the fastest convergence and the highest
average reward upon convergence. By comparing Figure 4a and Figure 4b, we further show that our
full model has a bigger advantage when learning more complex tasks.
To demonstrate the effect of our 2-phase curriculum learning on the training efficiency, we visualize
the learning curves of our model trained with and without the curriculum learning respectively in
Figure 5. According to the results, the curriculum learning indeed helps accelerate the convergence,
which empirically proves the importance of encouraging a global policy to reuse relevant skills
learned by its base policy.
5.4

P OLICY G ENERALIZATION

Finally, we evaluate how the hierarchical design and encoding tasks by human instructions benefit
the generalization of learned policies in the following two ways.
First, we train π1 in a simpler setting where in each episode, only one item (i.e, the target item of
the given task) is present. We then test the policy π1 for “Get x” tasks in a room where there will be
multiple items serving as distraction and the agent must interact with the correct one. Both the flat
policy and the hierarchical policy can achieve near perfect testing success rate in the simple setting.
However, in the more complex setting, flat policy can not differentiate the target item from other
items that are also placed in the room (the success rate drops to 29%), whereas our hierarchical
policy still maintains a high success rate (94%). This finding suggests that the hierarchical policy
not only picks up the concept of “find” and “get” skills as the flat policy does, but also inherits the
concept of items from the base policy by learning to utter correct instructions to deploy “find” skill
in the base policy.
Second, we remove the wall between the two rooms shown in Figure 3 and test the flat policy and
our full model in this bigger room for various tasks. Both policies are trained in the two small rooms.
There are multiple items in a room for both training and testing cases. Note that the success rate of
stacking tasks by the flat policy is not shown here since it is extremely difficult for the flat policy to
converge to a decent policy for the full task set G3 even after 4 days of training. The success rates
are summarized in Tab. 1. Using the flat policy results in a much bigger drop in the testing success
rate compared to using out full model. This is mainly because that our global policy will repeatedly
call its base policy to execute the same task until the agent finally achieves the goal even though
the trained agent is unable to reach the goal by just one shot due to the simplicity of the training
environment.
5.5

P OLICY I NTERPRETABILITY

We visualize typical hierarchical plans of several tasks generated by global policies learned by our
full model in Appendix C (Figure 6). It can been seen from the examples that our global policies
8

Under review as a conference paper at ICLR 2018

adjust the composed plans in different scenarios. For instance, in the second plan on the first row, π1
did not deploy base policy π0 as the agent was already in front of the target item at the beginning of
the episode, whereas in the plan on the second row, π1 deployed π0 for the “Find x” base task twice
consecutively, as it did not finish the base task in the first call.

6

C ONCLUSION

In this work, we have proposed a hierarchal policy modulated by a stochastic temporal grammar as
a novel framework for efficient multi-task reinforcement learning through multiple training stages.
Each task in our settings is described by a human instruction. The resulting global policy is able
to reuse previously learned skills for new tasks by generating corresponding human instructions to
inform base policies to execute relevant base tasks. We evaluate this framework in Minecraft games
and have shown that our full model i) has a significantly higher learning efficiency than a flat policy
does, ii) generalizes well in unseen environments, and iii) is capable of composing hierarchical plans
in an interpretable manner.
Currently, we rely on weak supervision from humans to define what skills to be learned in each
training stage. In the future, we plan to automatically discover the optimal training procedures to
increase the task set.

R EFERENCES
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with
policy sketches. In International Conference on Machine Learning (ICML), 2017.
Pierre-Luc Bacon and Doina Precup. The option-critic architecture. In NIPS Deep Reinforcement
Learning Workshop, 2015.
Leonard E. Baum and Ted Petrie. Statistical inference for probabilistic functions of finite state
markov chains. The Annals of Mathematical Statistics, 37(6):1554–1563, 1966.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In JMLR:
Workshop on Unsupervised and Transfer Learning, 2012.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language
grounding. arXiv preprint arXiv:1706.0723, 2017.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David
Szepesvari, Wojtek Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris
Apps, Demis Hassabis, and Phil Blunsom. Grounded language learning in a simulated 3d world.
arXiv preprint arXiv:1706.06551, 2017.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In International Joint Conference on Artificial Intelligence
(IJCAI), 2016.
Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Tocze, and Wojciech Jaśkowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In IEEE Conference
on Computational Intelligence and Games (CIG), 2016.
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, , and Josh Tenenbaum. Hierarchical deep
reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in
Neural Information Processing Systems (NIPS), 2016.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016a.
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, , and Deirdre Quillen. Learning handeye coordination for robotic grasping with deep learning and large-scale data collection. In ”
International Symposium on Experimental Robotics (ISER), 2016b.
9

Under review as a conference paper at ICLR 2018

Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha
Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. In International Conference on Learning
Representations (ICLR), 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen,
Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning.
Nature, 518(7540):529–533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim
Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement
learning. In International Conference on Machine Learning (ICML), 2016.
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based
games using deep reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In International Conference on Learning Representations (ICLR),
2016.
Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and
700 robot hours. In IEEE Conference on Robotics and Automation (ICRA), 2016.
Hamed Pirsiavash and Deva Ramanan. Parsing videos of actions with segmental grammars. In IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In International Conference on Learning Representations (ICLR), 2016.
Zhangzhang Si, Mingtao Pei, Benjamin Yao, and Song-Chun Zhu. Unsupervised learning of event
and-or grammar and semantics from video. In IEEE International Conference on Computer Vision
(ICCV), 2011.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, and Julian Schrittwieser et al. Mastering the game of go with deep neural networks and tree
search. Nature, 529(7587):484–489, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez,
Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan
Hui, Laurent Sifre, George Van Den Driessche, Thore Graepel, and Demis Hassabis. Mastering
the game of go without human knowledge. Nature, 550(7676):354–359, 2017.
Pei-Hao Su, Pawel Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. Sample-efficient
actor-critic reinforcement learning with supervised data for dialogue management. In The 18th
Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL), 2017.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps anad semi-mdps: A framework
fro temporal abstraction in reinforcement learning. Artificial Intelligence, 112:181–211, 1999.
Yee Whye Teh, Victor Bapst, Wojciech Marian Czarnecki, John Quan, James Kirkpatrick, Raia
Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning.
In International Conference on Learning Representations (ICLR), 2017.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J. Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. In AAAI Conference on Artificial Intelligence
(AAAI), 2017.
Tijmen Tieleman and Geoffrey Hinto. Lecture 6.5—rmsprop: Divide the gradient by a running
average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
10

Under review as a conference paper at ICLR 2018

Alexander (Sasha) Vezhnevets, Volodymyr Mnih, John Agapiou, Simon Osindero, Alex Graves,
Orial Vinyals, and Koray Kavukcuoglu. Strategic attentive writer for learning macro-actions. In
Advances in Neural Information Processing Systems (NIPS), 2016.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu,
and Nando de Freitas. Sample efficient actor-critic with experience replay. In International
Conference on Learning Representations (ICLR), 2017.

11

Under review as a conference paper at ICLR 2018

A

P SEUDO C ODE OF O UR A LGORITHMS

Algorithm 1 RUN(k, g)
Input: Policy level k, task g ∈ Gk
Output: Episode trajectory Γ at the top level policy
1: t ← 0
2: Γ = ∅
3: Get initial state s0
4: repeat
5:
if k == 1 then
6:
Sample at ∼ πk (·|st , g) and execute at
7:
Get current state st+1
8:
rt ← R(st+1 , g)
9:
Add hst , at , rt , πk (·|st , g), gi to Γ
10:
else
11:
Sample et and gt0 as in Section 3.3 for using STG as guidance
12:
Sample at ∼ πkaug (·|st , g)
13:
if et = 0 then
14:
// Execute base policy πk−1 by giving instruction gt0
15:
RUN(k − 1, gt0 )
16:
else
17:
Execute at
18:
end if
19:
Get current state st+1
20:
rt ← R(st+1 , g)
21:
Add hst , et , gt0 , at , rt , πksw (·|st ), πkinst (·|st , g), πkaug (·|st , g), gi to Γ
22:
end if
23:
t←t+1
24: until t > T or rt 6= 0

Algorithm 2 Learning global policy and STG at stage k > 0
1: Specify λ, maximum training iterations N , alternating update rotation frequency M , and reward threshold
Rmin
2: Initialize total replay memory D ← ∅ and its subset for positive episodes D+ ← ∅
3: Initialize current iteration id i ← 0 and set current updating term to be τ ← 1
4: Initialize parameters of policies and value functions Θ = hθsw , θinst , θaug , θv , θvsw i
5: Initialize distributions of the STG as uniform distributions
6: repeat
7:
Determine current learning phase by comparing average rewards of tasks in Gk−1 with Rmin
8:
if in curriculum learning phase 1 then
9:
Sample a task g from base task set Gk−1
10:
else
11:
Sample a task g from global task set Gk
12:
end if
13:
//Run an episode
14:
Γ ← RUN(k, g)
15:
D ←D∪Γ
16:
if the maximum reward in Γ is +1 then
17:
D+ ← D+ ∪ Γ
18:
Re-estimate the distributions of the STG based on updated D+ by MLE
19:
end if
20:
Sample n ∼ Possion(λ)
21:
for j ∈ {1, · · · , n} do
22:
Sample a mini-batch S from D
23:
Update Θ based on (9) and the τ -th term in (8)
24:
i←i+1
25:
if i%M = 0 then
26:
τ ← τ %3 + 1
27:
end if
28:
end for
29: until i ≥ N

12

Under review as a conference paper at ICLR 2018

B

A RCHITECTURES OF M ODULES

The architecture designs of all modules in our model shown in Figure 2 are as follows:
Visual Encoder extracts feature maps from an input RGB frame with the size of 84 × 84 through
three convolutional layers: i) the first layer has 32 filters with kernel size of 8 × 8 and stride of 4; ii)
the second layer has 64 filters with kernel size of 4 × 4 and stride of 2; iii) the last layer includes 64
filters with kernel size of 3 × 3 and stride of 1. The feature maps are flatten into a 3136-dim vector.
We reduce the dimension of this vector to 256 by a fully connected (FC) layer resulting a 256-dim
visual feature as the final output of this module.
Instruction Encoder first embeds each word into a 128-dim vector and combines them into a single
vector by bag-of-words (BOW). Thus the output of this module is a 128-dim vector.
Fusion layer simply concatenates the encoded visual and language representations together and
outputs 384-dim fused representation. We then feed this 384-dim vector into an LSTM with 256
hidden units. The hidden layer output of the LSTM is served as the input of all policy modules and
value function modules.
Switch Policy module has a FC layer with output dimension of 2 and a softmax activation to
get πksw (e|s, g). Instruction Policy module has two separate FC layers, both of which are activated by softmax to output the distribution of skill, pskill
k (uskill |s, g), and the distribution of item,
pitem
(u
|s,
g),
respectively.
Augmented
Policy
module
outputs πaug (a|s, g) also through a FC
item
k
layer and softmax activation. The two Value Function modules, V (s, g) and V sw (s, e, g), all have
a scalar output through a FC layer.
Finally, the Selector module selects the action sampled from Augmented Policy module or Base
Policy module based on the switching decision sampled from the Switch Policy module.

C

C OMPOSED H IERARCHICAL P LANS

Figure 6 shows several plans for different tasks composed by executing our hierarchical policies.

13

Under review as a conference paper at ICLR 2018

Figure 6: Samples of typical hierarchical plans for different tasks composed by our global policies.
Note that all tasks must start from the top-level policy. The branches are ordered from left to right
in time indicating consecutive steps carried out by a policy. We also show the egocentric view and
the item in hands at critical moments for a real episode example.

14

