Hierarchical Meta Reinforcement Learning for Multi-Task Environments
Authors: Dongyang Zhao
Yue Huang
Changnan Xiao
Yue Li
Shihong Deng
Venue:  (2021)
Keywords: Reinforcement Learning
Multi-task
Hierarchical
Meta Learning

Abstract: Deep reinforcement learning algorithms aim to achieve human-level intelligence by solving practical decisions-making problems, which are often composed of multiple sub-tasks. Complex and subtle relationships between sub-tasks make traditional methods hard to give a promising solution. We implement a first-person shooting environment with random spatial structures to illustrate a typical representative of this kind. A desirable agent should be capable of balancing between different sub-tasks: navigation to find enemies and shooting to kill them. To address the problem brought by the environment, we propose a Meta Soft Hierarchical reinforcement learning framework (MeSH), in which each low-level sub-policy focuses on a specific sub-task respectively and high-level policy automatically learns to utilize low-level sub-policies through meta-gradients. The proposed framework is able to disentangle multiple sub-tasks and discover proper low-level policies under different situations. The effectiveness and efficiency of the framework are shown by a series of comparison experiments. Both environment and algorithm code will be provided for open source to encourage further research.

Under review as a conference paper at ICLR 2021
HIERARCHICAL META REINFORCEMENT LEARNING FOR MULTI-TASK ENVIRONMENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep reinforcement learning algorithms aim to achieve human-level intelligence by solving practical decisions-making problems, which are often composed of multiple sub-tasks. Complex and subtle relationships between sub-tasks make tra- ditional methods hard to give a promising solution. We implement first-person shooting environment with random spatial structures to illustrate a typical rep- resentative this kind. A desirable agent should be capable balancing be- tween different sub-tasks: navigation to find enemies and shooting to kill them. To address the problem brought by the environment, we propose a Meta Soft Hi- erarchical reinforcement learning framework (MeSH), in which each low-level sub-policy focuses on a specific sub-task respectively and high-level policy au- tomatically learns utilize low-level sub-policies through meta-gradients. The proposed framework able to disentangle multiple sub-tasks and discover proper low-level policies under different situations. The effectiveness and efficiency of the framework are shown by a series of comparison experiments. Both environ- ment and algorithm code will be provided for open source encourage further research.
INTRODUCTION
With great breakthrough of deep reinforcement learning (DRL) methods (Miih et al. 2015 Silver al.] 2016) Mnih al., 2016; SShhulman et al., 2015 Lillicrp et al.] 2015). an urgent need to use DRL. methods to solve more complex decision-making problems. The practical problem in real world is often a subtle combination of multiple sub-tasks, which may happen simultaneously and hard to disentangle by time series. For instance, in StarCraft games (Pang et al.) 2019), agents need to consider building units and organizing battles, sub-tasks may change rapidly over the whole game process; sweeping robots tradeoff' between navigating and collecting garbage; shooting agents should move to appropriate positions and launch attacks, etc. The relationship between sub-tasks is complex and subtle. Sometimes they compete with each other and need to focus one task to gain key advantages; at other times, they need to cooperate with each other to maintain the possibility of global exploration. It often time consuming and ineffective to learn simply by collecting experience and rewarding multiple objectives for different sub-tasks.
A reasonable idea is utilize deep hierarchical reinforcement learning (DHRL) methods (Vezhn-l evets et al. 2017 Igl et al. 2020). where the whole system is divided into a high-level agent and several levelevel agents. Low-level agents learn sub-policies, which select atomic actions for cor- responding sub-tasks. The high-level agent is responsible for a meta task in the abstract logic or coarser time granularity. guiding low-level agents by giving goal, or directly selecting among sub- policies. However, DHRL. methods face some inherent problems: due to the complex interaction between multi-level agents, there no theoretical guarantee of convergence, and shows unsta- ble experimental performance. Most DHRL methods require heavy manual design, and end-to-end system lacks reasonable semantic interpretation. In addition, these agents are often constrained by specific tasks and are easy to overfit. Even transferring between similar tasks, they perform poorly and need lot of additional adjustments.
shooting (FPS) environment with random spatial structures. The game contains a 3D scene from human perspective. When the player defeats all enemies, the player wins the game. When the player drops the ground losses all health points, the player loses the
We introduce

---

Under review as a conference paper at ICLR 2021
game. It is very risky for the player to drop to the ground, thus environment contains two key tasks: navigation and combat. The terrain and enemies in the game are randomly generated. This ensures: 1) the agent cannot learn useful information by memorizing coordinates; 2) the possibility of over- fitring is restrained and the generalization ability of learned policy is enhanced. The state information is expressed in the way of raycast. This representation of environment information requires much less computing resources than the raw image representation. It can be trained and tested even with only CPU machines, which makes us pay more attention to the reinforcement learning algorithm itself rather than the computing ablity related to image processing.
For this environment, we propose Meta Soft Hierarchical reinforcement learning framework (MeSH). The high-level policy is a differentiable meta parameter generator, and the low-level policy contains several sub-policies, which are the same form and differentiated automatically in the training procedure. The high-level policy selects and combines sub-policies through the meta pa- rameter and interacts with the environment. We find that the meta generator can adaptively combines sub-policies with the process of the task, and have strong interpretability in semantics. Compared with a series of baselines, the agent has achieved excellent performance FPS environment.
The main contributions of this work are as follows:
clarifying the complex relationship between multi-task composition.
novel meta soft hierarchical reinforcement learning framework, McSH, which uses differ- entiable meta generator to adaptively select sub-policies and shows strong interpretability.
a series of comparison experiments to show the effectiveness of the framework.
an open-sourced environment and code to encourage further research on multi-task RL
In this paper, we discuss the related work in Section 2 We introduce the details of the implemented
environment in Section 3] We show our proposed framework in Section 4 We demonstrate details
of our experiments in Section 5
At last, we conclude in Section
2 RELATED WORK
In decision-making problems with high-dimensional continuous state space, the agent often needs to complete tasks that contain multiple sub-tasks. To complete taxi agent problem (Dietterich) 2000).
the agent needs complete sub-tasks such as pickup, navigate, putdown. Menashe & Stone (2018) proposed Escape Room Domain, which is a testbed for HRL. The agent leaves the room from the starting point and needs to press four buttons of different colors leave the room. In these envi- ronments, the agent needs to optimize several sub-tasks and minimize the mutual negative influence between them. However, sub-tasks in these environments are timing dependent. The proposed meth- ods above are helpless in multi-task environment that needs to fulfill multiple tasks simultaneously.
Architectral solutions use hierarchical structure to decompose tasks into action primitives. [Sut- ton et al. (1999) models temporal abstraction an option top of extended actions, [aacon et al. (2017) proposes an actor-critic option method based on it. Henderson et al. (2017) extend the options framework to learn joint reward-policy options. Besides, Jiang et al. (2019) construct composi- tional structure with languages as abstraction or instruction. Due to specific structure design of these methods, high-level agent is unable to execute multiple sub-policies simultaneously in any form.
Recent HRL works learn intra-goals to instruct sub- sub-policies. Vezhnevets et al. (2017) proposes a manager-worker model, manager abstracts goals and instructs worker. This architecture uses di- rectional goal rather than absolute update goal. Oh et al.] (2017) learns a meta controller to instruct implementation and update of sub-tasks. [g] (2020) presents a new soft hierarchy method based on option, it learns with shared prior and hierarchical posterior policies. Yuu et al. (2020) proposes a method that projects the conflict gradient onto the normal plane to avoid some task gradients dom- inating others. Compared with the hard hierarchy methods, these methods use the state's 's natural features to update the upper-level policy, avoiding the timing constraints of handcrafted sub-tasks. Due to the lack of meaningful learning goals of sub-policies, the low-level policies fail to focus on explainable sub-tasks.
2

---

Under review as a conference paper at ICLR 2021
HRL has been recently combined with meta-learning. Frans al proposes Meta-Learning Shared Hierarchies (MLSH) (Frans et al. 2017), which updates master policy and sub-policy by meta- learning. Zou et al. 2019) uses meta-learning to learn reward shaping. Rakelly al. (2019) extends the idea of disentangling task inference and control to help agents leverage knowledges between tasks. Although these methods reduced the limitation of policy structures, it is difficult learn multi-tasks in parallel due to the fixed time steps selected by the master policy for every sub-policy.
also reasonable learn sub-tasks based on decomposition of rewards. [axe al. (2017) ex- presses behaviors as different weighted task blends by the communication between layers so that can maintain a parallel distributed representation of tasks, but the structure is hard to abstract the relationship between different sub-tasks, Grimm & Singh (2019) and Lin et al. (2019) allocate each state with one sub-reward function, which is a non-overlapping decomposition of spatial space. In contrast to these approaches, our method can represent complex and subtle relationship between multiple sub-tasks and perform them simultaneously with explainable sub-goals.
3 ENVIRONMENT
Lewel
Lavell
Level
Layal
Level 1
Lewel
Figure 1: First-person shooting environment with random terrain.
To better understand multi-task decision-making problems, we firstly introduce first-person shoot- ing environment with random terrain, as shown in Figure [ The game contains a 3D scene, which analogy to human perspective. This makes the behavior of trained agent similar human intelli- gence. The real-time information is shown in Table [ The condition of winning the game is to defeat all the enemies in the game. When the player drops to the ground or losses all health points, the game will be judged as failed. This game is very risky as it's easy to drop. Thus the envin environment contains two key tasks: navigation and combat. The terrain and enemies the game are randomly generated. This ensures: I) the agent cannot learn useful information by memorizing coordinates; 2) the possibility of over-fitting is restrained and the generalization ability learned policy is en- hanced. The state information of the agent is expressed in the way of raycast, shown in Figure This representation of environment information requires much less computing resources than raw image representation. It can be trained and tested even with only CPU machines, which makes us pay more attention to the reinforcement learning algorithm itself rather than the computing ability related image processing.
The generation rules of random terrain are as follows. The maximum generated height of random terrain is set to In the initial state, there only one parcel, which is also the place where the player is born. We add this parcel to the parcel queue. If the parcel queue is not empty, the parcel the head of the queue will be taken out. When the maximum height is reached, we expand the parcel to four directions of the higher level with equal probability. If there is new parcel in the corresponding position of the higher level, new parcel generated. ramp is established between the new parcel and the current parcel and random number of enemy is created at the random position of the new parcel, and then the new parcel is added to the parcel queue. If there new parcels in the corresponding position of the higher level, the parcels are not generated repeatedly, and only the ramp between the two parcels is added. Repeat adding parcels until the parcel queue is empty, then terrain generation is completed.
The FPS environment typically a combination two sub-tasks: navigation and combat. The relationship between these two sub-tasks is subtle and complex. Sometimes they compete with
3

---

Under review as a conference paper at ICLR 2021
The raycast information. The range of the game camera is 60 degrees of vertical view, 90 degrees of horizontal view, and the interval of each ray 5 degrees. Therefore, a total of 247 = (60 5+ 1) 590/5 + 1) rays emitted. The raycast return (object, distance, object, distance, ... as sequence. The object represented by {0: none, I: player itself, mesh, 3: blood bag, 4: enemy); ). and the distance representation is direct transmission value. When there is object touched, the distance is also O)
Figure 2:
Table Real-time Information
Field Name
Field Type
Simple Description
Data Range
hp
float
player's health point player's firing energy player's x-coordinate player's y-coordinate player's 's z-coordinate x-coordinate of player's orientation
[0, 100]
energy X
float
[0, 16]
float
[-30, 20]
y
float
[-1, 16]
z
float
[-80, 10]
rotation_x float
[0, 360]
rotation.y float
y-coordinate of player's orientation 7-coordinate of player's orientation player's vision for object & distance number of enemies killed
[0, 360]
rotation float
[0, 0]
raycast
repeated int32
object: 0-4 distance: [0, +xo]
kill
int32
[0, +00]
each other, but at other times, they cooperate for common objective. For instance, in navigation missions, order to explore more unseen terrain, sometimes we need to fight to clear enemies along the way; but at other times, we have to focus on navigation to pass through narrow terrain. Similarly, in combat missions, sometimes we need to move to get better position to shoot; but at other times, need to focus shooting to kill the enemy quickly. This environment very representative of practical problems, since a large number of them can be divided into several parts which are contradictory and unified. In addition, each sub-task in this environment is simple and clear, but the combination of them greatly increases the difficulty of solving the problem. This forces the RL algorithm focus more on dealing with these complex relationships, rather than the specific techniques for solving single problem.
4 POLICY OPTIMIZATION
4.1 FRAMEWORK
The proposed McSH framework includes two policies: high-level policy and low-level policy, as shown in Figure 3 The high-level policy a differentiable meta parameter generator, and the low- level policy contains N sub-policies, which are in the same form and correspond to N sub-tasks respectively. The high-level policy automatically selects and combines sub- policies of the low-level through the meta parameter generator and interacts with the environment.
In the proposed framework, firstly, shared encoder layer and RNN layer are deployed to leam the environmental state representation 8, from the observation history. Based 8t, high-level meta-
4

---

Under review as a conference paper at ICLR 2021
Environment
Action
Policy
WeightedSum
Reward_1
Reward_N
ALPHAS
Policy_1
Policy_N N
V-Value
WeightedSum
MetaNet
PollcyNet
PolicyNet
ValueNet
Reward
RNN
Encoder
Figure 3: The MeSH framework.
parameter network is established to generate meta-parameters a (01, (01,02, .N)) And N low-level policy networks are established to generate N different sub-policies (T,, T2, ... , nN), respectively. The policy æ ultimately used choose actions the weighted sum of the N policies:
03 n.
When the policy æ chooses the action and interacts with the environment, the environment moves to next state and returns corresponding rewards (Ri, .2,..., Rx). Among them, Rú is the reward for corresponding sub-tasks. The final reward received by the agent weighted sum of the N rewards:
R
Σ a R.
This setting has two advantages: 1) can automatically select the weight of the corresponding policy according to the state of the environment; 2) derive the differentiation of the N policies in an implicit way.
In the training process, we use IMPALA (Espeholt et al. 2018) as the basic framework of large-scale distributed reinforcement learning. Since the process of sample collection and parameter updating are decoupled, learning is off-policy and V-trace technique is used to reduce this difference. The Loss of the framework is
Loss C1 min(clip(p) A,p- + e2 MSE((,, us) + cs Entropy-
where A and us are advontages and target V-value estimated by V-trace.
Due to the complexity of composite tasks, it is usually difficult to obtain positive reward from naive policy. We utilize the idea of self-imitation learning (SIL.) (Oh et al. 2018) to speed up the learning of positive behaviors. Specifically,, the original SIL algorithm has not been adopted. Only those samples whose return exceeds the current value estimation are saved in a special buffer, from which a data is extracted and learned together with the normal samples in every update step.
4.2 META-GRADIENT
We divide the extracted buffer into two parts to calculate the loss to be optimized respectively, which is denoted as Ltrain and Leal. Both losses are determined not only by meta- meta-parameter a but also the parameters of policy networks w. This implies bilevel optimization problem with as the upper-level variable and w as the lower-level variable (Liu et al. 2018):
min Leat (wat(w" (a),
(4)
s.t. (((a) argmin Lsrain (w, a).
w
5

---

Under review as a conference paper at ICLR 2021
Due to the expensive inner optimization of evaluating the gradient exactly, use the approximation scheme as follows:
Va Lal (a), Ro Valraalw - Ve Lerain (w, a),
(5)
where w denotes the current policy weights, and the learning rate for step of inner optimization. The idea is to approximate w" (a) by adapting using only a single training step, without solving the inner optimization completely. Denote w - V Lerain (w, a:), we can approximate (5) by
V Leat (eat(w"(a)).. (a),
V (w a) (-f Vu.a Lerain a)) + V LaLaall
(6)
5. (Va +c - V VaLerain +c,a)) + (aLra((0,a))
V Leat (ws a)
2e
4.3 TRAINING ALGORITIIM
Our hierarchical framework is end-to-end, and the influence high-level on low-level is realized by differentiable meta parameters. Therefore, in the process of forward inference, regard the whole framework as a whole, and no longer emphasize the concept of hierarchy. Only in the process of backward update, need to meta-gradient update the meta parameters Thus will distinguish the different levels of the framework.
Algorithm Meta Soft Hierarchical reinforcement learning framework (MeSH).
1: Initialize parameters w and a.
2: Initialize replay buffer D and SIL buffer Ds-
3: Initialize & <-
4: while True do
5: //Stage 1. Transition Generating Stage.
6:
Sample A, ~ π (AA St, w, w.a).. a).
7:
Generate S,+1, Ry. RN p(S++1, Bu..... RN
8:
Calculate R, by ②
9:
Store (St, A,, St41) Dx-
10:
//Stage 2. Parameter updating stage.
Sample mini-batch of transitions from DN and Ds.
12:
Update w by minimizing B
Compute and accumulate meta-gradien of by (6).
14:
Update SIL buffer Ds-
I = 0(mod e) then
16:
Apply meta-gradient of c.
17:
end
18:
t- ++1
19:
end while
EVALUATION
In this section, we conduct a series experiments in the proposed first-person shooting environment. There questions mainly focus on: 1) how the proposed framework performs compared to representative baselines; and 2) whether the proposed framework can learn different meaningful sub- sub-policies and combine them appropriately.
5.1 EXPERIMENTAL SETUP
In the training process, use IMPALA as the basic framework of large-scale distributed reinforce- ment learning. Four CPU-only machines are used as workers, responsible for interacting with the game environment and collecting the transition sequences. A machine with GPU serves learner, receives the transition sequences transmitted by the worker and updates the parameters. Since the proposed first-person shooter environment typically combination of two sub-tasks (navigation

---

Under review as a conference paper at ICLR 2021
and combat), set the number of sub-policies N 2. The rewards for navigation Rı set as 0.01 per step on the level, and 2.0 per meter on the ramps. The reward for combat R2 is set 10.0 per enemy killed. Thus we hope that to learn the navigation sub-policy and π2 to learn the combat sub-policy. We set the discount factor = 0.997. We use the Adam optimizer to minimize the losses, and the initial learning rate is set to 10-3 with linear decay. The time in- terval for meta- graddentt update is set as = 8. The batch size for normal buffer and SIL buffer 512 and 64 respectively with the sequences' length set 40. All experiments in this work use the same state encoder layer and RNN layer with LSTM units. To ensure the stability of training LSTM in the dynamic environment, we utilize the previous hidden state as initial state introduced in R2D2 (Kapturowski et al. 2018). In addition, SIL buffer is also implemented in all experiments to accelerate the learning speed of behaviors with sparse but large reward.
5.2
PERFORMANCE
5.2.1 BASELINES
To verify the effectiveness of our proposed framework, two types of baselines are chosen for compar- ison. The first is the classical methods in reinforcement learning (a '+' indicates that SIL module added the original method) and the second is the variant the framework show its ablation performance. To construct a fair comparison, we also do parameter-tuning for all baselines.
IMPALA+: A typical distributed reinforcement learning framework with high throughput, scalability, and data-efficiency.
FuNs+: A goal-based hierarchical reinforcement learning framework with abstract goal generated by the high-level agent and guiding the behaviors of low-level agent.
HardHrl: A variant of our proposed framework with parameeteeeer constrained as the form of one-hot encoding, only one sub-policy is executed per step.
5.2.2
PERFORMANCE COMPARISON
Return
Masiimnn
130
IMPALAA
IAPALA+
100
Hardliit
200
Hardiid
M.SH
MUSH
100
50-
4000
6000
S000
2000 6000
3000
Steps
Stops
Figure Average Return in Training.
Figure 5: Maximum Return in Training.
Figure 4 and Figure 5 show the average and maximum return of the collected buffer in the training process. Only the proposed framework MeSH achieves good performance, which can acquire high return in both navigation and combat sub-tasks. Other methods perform poorly. Among them, HardHrl performs slightly better than IMPALA+. From the observation of rollout result, Ha can execute both navigation and combat sub-tasks in lower-level. While IMPALA+ can only execute navigation task with none of enemy killed, which shows that a single policy cannot handle multiple sub-tasks at the same time. FuNs+ has hardly learned any reasonable behavior, which indicates that abstract goal can hardly deal with the complex relationship between multiple sub- tasks simultaneously.
5.3 DISCUSSION ON DEALING WITH COMPLEX RELATIONSHIP
Figure Ohsows a fragment in an episode in the test process. We can observe that when the agent execute navigation sub-task, the value of 02 is small, which indicates that the current behavior is

---

Under review as a conference paper at ICLR 2021
walk
upstairs
shoot (1 ememy)
shoot (3 enemies)
1.0
0.5
0.6
0.4
0.2
0.0
50
100
150
200
250
300
350
Staps
J8
alpha2
Figure 6: Test Procedure. The green dashed line indicates the time step when enemy is killed. We record 02 (solid blue line, the second component of meta meta-parameter o, representing the weight of the combat sub-policy TT) and JS divergence between the two learned sub-policies (solid red line) with the test process.
more influenced by the navigation policy TI5 when the agent execute combat sub-task, the value of 02 increases rapidly and approaches 1.0, which indicates that the current behavior is almost combine different sub-policies appropriately controlled by the combat policy πτ. Therefore, to adapt to complex conditions.
The JS divergence shows the difference between the two sub-policies. We can observe that when a2 is significantly small or large (inclined to one sub-policy), the JS divergence is larger; while close 0.5 (influenced by the two sub-policies equally), the IS divergence is small. Besides, we performed single sub- sub-policy in rollout test. The agent with only executing can move flexibly without shooting any enemy, while the agent with only executing IT₂ can kill the enemies but fall easily. Therefore, the framework has learned different meaningful sub-policies without specify the objectives of each sub-policy artificially.
In addition, we also observed that agent has learned variety of combat policies. The agent tends to shoot at long distances when facing a single enemy. When facing multiple enemies at the same time, the agent are more inclined to close combat. On the one hand, it can avoid being attacked intensively by moving. On the other hand, can get health point packets while fighting to supplement its own consumption, to ensure continuous combat.
CONCLUSION
In order to research practical problems with multi-task combination, implement first-person shooting environment with random terrain, which is a typical representative of such problems. To deal with complex and subtle relationships between multiple sub-tasks, we propose Meta Soft Hier- archical reinforcement learning agent, in which the high-level policy learns to combine the low-level sub-policies end-to-end through meta-gradients. Experiments show that the proposed framework outperforms state-of-art baselines and learns different meaningful sub-policies and combine them appropriately. We provide the open- sourced environment and code encourage further research.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pierre-Lur Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decompo- sition. Journal of artificial intelligence research, 13:227-303, 2000.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint 1 2018.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. arXiv preprint wtXvv:1710009761 2017.
Christopher Grimm and Satinder Singh. Learning independeenty-obbainablle reward functions. arXiv preprint arXiv:1901.0866 2019.
Peter Henderson, Wei-Di Chang, Pierre-Luc Bacon, David Meger, Joelle Pineau, and Doina Precup. Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforce- ment learning. arXiv preprint arXiv: (ttd::/7990663,,, 2017.
Maximilian Igl, Andrew Gambardella, Jinke He, Nantas Nardelli, N Siddharth, Wendelin Böhmer, and Shimon Whiteson. Multitask soft option learning. In Conference on Uncertainty in Artificial Intelligence, pp. 969-978. PMLR, 2020.
Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstrac- tion for hierarchical deep reinforcement learning. In Advances Neural Information Processing Systems, pp. 991111 2019.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney, Recurrent ex- perience replay in distributed reinforcement learning. International conference on learning representations, 2018.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 0ww.000990000022. 2015.
Zichuan Lin, Li Zhao, Derek Yang. Tao Qin, Tie- Yan Liu, and Guangwen Yang. Distributional reward decomposition for reinforcement learning. In Advances Neural Information Processing Systems, pp. 6215-6224, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 10000018, 2018.
Jacob Menashe and Peter Stone. Escape room: a configurable testbed for hicrarchical reinforcement learning. arXiv preprint arXiv:1812.09521,, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. nature. 518(7540):529-533 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928 1988-197,,00 2016.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. arXiv preprint arXiv: 1701v:1706.00044 2017.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lec. Self-imitation learning. arXiv preprint arXiv:18066 05635,56.551 2018
Zhen-Jia Pang, Ruo-Ze Liu, Zhou- Yu Meng, Yi Zhang, Yang Yu, and Tong Lu. On reinforcement learning for full-length game of starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.
2019.

---

Under review as a conference paper at ICLR 2021
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off off-po reta-reiinforemment learning via probabilistic context variables. In International conference on machine learning, pp. 5331-5340, 2019.
Andrew M Saxe, Adam Christopher Earle, and Benjamin Rosman. Hierarchy through composition with multitask Imdps. 2017.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam,, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):8844489) 2016.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv:1703.0116 2017.
Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.0675 2020.
Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, and Jun Zhu. Reward shaping via meta- learning. arXiv preprint 100.93000, 2019.
10


Reviews:
[{"review_id": "ynFpwHSVCze", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new first-person shooting environment consisting of two tasks: navigation and eliminating enemies via shooting. It describes a reinforcement learning architecture to that trains individual policies for each task and automatically balances between those policies by computing weights to mix the action distributions and the rewards.\n\nUnfortunately, this paper leaves significant open questions about the environment and experiments. This makes it hard to judge the reported results. Chiefly, it is not clear what kind of reward should be maximized; the environment provides two rewards (one for each subtask), but Figure 4 and 5 compare algorithms for a single one-dimensional measure of performance. There are further questions around this: what kind of reward is used for the baselines (which are supposedly trained with a single reward)? What kind of reward does the value function for MeSH predict? The mixed reward as determined by the MetaNet? Then: The paper mentions that LSTM networks are used, but of what size? There are no details on complex baselines such as FuN, which are not trivial to implement or tune. Are the returns reported on a fixed set of levels? Figure 6 seems to imply that alpha weights sum up to one, but I couldn't find any description of this. Furthermore, details regarding the action space of the environment are missing.\n\nThe write-up would strongly benefit from some proof-reading to fix grammar and expression. Technical details such as the algorithm for procedural level generation or the observation space could be put in an Appendix.", "extra_scores": {}, "extra_text": {}}, {"review_id": "mS-YKxHrOzf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a meta-learning approach for hierarchical reinforcement learning, essentially meta-learning parametrised weights of a high-level controller. The method is tested on a single, new environment (a first person shooter) against a small set of baselines.\n\nThis review will focus on the method but I personally find that this application with particular emphasis on ‘killing’ (as terminology from the paper), is highly inappropriate and the method could easily be tested on a less problematic domain.\n\nOverall, the paper is clearly written and the additional analysis on JS divergence and alphas in Section 5.3 provides an interesting perspective (which sadly turns into an analysis of semantic aspects of the game in terms of killing and how different amounts of enemies are handled).\n\nA main shortcoming of the submission is the evaluation section. Even ethical concerns aside, additional baselines from the existing HRL literature would be beneficial as e.g. [2]. In addition, another baseline MLSH (Frans et al 2017, referred to quickly in the submission) would be a good baseline. Meta-learning the high-level while applying IMPALA for the low-level (proposed method) is highly related to MLSH which meta-learns the low-level while learning the high-level individually for each task.\n\nOne core argument of the paper surrounds the combination of multiple policies in a single time step vs ‘hard’ switching between policies across time-steps. The combination of objectives for every output is familiar in the multi-objective RL literature [1] as well as other methods in hierarchical RL [2], work that would provide stronger context for the investigation. \n\nFinally, the evaluation focuses on a single task which has been introduced for this paper, strongly limiting the viability of the analysis. Further multi-task domains could be taken from the existing multi-objective/multi-task RL literature as well as via simple extensions of existing domains (imagine a cartpole environment with additional action penalty).\n\nThe final controversial point of this submission is the fact that the weights for combining rewards, which seem to also be used for the overall evaluation of the agent itself (though this remains ambiguous in the submission), are learned by the agent. This could on one hand lead to the agent finding a shortcut and ignoring some given objectives and on the other hand renders these weighted returns irrelevant for comparing different agents as the weights underlying the returns are part of the agent.\n\nOverall, the paper proposes an interesting method but the analysis has ethical problems as well as only a single environment which was created purely for this submission and limited baselines. While the main ideas underlying the method remain interesting, all these problems should be addressed before considering publication.\n\n\n[1] Vamplew, Peter, et al. \"Empirical evaluation methods for multiobjective reinforcement learning algorithms.\" Machine learning 84.1-2 (2011): 51-80.\n[2] Peng, Xue Bin, et al. \"MCP: Learning composable hierarchical control with multiplicative compositional policies.\" Advances in Neural Information Processing Systems. 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uv5MdeTXmDP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First of all I want to point something out I found quite bothersome:\nThe abstract states \" A desirable agent should be capable of balancing between different sub-tasks: navigation to find enemies and shooting to kill them.\"  and the intro begins with \"...it is an urgent need to use DRL methods to solve more complex decision-making problems.\"  I want to state that I strongly believe we should not be framing our research problems with these types of problems, nor trivializing concepts such as killing 'enemies'.  I'll try to be as unbiased in my scientific evaluation of this paper, but I would request that the language be toned down a bit, and ideally other types of tasks considered down the road.\n\nBack to the review:\nThis paper presents a multi-task agent architecture with a final mixture component.  The authors show that this approach can work on a custom-built FPS game better than competing SoTA methods (both multi-task and mono-task agents.  Overall the method uses a bi-level optimization to find the optimal mixture of sub-policies as well as individually optimize each sub-policy.\n\nPros:\nThis seems like a relatively simple architecture and the empirical results are promising, and the analysis of alpha values correlation to sub-tasks is interesting and seems to indicate that the meta-controller does gain some insight into sub-task structure.\n\nCons:\nThere are many confusing points about the paper that made it hard to follow and that I would need clarified to argue for acceptance:\n\n1. Doesn't min(clip(ρ) · A, ρ · A) = clip(ρ) A ?\n2. Doesn't $\\grad \\alpha L train(ω + \\epsilon \u000f, \\alpha) − \\grad \\alpha Ltrain(ω + \\epsilon\u000f, \\alpha) = 0$. Perhaps you meant \\grad w on the second term?\n3. What are L_train and L_val?  I couldn't find a clear definition.\n4.  Do the sub-agents receive the values of the respective R_i or is only $R = \\sum_i \\alpha_i R_i$ passed to the agent?\n5.  I am not familiar with bi-level optimization, it could be worth talking a bit more about this instead of environmental architecture choices which are relatively irrelevant to the core of the paper.\n6. Could you come up with  a task with more than 2 sub-tasks?  I find that 2 is likely a corner case and going beyond just two policies would make the method more convincing.  Trying out on a smaller more synthetic environment that would more easily allow task factorization would also allow for better empirical evaluation.\n7. Although I am not super familiar with the FPS environments for RL, are there some already existant environments that have been independently benchmarked?  Any FPS env has a mixture of navigation and fighting, so I wonder what the value of proposing a new environment for this would be.\n\nConclusion:\nI am not an expert of the multi-task literature but although the proposed idea seems to have merrits, the paper would need to be more clear on the points above for me to consider it clean enough for publication.  As it stands the approach is too opaque to really understand what is going on.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3N2W64g2nl3", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers a FPS game that can be decomposed into two sub-tasks, navigation and shooting. A hierarchical meta RL method is introduced and the updating rules for sub-policies and meta parameters are provided. Experiments focus on this specific environment and hence the hierarchical structure is also specified as a meta controller over two sub-policies defined for navigation and shooting explicitly.\n\nThe proposed hierarchical RL method is not novel and indeed a most straightforward way to control sub-policies with a meta controller. The number of sub-policies, each of which is specified to solve an explicit sub-task, is fixed given the environment and the rewarding scheme is also clear that each sub-task has its own reward. The final policy is simply presented as linear combination of sub-policies. Actually, similar studies has been well studied in the literature, like Feudal networks and MAML, which are even more general meta learning methods to automatically find sub-policies. These highly related approaches are ignored from the discussion and not considered in the experiments as a baseline.\n\nAnother question is about the considered global reward, which is also formulated as linear combination of the sub-tasks' reward using the meta parameters \\alpha_i's. Under such a formulation, the rewarding scheme is dynamically varying as the training goes. One concern is that in IMPALA the estimated values in vtrace or advantage might suffer large variance since at early stages the meta parameters are almost from scratch. Another concern is that this rewarding scheme naturally restricts the meta controller to select sub-policy with high immediate R_i at a specific time step, and hence it seems the meta parameters are almost determined by the rewarding scheme.\n\nExperimental parts lack comparison of many related works as mentioned above. Moreover, only one specific environment is studied with only two sub-tasks. It is hard to see the generality of the method when scaling to cases where a large number of sub-tasks exist.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that the paper presents interesting ideas but the presentation of the paper needs be improved. Also, the experiments and the related work section need be improved.
Decision: Reject