Feudal Reinforcement Learning by Reading Manuals

1 INTRODUCTION . Recently , there are increasing interests in building reinforcement learning ( RL ) agents that interact with humans via natural language , such as follow natural language instructions and complete goals specified in natural language . The successes of these studies will boost the user experience in a wide range of real-world applications , such as visual language navigation ( Anderson et al. , 2018 ; Wang et al. , 2019b ) , interactive games ( Gray et al. , 2019 ) , robot control ( Tellex et al. , 2020 ) , goal-oriented dialog systems and other personal assistant applications ( Dhingra et al. , 2017 ) . In order to generalize to real-world use cases , the research of RL with language instructions faces various kinds of complexity . One critical demand of these use cases is that humans tend to give concise instructions , which specify the goals they hope to achieve , instead of providing complete information for the intermediate steps . For example , users would prefer to ask a personal assistant “ share my recent photo to my parents ” . But if the agent only works when specified the unusually complex instruction “ open the photo albums , select the recent one and click on sharing , then click Messages and select the contacts named mom and dad ” , it will significantly lower the users ’ satisfaction . To deal with this challenge , it is important for an agent to reason a procedure to accomplish the goal , with the latent unspecified information supplemented with necessary prior knowledge about the environment . One natural and realistic source of such knowledge is the textual descriptions about the environment and its dynamics , e.g. , the textual manual of the environment . This setting gives rise to a new problem of reading to act , where the agents require to making actions via comprehension of both the manuals and the environments , then bridging them together . Recently , this direction draws increasing attention with new benchmarks established ( Branavan et al. , 2012 ; Narasimhan et al. , 2017 ; Zhong et al. , 2020 ; Wang & Narasimhan , 2021 ) . While progresses have been made in the field of reading to act , there is one fundamental challenge not studied much , i.e. , the semantic mismatch between the low-level actions and states agents have access to and the high-level language descriptions in the instructions and manuals . Specifically , in the example shown in Figure 1 , it is easy for human players to derive a sequence of steps to complete the goal , when they have access the game and the manual . These human rationales can be logically derived from the texts in the manuals , with necessary knowledge about the game setting . Therefore , it is feasible to have a natural language processing ( NLP ) model to learn the reasoning process of manual reading , by imitating the human players . However , the results from this reasoning process mainly specify classes of target states an agent should arrive at in a higher level , but do not correspond to any specific action an agent can take . As a result , there exists a gap between the reasoning process in the manual reading and the actual operations a game agent could perform . The aforementioned semantic mismatch introduces difficult to the reading-to-act agent design : To make agent aware of the high-level textual information , existing studies ( Zhong et al. , 2020 ; Wang & Narasimhan , 2021 ) feed the instructions and manuals to the agent at each step . The agent policy models then rely on the cross-attention between the texts and the environment states to achieve a textenhance state representation , from which an action is predicted . However , like the human rationales we show earlier , the strategies derived from the texts usually can not directly map to the action space . With the existence of this semantic mismatch , the agents map textual form of knowledge to actions in an indirect and implicit way , thus are not powerful and efficient enough to handle the reading-to-act tasks . We deal with this semantic mismatch challenge in two folds . First , we propose a feudal reinforcement learning ( FRL ) framework to handle the semantic mismatch . The framework consists of two agents . The manager agent works on the high-level abstract information from the texts , and specifies a strategy , i.e. , a sequence of sub-goals , to achieve the final goal . In other words , the manager agent reads the instructions and manuals to propose a plan . Then another worker agent achieves the sub-goals in a plan one-by-one . The worker works with the low-level perceptions and actions in the environments , and is rewarded according to whether it accomplishes the sub-goals specified by the manager . Second , we equip the manager agent with a multi-hop reader model , so that it can better generate the high-level plans when the goal require multiple steps of reasoning to achieve . This enhanced manager architecture , named the multi-hop plan generator , aims to address the challenging scenarios where the sub-goals need to be inferred conditioned on both ( 1 ) multiple pieces of the textual knowledge and ( 2 ) the relation between the history sub-goal sequence and the final goal . Specifically , the multi-hop generator predicts the sub-goals in a backward manner – starting from the final goal , it sequentially predicts sub-goals , with each one enables its next sub-goal based on the text knowledge . Like the example in Figure 1 , with the ghost generated as a sub-goal that relates to the target army , the model should reason to get a weapon that can defeat the ghost as another sub-goal , so that completing both sub-goals fulfills the end goal . Our FRL framework effectively alleviated the mismatch between high-level textual information and low-level perceptions and actions ; and is general to various forms of environments , instructions and manuals . Our multi-hop plan generator , on the other hand , can significantly boost the FRL framework ’ s ability to deal with challenging tasks where reasoning over the texts are critical to resolving the instructed goals . We verify our approach mainly on the challenging benchmark RTFM ( Zhong et al. , 2020 ) and verify the generalization ability on the benchmark Messenger ( Wang & Narasimhan , 2021 ) . On both benchmarks , prior methods are not able to generate non-trivial results with straightforward end-to-end training . Some limited progresses have be made when these methods are trained with human-designed curriculum . However , the design of the curriculum makes the assumption of white-box environments , i.e. , the curriculum requires modifications on the environment simulators to simpler versions . In comparison , our proposed approach achieves good performance with no need for curriculum learning with simulator modifications , giving a more powerful and more realistic solution . Specifically , our approach achieves near perfect results on the RTFM benchmark , leading to > 60 % and > 40 % absolute win rate improvement compared to the previous best numbers without and with curriculum learning , respectively . 2 METHODOLOGY . Feudal Reinforcement Learning ( FRL ) ( Dayan & Hinton , 1993 ) proposes a managerial hierarchy that dissects a holistic task into multi-level sub-tasks to speed up the reinforcement learning process . It creates multi-level manager-workers where high-level managers set tasks for the low-level workers and , in return , the low-level workers aim to accomplish the tasks set by the high-level managers . In conjunction with deep learning ( Goodfellow et al. , 2016 ) , various FRL methods ( Vezhnevets et al. , 2017 ; Nachum et al. , 2018 ; Casanueva et al. , 2018 ) achieve breakthroughs on specific domains such as video games , dialogue management and robotics etc .. In our case , we aim to solve a more complicated multi-domain task – our agent is required to comprehend the linguistic guidance and act properly in a visual environment to achieve a goal . Observing that the agent ’ s actions subordinate to the language guidance , we propose a hierarchical manager-worker framework . As shown in Figure 1 , the high-level manager translates the linguistic information into a series of sub-goals , and the low-level worker tries to fulfill the sub-goals by making specific actions in the visual environment . 2.1 TASK SETTINGS . We evaluate our model on RTFM ( Zhong et al. , 2020 ) task . In RTFM , the agent is given a document of environment dynamics , observations of the environment , and an underspecified goal instruction . The document includes information about which monsters belong to which team , which modifiers are effective against which element . The goal indicates which team the player should defeat . Both of the document and the goal are generated from human-written templates . The environment is represented as a matrix of text in which each cell describes the entity in the cell . When the player is in the same cell with an item or a monster , the player will pick up the item or engage in combat with the monster . The player can carry only one item at a time . When encountering a new item , the player will pick up the new item and lose the previous item forever . A monster moves towards the agent with 60 % probability , and moves randomly otherwise . In a full RTFM task , there will be two monsters and two items as the target and the distractor respectively . To accomplish the task , there are multiple reasoning and action steps to do : ( 1 ) identify the target team from the goal ; ( 2 ) identify which monster in the environment belongs to the target team , and its element ; ( 3 ) identify the modifiers effectively against the element of the target monster ; ( 4 ) identify which item in the environment has the desired modifier ; ( 5 ) pick up the target item ; ( 6 ) beat the target monster . If the agent fails to carry the target item or engages a combat with the distractor monster , it will lose the game . The agent receives a reward of +1 if it wins and −1 otherwise . Furthermore , a recent work proposed a related reading-to-act environment named Messenger ( Wang & Narasimhan , 2021 ) in concurrent to our work of this paper . For completeness , we conduct experiments on the Messenger task and report the results in Appendix A . 2.2 FEDUAL REINFORCEMENT LEARNING FORMULATION . In the fedual reinforcement learning ( FRL ) formulation , both the manager and the worker subject to Markov Decision Processes ( MDP ) . The two agents differ in the spaces of decision making : the manager generates sequential targets based on language inputs ; the worker generates a series of control signals to drive the agent in the environment , only if they cooperate can the model attain the final goal . Specifically , the manager strives to generate an accurate target and pass it to the worker while the worker maximizes its capability to reach that target . In our method , the states of the manager consists of the input manual , the goal instruction , and the history sub-goals , and the action of the manager is a new generated sub-goal , which is an object in the game environment . The worker agent only sees the sub-goal from the manager and the lower level game environment without the manual . Besides , the sub-goal from the manager is transformed to the position of the sub-goal object in each time step . Therefore , in each step , the state of the worker consists of the current game state plus an indicator of the sub-goal position . Rewards For the worker , the problem setting is analogous to the Maze problem ( Barto et al. , 1989 ) . Given the target object and the current state , the agent is expected to make actions to move toward the target . If it finally touches all the targets generated by the manager , the sequence of actions will be rewarded ; if it touches a wrong object or dies , the sequence of actions will be penalized . For the manager , the generation of the next target is based on the word description and the current target . Its RL loss is from the final reward of the game . In the game setting , we judge whether the agent successfully achieves or fails the goal instruction according to the true rewards returned by the environment . The sequence of generated sub-goals will be rewarded for the prior case and be penalized for the latter one . We can plug in the Bellman equation ( Baird , 1995 ) and follow a typical policy learning schedule ( Zhong et al. , 2020 ) to optimize this manager-worker system .

Summary: The paper presents a hierarchical RL model to solve tasks that require reading and understanding instructions or manuals to solve complex goals (these goals usually require multiple steps of solving subgoals). The proposed model has two parts: a manager that uses the instructions/ manuals to design subgoals, and a worker that solves the subgoals and deals with low-level perceptions. The manager and the workers are trained separately. The Feudal model achieves SOTA scores on Messenger and RTFM without the use of a hand-designed curriculum.