Feudal Reinforcement Learning by Reading Manuals

1 INTRODUCTION . Recently , there are increasing interests in building reinforcement learning ( RL ) agents that interact with humans via natural language , such as follow natural language instructions and complete goals specified in natural language . The successes of these studies will boost the user experience in a wide range of real-world applications , such as visual language navigation ( Anderson et al. , 2018 ; Wang et al. , 2019b ) , interactive games ( Gray et al. , 2019 ) , robot control ( Tellex et al. , 2020 ) , goal-oriented dialog systems and other personal assistant applications ( Dhingra et al. , 2017 ) . In order to generalize to real-world use cases , the research of RL with language instructions faces various kinds of complexity . One critical demand of these use cases is that humans tend to give concise instructions , which specify the goals they hope to achieve , instead of providing complete information for the intermediate steps . For example , users would prefer to ask a personal assistant “ share my recent photo to my parents ” . But if the agent only works when specified the unusually complex instruction “ open the photo albums , select the recent one and click on sharing , then click Messages and select the contacts named mom and dad ” , it will significantly lower the users ’ satisfaction . To deal with this challenge , it is important for an agent to reason a procedure to accomplish the goal , with the latent unspecified information supplemented with necessary prior knowledge about the environment . One natural and realistic source of such knowledge is the textual descriptions about the environment and its dynamics , e.g. , the textual manual of the environment . This setting gives rise to a new problem of reading to act , where the agents require to making actions via comprehension of both the manuals and the environments , then bridging them together . Recently , this direction draws increasing attention with new benchmarks established ( Branavan et al. , 2012 ; Narasimhan et al. , 2017 ; Zhong et al. , 2020 ; Wang & Narasimhan , 2021 ) . While progresses have been made in the field of reading to act , there is one fundamental challenge not studied much , i.e. , the semantic mismatch between the low-level actions and states agents have access to and the high-level language descriptions in the instructions and manuals . Specifically , in the example shown in Figure 1 , it is easy for human players to derive a sequence of steps to complete the goal , when they have access the game and the manual . These human rationales can be logically derived from the texts in the manuals , with necessary knowledge about the game setting . Therefore , it is feasible to have a natural language processing ( NLP ) model to learn the reasoning process of manual reading , by imitating the human players . However , the results from this reasoning process mainly specify classes of target states an agent should arrive at in a higher level , but do not correspond to any specific action an agent can take . As a result , there exists a gap between the reasoning process in the manual reading and the actual operations a game agent could perform . The aforementioned semantic mismatch introduces difficult to the reading-to-act agent design : To make agent aware of the high-level textual information , existing studies ( Zhong et al. , 2020 ; Wang & Narasimhan , 2021 ) feed the instructions and manuals to the agent at each step . The agent policy models then rely on the cross-attention between the texts and the environment states to achieve a textenhance state representation , from which an action is predicted . However , like the human rationales we show earlier , the strategies derived from the texts usually can not directly map to the action space . With the existence of this semantic mismatch , the agents map textual form of knowledge to actions in an indirect and implicit way , thus are not powerful and efficient enough to handle the reading-to-act tasks . We deal with this semantic mismatch challenge in two folds . First , we propose a feudal reinforcement learning ( FRL ) framework to handle the semantic mismatch . The framework consists of two agents . The manager agent works on the high-level abstract information from the texts , and specifies a strategy , i.e. , a sequence of sub-goals , to achieve the final goal . In other words , the manager agent reads the instructions and manuals to propose a plan . Then another worker agent achieves the sub-goals in a plan one-by-one . The worker works with the low-level perceptions and actions in the environments , and is rewarded according to whether it accomplishes the sub-goals specified by the manager . Second , we equip the manager agent with a multi-hop reader model , so that it can better generate the high-level plans when the goal require multiple steps of reasoning to achieve . This enhanced manager architecture , named the multi-hop plan generator , aims to address the challenging scenarios where the sub-goals need to be inferred conditioned on both ( 1 ) multiple pieces of the textual knowledge and ( 2 ) the relation between the history sub-goal sequence and the final goal . Specifically , the multi-hop generator predicts the sub-goals in a backward manner – starting from the final goal , it sequentially predicts sub-goals , with each one enables its next sub-goal based on the text knowledge . Like the example in Figure 1 , with the ghost generated as a sub-goal that relates to the target army , the model should reason to get a weapon that can defeat the ghost as another sub-goal , so that completing both sub-goals fulfills the end goal . Our FRL framework effectively alleviated the mismatch between high-level textual information and low-level perceptions and actions ; and is general to various forms of environments , instructions and manuals . Our multi-hop plan generator , on the other hand , can significantly boost the FRL framework ’ s ability to deal with challenging tasks where reasoning over the texts are critical to resolving the instructed goals . We verify our approach mainly on the challenging benchmark RTFM ( Zhong et al. , 2020 ) and verify the generalization ability on the benchmark Messenger ( Wang & Narasimhan , 2021 ) . On both benchmarks , prior methods are not able to generate non-trivial results with straightforward end-to-end training . Some limited progresses have be made when these methods are trained with human-designed curriculum . However , the design of the curriculum makes the assumption of white-box environments , i.e. , the curriculum requires modifications on the environment simulators to simpler versions . In comparison , our proposed approach achieves good performance with no need for curriculum learning with simulator modifications , giving a more powerful and more realistic solution . Specifically , our approach achieves near perfect results on the RTFM benchmark , leading to > 60 % and > 40 % absolute win rate improvement compared to the previous best numbers without and with curriculum learning , respectively . 2 METHODOLOGY . Feudal Reinforcement Learning ( FRL ) ( Dayan & Hinton , 1993 ) proposes a managerial hierarchy that dissects a holistic task into multi-level sub-tasks to speed up the reinforcement learning process . It creates multi-level manager-workers where high-level managers set tasks for the low-level workers and , in return , the low-level workers aim to accomplish the tasks set by the high-level managers . In conjunction with deep learning ( Goodfellow et al. , 2016 ) , various FRL methods ( Vezhnevets et al. , 2017 ; Nachum et al. , 2018 ; Casanueva et al. , 2018 ) achieve breakthroughs on specific domains such as video games , dialogue management and robotics etc .. In our case , we aim to solve a more complicated multi-domain task – our agent is required to comprehend the linguistic guidance and act properly in a visual environment to achieve a goal . Observing that the agent ’ s actions subordinate to the language guidance , we propose a hierarchical manager-worker framework . As shown in Figure 1 , the high-level manager translates the linguistic information into a series of sub-goals , and the low-level worker tries to fulfill the sub-goals by making specific actions in the visual environment . 2.1 TASK SETTINGS . We evaluate our model on RTFM ( Zhong et al. , 2020 ) task . In RTFM , the agent is given a document of environment dynamics , observations of the environment , and an underspecified goal instruction . The document includes information about which monsters belong to which team , which modifiers are effective against which element . The goal indicates which team the player should defeat . Both of the document and the goal are generated from human-written templates . The environment is represented as a matrix of text in which each cell describes the entity in the cell . When the player is in the same cell with an item or a monster , the player will pick up the item or engage in combat with the monster . The player can carry only one item at a time . When encountering a new item , the player will pick up the new item and lose the previous item forever . A monster moves towards the agent with 60 % probability , and moves randomly otherwise . In a full RTFM task , there will be two monsters and two items as the target and the distractor respectively . To accomplish the task , there are multiple reasoning and action steps to do : ( 1 ) identify the target team from the goal ; ( 2 ) identify which monster in the environment belongs to the target team , and its element ; ( 3 ) identify the modifiers effectively against the element of the target monster ; ( 4 ) identify which item in the environment has the desired modifier ; ( 5 ) pick up the target item ; ( 6 ) beat the target monster . If the agent fails to carry the target item or engages a combat with the distractor monster , it will lose the game . The agent receives a reward of +1 if it wins and −1 otherwise . Furthermore , a recent work proposed a related reading-to-act environment named Messenger ( Wang & Narasimhan , 2021 ) in concurrent to our work of this paper . For completeness , we conduct experiments on the Messenger task and report the results in Appendix A . 2.2 FEDUAL REINFORCEMENT LEARNING FORMULATION . In the fedual reinforcement learning ( FRL ) formulation , both the manager and the worker subject to Markov Decision Processes ( MDP ) . The two agents differ in the spaces of decision making : the manager generates sequential targets based on language inputs ; the worker generates a series of control signals to drive the agent in the environment , only if they cooperate can the model attain the final goal . Specifically , the manager strives to generate an accurate target and pass it to the worker while the worker maximizes its capability to reach that target . In our method , the states of the manager consists of the input manual , the goal instruction , and the history sub-goals , and the action of the manager is a new generated sub-goal , which is an object in the game environment . The worker agent only sees the sub-goal from the manager and the lower level game environment without the manual . Besides , the sub-goal from the manager is transformed to the position of the sub-goal object in each time step . Therefore , in each step , the state of the worker consists of the current game state plus an indicator of the sub-goal position . Rewards For the worker , the problem setting is analogous to the Maze problem ( Barto et al. , 1989 ) . Given the target object and the current state , the agent is expected to make actions to move toward the target . If it finally touches all the targets generated by the manager , the sequence of actions will be rewarded ; if it touches a wrong object or dies , the sequence of actions will be penalized . For the manager , the generation of the next target is based on the word description and the current target . Its RL loss is from the final reward of the game . In the game setting , we judge whether the agent successfully achieves or fails the goal instruction according to the true rewards returned by the environment . The sequence of generated sub-goals will be rewarded for the prior case and be penalized for the latter one . We can plug in the Bellman equation ( Baird , 1995 ) and follow a typical policy learning schedule ( Zhong et al. , 2020 ) to optimize this manager-worker system .

Summary: In this paper, the authors propose a Feudal Reinforcement Learning (FRL) algorithm for improving the miss-match between high-level natural language commands (e.g., "share my recent photo to my parents") and the actual complex set of operations required by the systems (e.g., open the album --> select recent photo --> etc.). FRL (Dayan & Hinton, 1993) is a hierarchical RL algorithm with two agents: manager and worker. The manager issues a plan which is executed by the worker. To elaborate, the manager learns to generate a sequence of targets given $Q$ the goal description, $O$ the wiki paragraph which describes the game, $A$ the object names in the environment, and $H$ the subgoal history ($h_t$ refers to the target object at time $t$). The worker, instead, uses the plan from the manager (in terms of coordinate to the target ($X_target$)), the observation $E_{obs}$, and the position to the other player ($X_{pos}$) (not sure I understood correctly this) to interact with the environment.   The manager and the worker are trained separately using imitation learning (also here it is a bit confusing), from a 100K trajectory collected by a random walk in the map.   The authors benchmarked the FRL with two text-based interactive games such as RTFM (Zhong et al. 2020) and Messager (Wang & Narasimhan, et al 2021) (the results are only in the appendix, but this paper is very recent). The results show that FRL is better than txt2$\pi$ without curriculum learning (here I am a bit unsure because the results in the original paper are different), in the RTFM.