Feudal Reinforcement Learning by Reading Manuals
Authors: Kai Wang
Zhonghao Wang
Mo Yu
Humphrey Shi
Venue: ICLR 2022 Submitted (2022)
Keywords: feudal reinforcement learning
textual instruction following
reading to act
text games
multi-hop reasoning

Abstract: Reading to act is a prevalent but challenging task that requires the ability to follow a concise language instruction in an environment, with the help of textual knowledge about the environment. Previous works face the semantic mismatch between the low-level actions and the high-level language descriptions and require the human-designed curriculum to work properly. In this paper, we present a Feudal Reinforcement Learning (FRL) model consisting of a manager agent and a worker agent. The manager agent is a multi-hop planner, which deals with high-level abstract information and generates a series of sub-goals. The worker agent deals with the low-level perceptions and actions to achieve the sub-goals one by one. Our FRL framework effectively alleviates the mismatching between the text-level inference and the low-level perceptions and actions; and is general to various forms of environments, instructions and manuals. Our multi-hop planner contributes to the framework by further boosting the challenging tasks where multi-step reasoning from the texts is critical to achieving the instructed goals. We showcase our approach achieves competitive performance on two challenging tasks, Read to Fight Monsters (RTFM) and Messenger, without human-designed curriculum learning.

Under review as a conference paper at ICLR 2022
FEUDAL REINFORCEMENT LEARNING BY READING MANUALS
Anonymous authors Paper under double-blind review
ABSTRACT
Reading to act is a prevalent but challenging task which requires the ability to rea- son from a concise instruction. However, previous works face the semantic mis- match between the low-level actions and the high-level language descriptions and require the human-designed curriculum to work properly. In this paper, we present Feudal Reinforcement Learning (FRL) model consisting of manager agent and a worker agent. The manager agent is a multi-hop plan generator dealing with high-level abstract information and generating a series of sub-goals in a backward manner. The worker agent deals with the low-level perceptions and actions to achieve the sub-goals one by one. comparison, our FRI. model effectively alle- viate the mismatching between text-level inference and low-level perceptions and actions; and is general to various forms of environments, instructions and manu- als; and our multi-ho hopp plan generator significantly boost for challenging tasks where multi-step reasoning form the texts is critical to resolve the instructed goals. We showcase our approach achieves competitive performance on challenging tasks, Read to Fight Monsters (RTFM) and Messenger, without human-designed curriculum learning.
INTRODUCTION
Recently, there are increasing interests in building reinforcement learning (RL) agents that interact with humans via natural language, such as follow natural language instructions and complete goals specified in natural language. The successes of these studies will boost the user experience in a wide range of weal-world applications, such as visual language navigation (Anderson et al.) 2018: Wang et al. (2019)), interactive games (Gray al.) 2019). robot control (Tellex al. (020), goal-oriented dialog systems and other personal assistant applications (Dhingra et al. 2017).
In order to generalize to real-world cases, the research of RL with language instructions faces various kinds of complexity. One critical demand of these use cases is that humans tend to give concise instructions, which specify the goals they hope to achieve, instead of providing complete information for the intermediate steps. For example, users would prefer to ask a personal assistant "share my recent photo to my parents". But if the agent only works when specified the unusu- ally complex instruction "open the photo albums, select the recent one and click on sharing, then click Messages and select the contacts named mom and dad", it will significantly lower the users' satisfaction.
To deal with this challenge, it is important for agent reason a procedure accomplish the goal, with the latent unspecified information supplemented with necessary prior knowledge about the environment. One natural and realistic source of such knowledge the textual descriptions about the environment and its dynamics, e.g., the textual manual of the environment. This setting gives rise to new problem of reading to act, where the agents require to making actions via comprehension of both the manuals and the environments, then bridging them together. Recently, this direction draws increasing attention with new benchmarks established (Branavan et al. 2012; Narasimhan et al. 2017 Zhong et al. 2020 Wang & Narasimhan 2021).
While progresses have been made in the field of reading to act, there one fundamental challenge not studied much, i.c., the semantic mismatch between the low-level actions and states agents have access to and the high-level language descriptions in the instructions and manuals. Specifically, in the example shown in Figure 1. it is easy for human players derive sequence of steps to

---

Under review as a conference paper at ICLR 2022
Wike:
Inidal state
You dhould unc arcasc and mystarioucilees bes poisor mossterss Pazther goblin and welf are froms the Outer the Forest alessed and Clancraser's itee beat lighting munseen Cold not good sqairal firatica and dhimmein ing weapons Suar tmp. jegur, lyns up jeckel, and glont. Objecte:
Humen retionale
1. Gbost belongso Star Allianee Ghost is prison mussstr
cold Japar
Messed ove
cold Jppar
the
Arzzn is kagainnr poison menatoe
suapara
cuca wespos
Goal:
polaot cold jagees, cutlais, blessd
pelsonghhot
polscm ghool
Lifeor the Stor Allianee
Worker
Gool 1
arcane cutlass
Text encoder (RNN)
poison ghost
ATTO
Worker
Manager
7840
bleceed Jae
Mesiee
piince
WIN
1530
odd Japar
Manager
ond Ope Houeeiae
The overall pipeline of the FRL model. Given a document about the environment dy- namics, a goal description, and the observation of the initial environment, the manager module will generate a plan to reach the goal in a backwards manner. The worker will fulfill the sub-goals of the plan one by one according to the observation of the environment and the sub-goal. The red arrows and the black arrows represent the data flow with and without gradient propagation respectively.
Figure 1:
complete the goal, when they have access the game and the manual. These human rationales can be logically derived from the texts in the manuals, with necessary knowledge about the game setting. Therefore, it is feasible to have a natural language processing (NLP) model to learn the reasoning process of manual reading, by imitating the human players. However, the results from this reasoning process mainly specify classes of target states an agent should arrive at in a higher level, but do not correspond to any specific action an agent can take. As a result, there exists a gap between the reasoning process in the manual reading and the actual operations game agent could perform.
The aforementioned semantic mismatch introduces difficult to the reading-to-ace agent design: To make agent aware of the high-level textual information, existing studies (Zhong et al. 2020 Wang & Narasimhan (02T) feed the instructions and manuals to the agent at each step. The agent policy models then rely on the cross-attention between the texts and the environment states to achieve text- enhance state representation, from which an action is predicted. However, like the human rationales show earlier, the strategies derived from the texts usually cannot directly map to the action space. With the existence of this semantic mismatch, the agents map textual form of knowledge to actions in an indirect and implicit way, thus are not powerful and efficient enough to handle the reading-to-ace tasks.
We deal with this semantic mismatch challenge in two folds. First, propose feudal reinforce- ment learning (FRL) framework to handle the semantic mismatch. The framework consists of two agents. The manager agent works on the high-level abstract information from the texts, and spec- ifies a strategy, i.e., a sequence of sub-goals, to achieve the final goal. In other words, the manager agent reads the instructions and manuals to propose a plan. Then another worker agent achieves the in plan one-by-one. The worker works with the low-level perceptions and actions in the environments, and is rewarded according to whether it accomplishes the sub-goals specified by the manager. Second, we equip the manager agent with multi-hop reader model, so that it can better generate the high-level plans when the goal require multiple steps of reasoning to achieve. This enhanced manager architecture, named the multi-hop plan generator, aims to address the chal- lenging scenarios where the sub-goals need to be inferred conditioned on both (1) multiple pieces of the textual knowledge and (2) the relation between the history sub-goal sequence and the final goal. 1
Specifically, the multi-hop generator predicts the sub-goals in a backward manner starting from the final goal, it sequentially predicts sub-goals, with each one enables its next sub-goal based on
2

---

Under review as a conference paper at ICLR 2022
the text knowledge. Like the example in Figure with the ghost generated a sub-goal that relates to the target army, the model should reason to get a weapon that can defeat the ghost as another sub-goal, that completing both sub-goals fulfills the end goal.
Our FRL framework effectively alleviated the mismatch between high-level textual information and low-level perceptions and actions; and is general to various forms of environments, instructions and manuals. Our multi-hop plan generator, on the other hand, can significantly boost the FRL frame- work's ability deal with challenging tasks where reasoning over the texts are critical resolving the instructed goals. We verify our approach mainly on the challenging benchmark RTFM (Zhong al. 2020) and verify the generalization ability on the benchmark Messenger (Wang & Narasimhan, 2021). On both benchmarks, prior methods are not able to generate non-trivial results with straight- forward end-to-end training. Some limited progresses have be made when these methods are trained with human-designed curriculum. However, the design of the curriculum makes the assumption of white-box environments, i.e., the currieulum requires modifications on the environment simulators to simpler versions. In comparison, for curriculum learning with simulator modifications, giving a more powerful and more realistic so- lution. Specifically, our approach achieves near perfect results on the RTFM benchmark, leading to > 60% and > 40% absolute win rate improvement compared to the previous best numbers without and with curriculum learning, respectively.
proposed approach achieves good performance with no need
2 METHODOLOGY
Feudal Reinforcement Learning (FRL) (Dayan & Hinton, 1993) proposes a managerial hierarchy that dissects holistic task into multi-level sub-tasks to speed up the reinforcement learning process. creates multi-level markger-workers where high-level managers set tasks for the low-level workers and, in return, the low-level workers aim to accomplish the tasks set by the high-level managers. In conjunction with deep learning (Goodfellow et al. 2016), various FRL methods (Vezhnevets et al,, 2017 Nachum et al. 2018 Casanueva et al. 2018) achieve breakthroughs such as video games, dialogue management and robotics etc.. In our case, we aim to solve a more complicated multi-domain task our agent is required to comprehend the linguistic guidance and act properly in a visual environment to achieve a goal. Observing that the agent's 's actions subordinate to the language guidance, we propose hierarchical wooreerrwoorker framework. As shown in Figure I the high-level manager translates the linguistic information into a series of sub-goals, and the low-level worker tries fulfill the sub-goals by making specific actions in the visual environment.
specific domains
2.1 TASK SETTINGS
We evaluate our model on RTFM (Zhong et al. 2020) task. In RTFM, the agent is given document of environment dynamics, observations of the environment, and an underspecified goal instruction. The document includes information about which monsters belong which team, which modifiers are effective against which element. The goal indicates which team the player should defeat. Both of the document and the goal are generated from numan-writter templates. The environment is represented as a matrix of text in which each cell describes the entity in the cell. When the player monster, the player will pick up the item or engage in combat
in the same cell with an item with the monster. The player can carry only one item at time. When encountering a new item, the player will pick up the new item and lose the previous item forever. A monster moves towards the agent with 60% probability, and moves randomly otherwise. In a full RTFM task, there will be two monsters and two items as the target and the distractor respectively. To accomplish the task, there are multiple reasoning and action steps to do: (1) identify the target team from the goal; (2) identify which monster in the environment belongs to the target team, and its element; (3) identify the modifiers efffcctivey against the element of the target monster; (4) identify which item in the environment has the desired modifier; (5) pick up the target item; (6) beat the target monster. If the agent fails to carry the target item or engages a combat with the distractor monster, it will lose the game. The agent receives a reward of +1 if it wins and -1 otherwise.
Furthermore, recent work proposed related reading-to to- environment named Messenger (Wang & Narasimhan 202T) in concurrent to our work of this paper. For completeness, we conduct exper- iments the Messenger task and report the results in Appendix A
3

---

Under review as a conference paper at ICLR 2022
2.2 FEDUAL REINFORCEMENT LEARNING FORMULATION
In the fedual reinforcement learning (FRL) formulation, both the manager and the worker subject Markov Decision Processes (MDP). The agents differ in the spaces of decision making: the manager generates sequential targets based on language inputs; the worker generates a series of control signals drive the agent the environment, only they cooperate can the model attain the final goal.
Specifically, the manager strives to generate an accurate target and pass it to the worker while the worker maximizes its capability to reach that target. In our method, the states of the manager consists of the input manual, the goal instruction, and the history sub- sub-goals, and the action of the manager is new generated sub-goal, which is an object in the game environment. The worker agent only sees the sub-goal from the manager and the lower level game environment without the sub-goal from the manager is transformed to the position of the sub-goal object in each time step. Therefore, in each step, the state of the worker consists of the current game state plus an indicator of the sub-goal position.
manual, Besides,
Rewards For worker, the problem setting analogous to the Maze problem (Barto et al, 1989). Given the target object and the current state, the agent is expected to make actions to move toward the target. finally touches the targets generated by the manager, the sequence of ac- tions will be rewarded; it touches wrong object dies, the sequence of actions will be penalized.
For the manager, the generation of the next target is based on the word description and the current target. Its RL loss is from the final reward of the game. In the game setting, judge whether the agent successfully achieves or fails the goal instruction according to the true rewards returned by the environment. The sequence of generated sub-goals will be rewarded for the prior case and be penalized for the latter one.
We can plug in the Bellman equation (Baird 1995) and follow a typical policy learning schedule (Zhong et al. (020) to optimize this manager- worker system.
2.3
WORKER
X target
Palicy
policy
MLP
oobs
concat
Conv
concat
Conv
Conv
MaxPool
X
pos
Baseline
MLP
baseline
Figure The structure of the worker agent.
The worker interacts with the observation from the environment and the sub-goal from the manager. In this case of a textual environment, we treat the grid of word embeddings as the visual features for the worker.
We use residual CNN model as our backbone for the worker. The worker consists of 5 convolution layers, as shown in Figure 2 There a residual connection from the third layer to the fifth layer. Let E,bs denote word embedding corresponding to the observations from the environment, where (o,,(,,;, represents the embeddings corresponding to the woord string that describes the objects in location (i,j) in the grid world. For each cell, the positional feature Xpos consists of the x and distance from the cell to the player respectively, normalized by the width and height of the grid-world. The target feature Xtarget is the æ and coordinates of the sub-goal target in the grid world. The input to each layer consists of the output from the previous layer, concatenated with positional features. For the ith layer, we have
R (V(i-)) X Xpos]
( = Conv (1)
(2)
For we concatenate the bfg-of-words embeddings with the target feature as the initial visual
features (0) EE Eohs, Xtarget). We do MaxPool ((ast)))((lst)) over the spacial dimension
4

---

Under review as a conference paper at ICLR 2022
and compute the policy ypolicy and Ybaseline
Y ypolicy MLP policy (v)
(3)
Vbaseline = MLP baseline (v)
(4)
where MLP policy and MLP MLPbaseline baseline are 2-layer perceptrons with Tanh activation.
policy
2.4 MANAGER
Q
Match Modue
Linear
Encoder
Match Module
concat
Linear
Encoder
htt1
h2
Match Modue
Linear
Encoder 1
Match Module
Figure 3: The structure of the manager agent.
The manager aims to predict the step-wise target object given the textual information and the agent's s past targets. Inspired by human logic process in such complicated task, the reasoning order of the sub- sob-goals//n is in reverse the action order, i.e. to fulfill the goal instruction, it needs to identify the target monster, to beat the target monster, it needs to identify the target item. The overall structure of the manager is shown in Figure 3
The inputs (Q.0.A, A, H)) the manager are encoded textual embeddings from shared bidirectional LSTM (Hochreiter Schmidhuber 1997; Schuster & Paliwal 1997). We denote de and d, as the embedding dimension and the length of an object name respectively. = {oo, AI...., ... .. R xd. represents the word embeddings for the goal description. We concatenate the wiki paragraph with all object names and denote their encoded embeddings A {ag. a..... ac) I xd,, xd, represents the embeddings for object names. The past target object embeddings are denoted H (h.. h...... hv } E F xd,, xd.
O foo, 01 ........ Os R xd. xd.
The manager architecture derives from a Co-match LSTM model (Wang et al. 2018b). The Co- match LSTM was first proposed for matching among multiple sequences, e.g., matching a question and an answer option to a paragraph in Wang et al. (2018b). Waan et al. (2019a) applies this model to multi-hop reasoning over texts, benefiting from its ability to capture sequence order information. In our case, four information sources need be matched with each other and thus the original Co-match LSTM (Wang et al. 2018b] 2019a) cannot be simply plugged in. Borrowing the idea of matching sequences, we design match module M(X,XX Y) that performs the following operations:
Ejk T y/
Σ exp(ej*)
z
k= Et 21-0 exp Ezt) 1/c
(5)
m, [xz, 2. T Zjj I3 2jj0 #
M [nã,, mi,.... ..., mowl:
where X = [26, 2i. 2i.... E Wxx, and Y = (1o. Vi. 82 E R Zxde
Our model needs predict the next target object based on the current attained target (current state) while bearing the final goal in memory. For example, in Figure after the agent acquires the arcane cutlass, our model should acknowledge that the agent has that weapon in hand and its goal is to defeat the Star Alliance. The next target prediction should subject to the constraints of Wiki and be one the observed objects. Therefore, we design a dual-path framework where one path projects the constraint of wiki (O) to the the current state (h₂) and projects the result (h₂) to the observed objects (A); the other path projects the constraint of wiki (O) to the final goal (Q) and projects
the results (Qº) to the observed objects (A). Then, concatenate the results of these two paths
and regress the concatenated features to probabilities of all observed objects. Finally, we select the object with the highest probability the next target.
5

---

Under review as a conference paper at ICLR 2022
The specific design of our model is shown in Figure B] Encoder 1 (E1) and Encoder 2 (E2) are two different bidirectional LSTMs. Linear represents independent multi-layer perceptrons (MLP) and is denoted by F. We further denote C' as the concatenation operation, B as the maxpool operation. P as the expand operation and R as the reshape operation. We describe the detailed operations of our manager with Eqn. (6).
2 (E (E(()(MMO QQ R xd. Nxd, Q P([Q .... Q91) E R hy Q = [E1 (M 0))) +h] Rd. xd= hh = P(h h2 RUxdi Xd,;
A -M(AQQ9) E ER X4de; ; M(A, hf) Uxd X4de;
(6)
AQOA = C([AAO 1Ah01) -RUxdxx2d, Ao (AQON) E RUddxxx xde; A = B(E((A)) E UUxd, A (AF((A E :R: arg max di he+1 ax
EXPERIMENT
3.1 TRAINING DETAILS
The aforementioned reasoning and action steps in Section 2.1 can be divided into reasoning part and a action part evidently. The manager only needs to learn to identify the target monster and the target item sequentially, while the worker only needs to learn to get to the target object and avoid all other objects with the sub-goal given by the manager. In other words, we can train the manager and the worker individually as they take care of different parts of the whole task asynchronously. Since the whole environment is represented as text cells and is visible to the agent, we slightly change the environment such that all the obossts and items) and their positions observation. We shuffle the order of all objects to avoid that the agent takes advantage of order information. The only difference is that we remit the messy procedure go through all cells get all the items and their corresponding positions.
a part of the
To train the worker, a random goal object is selected for the worker to reach. The worker needs learn to reach the goal and avoid touching any other objects to stay alive. Since the agent will die when reaching the goal if the goal is a monster. When a monster is selected as the goal, we weaken the goal monster so that the agent will not die for reaching the goal. The agent will still die if it touches the other unselected monster. The worker needs to reach the goal object and stay alive to win. We implementation of IMPALA (Espeholi
train the worker with TorchBeast (Küttler et al.] 2019), al. 2018). We use 20 actors and a batch size of 24. We set the maximum unroll length as 80 frames. Each episode lasts for a maximum of 1000 frames. The worker is optimized using RMSProp (Tieleman & Hinton 2017) with a = 0.99 and = 0.01. It takes less than 10 hours to train the worker for 50 million frames 1 Nvidia RTX2080ti GPU.
To train the manager, we need collect a bunch of trajectories and corresponding end-game rewards for policy update. Notice that manager only cares about the sub-goal sequence and the end-game reward, so with a well trained worker, the training of the manager degenerate to supervised manner: if the manager generates a correct sub-goal sequence, i.e. the target monster and the target item, gets reward +1, otherwise it gets reward 11 For each game, we can easily extract the ground truth sub-goal sequence, so we randomly generate 100 thousand initial observations and corresponding sub-goa sequences as the dataset. We split the datset as 80/20 thousand train/val set. There are more than 2 million different monster-team-man monste-tean-moofffer-eeeeee modiffer-element combinations without considering the natural language templates, and 200 million otherwise. Thus the sub-goal dataset is a very small part of the possible scenarios. The manager will do 2-step reasoning to determine the target monster and target item sequentially. In the first step, we use (NULL) as the previous sub-goal input. In the second step, we use the firstly generated sub-goal as the previous sub-goal input. We update the parameters with eross entropy loss at each step. We use Adam optimizer (Kingma & Ba 2015) with learning rate 10 4 We train the manager module on Nvidia RTX2080t GPU with batch size 200 for 100 epochs.

---

Under review as a conference paper at ICLR 2022
Table I: Win rate performance on full RTFM
Method
x 6
10 x 10
IX12
23 + 2
worker (random)
12 2.6
12 4 0.1
FRL (forwards)
43 = 0.6
47 2.2
FRL (backwards)
84.2 + 0.3
95.7 0.1
txt2 (w/ curriculum)
55 = 22
43 - 13/55
Upperbound
~ 86
96
Figure 4: Common failure
"Result generalised from model trained on 6 games
case in FRL
3.2 RESULTS AND COMPARISON
Overall results The performance of our model with other models is show in Table [ In the table, worker (random) denotes a worker with a random manager, and FRL (backwards) denotes our framework with manager generating sub sub-goals a backwards manner, i.e., with the multi-hop manager model in 2.4 FRL (forwards) is an ablation of our solution, with the manager generating sub-goals in a forward manner. We run randomly initialized worker training on 10 grid-sized RTFM games respectively. Upperbound is the performance of our worker with the groundtruth sub-goals provided.
and x
We make the following observations. First, without curriculum learning, the existing txt2 model cannot learn policy effectively, while our model reaches near upperbound performance in both x and 10 10 RTFM games. Second, the worker (random) represents lowerbound of our FRL framework. It achieves a win rate about 12% instead of 1/4 1/3 1/12 since there another win trajectory that the player can pick up the distractor item first, then the target item and finally beat the target monster. Still, both of our FRI. models achieve much better results compared to this lowerbound. Third, txt2 trained in environment. While our FRL model performs better in open environment, since the main case that causes failure is when the monsters surround the player the corner, shown in Figure Thus in open space, the player is less likely to be trapped by monsters.
environment performs better than that in 10 x 10
Finally, the FRL (forward) outperforms the txt2* baseline, but significantly worse compared to our FRL (backward) solution. In a reasoning process from the goal to specific steps, the agent would have to know all the following steps before outputs the first step a forwards manner. This explains why the forwards manager performs much worse than the backwards one. The results demonstrate the advantage of our multi-hop reasoning model the manager agent. Also we can see that FRL (backward) performs similarly to the upperbound, which indicates that the manager does near perfect job. However this may not hold in the real-world applications with noisy texts and long reasoning sequences. In these scenarios, there by the manager. In future work, this should be addressed by injecting the real-time real-time feedback from the game environment revise the sub-goals, like in Vezhnevets al. (2177).
be error propagation in the sub-goals generated
At the beginning of the game, the manager will set the target item as the goal (bounded in red frame), then the worker will control the agent to reach the goal. Once the agent reaches the target item, the goal will change to the target monster. The worker continues to reach the goal and wins the game.
Evaluation the worker agent We group the log points in every 10.000-frame interval and show the worker training process in Figure 5 We take log points in the interval with upper bound 5 x 107 as the training accuracy. We run tests on all these five trained workers as testing accuracy, which is the upper bound performance our markger-worker framework can reach in theory. Since the sub- goals generated from the ideal manager are in the same latent space as the observation information, training process is stable and converges quickly. In other words, by generating subtly designed sub-gad -oblg the manager able to transfer the challenging reading to act problem into some simple problems and greatly alleviate the mismatch between the high-level linguistic information and the

---

Under review as a conference paper at ICLR 2022
Worker
Train Acc
Test Acc
6 x 6
85 + 0.1
85 + 1.7
10 x 10
95
= 0.1
96 + 0.3
185k
Table 2: Average train/test accuracy of five ran- domly initialized worker models.
10x10
010
Figure 5: Average win rate of 5 worker training runs.
low-level perception and actions. As result, the worker can perform very well even with simple structures.
Study of the training strategy of workers We also compare the different reward setting for train- ing the worker. Since in the RTFM setting, the game does not end immediately when the agent kills the monster. Instead, the system will determine whether the game ends after every round of movements for all entities. One possible training process be set rewarding the agent long as it reaches the target object(item or monster) without alive requirement and a training trajectory is done it reaches the target object dies. Since the agent moves first in each round, the nearby dis- tractor monster may still kill the agent after agent kills the target monster. Another training process can be rewarding the agent for reaching a sub-goal pair generated by perfect manager and staying alive. Compared with rewarding for reaching single goal and staying alive, the absence of alive requirement makes the agent short-sighted during testing, as shown in Figure 6 The item bounded in the red frame is the current target item. There is a monster next to the target item, thus there is some risk that the monster reaches where the target item is, leading to the death of the agent. For the worker trained with rewarding for reaching one object without alive requirement, the worker tends to control the agent to reach the target without considering the risk. While for the worker trained with rewarding for staying alive and reaching all objects (single pair), it tends to wait until the monster leaves the target item and avoid the risk being killed during reaching the target. The performance for the
object rewarding worker without alive requirement is about 66% and 76% in 6 x and 10 x 10 RTFM games respectively, which are obviously worse than the all object re- warding one shown in Table 5 The performance for the pair of objects rewarding worker with alive requirement is about 85% and 96% in 6 x 6 and 10 x 10 RTFM games respectively, which is close to the upperbound.
Reward for reaching one object w/o alive req.
Reward for reaching objects w/ alive req.
Figure 6: Worker trained with reward for reaching an object w/o alive requirement tends to be short- sighted. While the one trained with alive requirement tries avoid risk.
8

---

Under review as a conference paper at ICLR 2022
4 RELATED WORK
Langunge-conditioned reinforcement learning Reinforcement learning has been applied to many environments with textual observations. Representative research directions under this scope include: (1) reinforcement learning for textual instruction following: (2) reinforcement learning with textual knowledge enhancement, i.e., the reading to action direction this paper studies; (3) reinforce- ment learning in textual environments.
Most of the language-conditioned RL work belongs to the instruction following direction. such as Souual-anguage navigation (Anderson et al., 22018: Wang et al. 2019b), video gaming (Hermann 2017: [ahddanuu al., (01)), robot control (Tellxx et al. 2020) and more (Branavan et al.) 2009 The language instructions in these tasks serve as guidance to supervise the model to work a non-linguistic domain. The instructions are usually long and concrete descriptions of action sequences.
The second direction of reading to act (Branavan et al. 2009: Narasimhan al. 2017: Wang &
Narasimhan 2021 Zhong et al. (020), as discussed in Section 1 differs from the conventional in- struction following in two perspectives. First, the language instructions are usually short and only describe the target goal states. Second, the agents usually provided additional text descrip- tions about the environment as prior knowledge. Therefore, the agents need to comprehend the text knowledge in order to derive solution the specific goal. In most of the example works, the text descriptions have the form of manuals of the environments.
Finally, in the natural language processing community, there many pure text environments estab- lished for reinforcement learning research. The most important example is the dialog systems (Dhin- gra et al. 2017). The text games (Côté et al. 2018 Hausknecht al.! 2020) a recent popular field in this direction. Other examples include studies that apply reinforcement learning to conven- tional NLP tasks, like information extraction (Narasimhan al. 2016) and open-domain question answering WWan et al. 20188).
Evaluation of machine comprehension Finally, we would like to point out that our work related to a long line of work on evaluation of machine reading comprehension. Machine com- prehension capability is usually evaluated as question answering (QA) tasks, from the early at- tempts like MCTest (Richardson et al. 2013), to many QA tasks for neural reading models like CNN//aillymiil (Hermann et al. 2015) and SQuAD (Rajpurkar et al. 2016). until some recent tasks that require deep story comprehension understanding (Kočisky et al. 2018) commonsense rea- soning (Huang et al.) 2019 Wang et al. (021) skills. However, researchers also identified shortcuts for models to solve the QA tasks, questioning the appropriateness of QA an evaluation of machine comprehension.
To deal with such deficiency, new evaluation benchmarks proposed. A common design guidance of these tasks is to require the model to first comprehend the texts and use the comprehended facts to achieve the target goals. One example of these efforts is the multi-hop reasoning tasks (Welbl al.)
2018 Khot et al. 2020). As an alternative, recent works also studied direct evaluation of machine comprehension in interactive fiction game playing (Guo al. (2000) Haaskknech et al.) 2020) Yao eet al. 2021). Our studied tasks can be viewed as the intersection between the aforementioned two types evaluation tasks.
CONCLUSION
In this paper, we propose a Feudal Reinforcement Learning (FRL) framework attack the challeng- ing read-to-act problem. We design a high-level manager agent to reason and translate the multi- linguistic information into multi-step sub-tasks and introduce a low-level worker agent to perceive and act in the environment to achieve the tasks set by the manager. Our framework effectively solve the mismatching problem between the text-level inference and the low-level perception and action without human-designed curriculum. We conduct experiments on challenging tasks including Read to Fight Monsters (RTFM) and Messenger. We analyze the module functions with adequate abla- tion studies and show that our model achieves far better performance that those of state-of- state-of-thh-an the-art- models. To our best knowledge, we do not identify significant negative impacts on society resulting from this work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton Van Den Hengel. lannn-agn- navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 3674-3683, 2018
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefen- stette. Learning to understand goal specifications by modelling reward. International Confer- ence on Learning Representations (ICLR), 201.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pp. 30-37. Elsevier, 1995.
Andrew Gehret Barto, Richard Sutton, and CJCH Watkins. Learning and sequential decision making, University of Massachusetts Amherst, MA, 1989.
SRK Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meet- ing of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 82 20-90,220 2009.
SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in a monte- carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.
Inigo Casanueva, Pawel Budzianowski, Pei-Hao Su, Stefan Ultes, Lina Rojas- Barahona, Bo-Hsiang Tseng, and Milica Gašic. Feudal reinforcement learning for dialogue management in large do- mains. In Proceedings of NAACL-HLT, pp. 714-719, 2018.
Alex-Ale Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. Workshop on Computer Games, pp. 41-75. Springer, 2018.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in Neural Infor- mation Processing Systems (NIPS), 1993. URL https: httpttppsp///ppoorreeeecceee ngs neur suturrpps.. cc/
pdf
Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmad, and Deng- Towards end-to-end reinforcement learning of dialogue agents for information access. In Pro- ceedings of the 55th Anmual Meeting of the Association for Computational Linguistics Coatticssoolllla (Volume 1; Long Papers). pp. 484-495, 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted learrrl architectures. In International Conference on Machine Learning, pp. 1407-1416. PMLR, 2018.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume MIT press Cambridge, 2016.
Jonathan Gray, Kavya Srinet, Yacine Jernite, Haonan Yu, Zhuoyuan Chen, Demi Guo, Siddharth Goyal, C Lawrence Zitnick, and Arthur Szlam. Craftassist: A framework for dialogue-enabled interactive agents. arXiv preprint 1rXiv:1907.08584, 2019.
Xiaoxiao Guo, Mo Yu, Yupeng Gao, Chuang Gan, Murray Campbell, and Shiyu Chang. Interac- tive fiction game playing as multi-paragrayy reading comprehension with reinforcement learning. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 7755-7765, 2020.
Matthew Hausknecht, Prithviraj Ammanabrolu, Mare-Alexandre Côté. and Xingdi Yuan. Interac- tive fiction games: colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7903-7910, 2020.
10

---

Under review as a conference paper at ICLR 2022
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, 2015.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded lan- guage learning in simulated 3d world. arXiv preprini arXiv: 7006 706.06551, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading the 2019 Conference
comprehension with contextual commonsense reasoning. Proceedings the on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 2391-2401, 2019.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8082-8090, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR (Poster), 2015.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. Torchbeast: A pytorch platform for distributed rl. arXiv preprint arXiv:1910.0355) 2019.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. arXiv preprint 1eX5v:8805.082 2018.
Karthik Narasimhan, Adam Yala, and Regina Barzilay. Improving information extraction by ac- quiring external evidence with reinforcement learning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2355-2365, 2016.
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Deep transfer in reinforcement learning by language grounding. arXiv preprint arXiv: / 77800003335. 2017.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383 2383-2392, 2016.
Matthew Richardson, Christopher JC Burges, and Erin Renshaw. Metest: challenge dataset for the open-domain machine comprehension of text. In Proceedings the 2013 conference on empirical methods natural language processing, pp. 193-203, 2013.
M. Schuster and K.K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 16661)22673-2611, 1997. doi: 10. 159/7..650093.
Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gaze and Cynthia Matuszek. Robots that use lan- guage. Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55, 2020.
Tieleeman and G Hinton. Divide the gradient by a running average its recent magnitude. cours- era: Neural networks for machine learning. Technical Report., 2017.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hicrarchical reinforcement learning. In International Conference Machine Learning, pp. 3540-3549, PMLR, 2017.
Haoyu Wang, Mo Yu, Xiaoxiao Guo, Rajarshi Das, Wenhan Xiong, and Tian Gao. Do multi-hop readers dream of reasoning chains? arXiv preprint arXiv: /9/0. 14520, 2019a.

---

Under review as a conference paper at ICLR 2022
HJ Wang and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. arXiv preprint arXiv:2101.07393, 2021.
Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 3: Reinforced ranker-reader for open-domain question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018a.
Shuohang Wang. Mo Yu, Jing Jiang, and Shiyu Chang. co-matching model for multi-choice reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 2: Short Papers), pp. 746-751, 2018b.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised im- itation learning for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019b.
Zhonghao Wang, Kai Wang, Mo Yu, Jinjun Xiong, Wen-mei Hwu, Mark Hasegawa-Johnson, and Humphrey Shi. Interpretable visual reasoning via induced symbolic space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1878-1887, 2021.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop read- ing comprehension across documents. Transactions of the Association for Computational Lin- guistics, 6:287-302, 2018.
Shunyu Yao, Karthik Narasimhan, and Matthew Hausknecht. Reading and acting while blindfolded: The need for semantics text game agents. arXiv preprint arXiv:2103.13552 2021.
Zhong, Rocktäschel, and E Grefenstette. Rtfm: Generalising to new environment dynamics via reading. In ICLR, pp- 1-17. ICLR, 2020.
12

---

Under review as a conference paper at ICLR 2022
A
ADDITIONAL EXPERIMENTS IN THE MESSENGER ENVIRONMENT
A.1
SETTINGS
Task We also evaluate our model on Messenger (Wang & Narasimhan, 2021 stage 2 and stage tasks.
In Messenger stage 2, there are three different entities: an enemy, a message, and a goal. Each of the entities is assigned stationary, chasing, or fleeing movement type. The player is provided with the manual of descriptions for each of the entities. The objective of the player is to bring the message to the goal while avoiding the enemy. If the player encounters the enemy during the game, or the goal without obtaining the message, it loses the game and obtains a reward of -1. Rewards of 0.5 and are provided for obtaining and delivering the message to the goal respectively.
In Messenger stage 3, distractor message and distractor goal are added. These two distractor entities have the same names as the goal entities but different motion types. The agent cannot distinguish the same named entities without observing their motion during the game. Other game settings are the same as the stage 2.
Different from the RTFM task, the agent needs to learn the one-to-one mapping from each of the description sentences to each of the entities in the visual space. There are not multiple reasoning steps to do to accomplish the task.
Training details Similar to training on the RTFM, we train the worker and the manager separately.
To train the manager, because of the gap among train, val, and test environments in Messenger, we randomly generate 1000/500/500 initial observations from train/val/test environments of stage and stage respectively. Since there is no explicit "task sentence" in the Messenger environment, set "get message then get goal" the task sentence for the manager. In stage 2, the manual is enough for determine the one-to-one mapping. While in stage 3, since the manager works at the beginning of each game, it cannot distinguish entities with the same name. So in both cases, only the manual without the entities is encoded as O. There is no difference between forward and backward manager, because the agent does not need the previous target reason the next target, in other words, it only need to learn the one-to-one mapping. We use the same cross entropy loss and Adam optimizer (Kingma & Ba 2015) with learning rate 10-4. We train the manager module on Nvidia RTX2080úi GPU with batch size 100 for 100 epochs.
To train the worker, we follow the same method as the RTFM task. We use TorchBeast (Küttler et al. 2019) with 20 actors and batch size of 24. We set the maximum unroll length as 80 frames and RMSProp (Tieleman & Hinton 2017) as the optimizer.
A.2 RESULTS AND COMPARISON
We show the worker performance in Stage 2 with ideal manager sub-goals in Table which can be considered the upperbound for our FRL model. Given correct sub-goals, worker agent can accomplish the task in a near perfect performance.
We perform evaluation in the Messenger test environment, which includes previously unseen games. As shown in Table EMMA (Wang & Narasimhan 202T) cannot learn anything without curricu-
Table 3: Average worker win rate in training, validation, and testing environment of Stage 2.
Train Win Rate
Val Win Rate
Test Win Rate
99.5 + 0.2
99.7 + 0.1
94.6 + 0.9
lum. Although Wang & Narasimhan 2 claim the player need to observe the dynamics to learn the mapping between the description sentences and the entities, they set a special Stage 1 during which all entities are immovable. In such case, there is dynamic difference between different entities. And the specific character representing the entity in the visual space has a specific mean- ing, such mage, queen and so Also shown as the win rate of Stage 3 of EMMA, the huge
13

---

Under review as a conference paper at ICLR 2022
Table 4: Performance on Messenger
Method
Stage 2 Test Win Rate
Stage 3 Test Win Rate
EMMA
FRL
93 + 0.2
13 + 1.4
EMMA (w/ curriculum)
85 = 0.6
10 = 0.8
Upperbound
~ 95
~ 95
drop of performance indicates that their model is actually learning the mapping from the description sentences to the characters in Stage 1 instead of learning the mapping through reasoning and inter- acting with the environment. While our manager agent well connects the linguistic information and the visual information, our model achieves extraordinary performance in Stage 2. However, due to the indistinguishable setting for the entities, the performance of our model also drops. With the ini- tial observation, which has no motion information, the manager cannot distinguish the same named entities, randomly guess the goal entity between them. We argue it not suitable setting for reading-to- lo- task, because with only the manual description, it cannot provide all the information finish task.
14


Reviews:
[{"review_id": "yGyW4ekz0_5", "reviewer": "Reviewer_RC7c", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive points:\n\n1. There seems to be quite a large gain in performance with the FRL model that generates sub-goals in a backwards manner.\n2. Although, this seems trivial, especially given that the \"upperbound\" i.e., a model that has perfect subgoal information gets nearly the same performance as the FRL-backward model.\n\nNegative points:\n\n1. There are several things in the paper that are not clearly explained at all that made this hard to follow. Example sentences are:\na) \"It achieves a win rate about 12% instead of 1/4 · 1/3 = 1/12\"\nb) \"In the fedual reinforcement learning (FRL) formulation, both the manager and the worker subject to Markov Decision Processes (MDP).\"\n2. There doesn't seem to be significant/novel introduction in terms of modelling changes for the model.", "summary_of_the_paper": "This paper proposes a feudal reinforcement learning model to solve the task of reasoning over textual (language) instructions that incorporate with low level state/action/environment information. This feudal model is composed of two agents, one of which is a \"manager\" that generates a multi-hop plan composed of several subgoals, while the other is a \"worker\" that takes in low-level information and executes actions to solve each sub-goal. They empirically evaluate on two challenging domains and show competitive performance to baselines.", "summary_of_the_review": "Limited novelty and improvements but significant gains in performance---although unclear because the writing isn't clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "gW7GPrR9Rty", "reviewer": "Reviewer_KMX2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Using the Neurips review:\n\nOriginality: The key novelty I find is to use a heirarchical RL approach to segregate the transformation of the textual instructions into goals and the actual RL style execution of the goals. This is a natural approach and intuitively might be generalizable to other contexts. The design of the manager using the backward multi-hop generator is somewhat novel. Experiments are mostly straightforward though  there is one  minor novelty in the ablation to a forward generating manager. \n\nQuality: I like the basic idea of the approach, i think a well-executed version would be worthy of an ICLR paper. But there are some concerns about what is done so far in this paper:\na. The modeling of the language seems out of date, using LSTMs etc. Transformer-based would be more state of the art (and really easy to implement I imagine using off the shelf tools these days), and the cross-attention mechanism would be an elegant way to do what the (hacky-looking?) matching module seems to be doing.\n\nb. Experiments seem hyper-specific to the RTFM environment. This is really concerning for being able to generalize conclusions of this work to other (esp. real world) applications. For example, I see it quite possible that the results of different modeling approaches are dependent on the way the instructions are written. For a different set of instructions, forward generation by the manager might be better (e.g if the instructions were more like a step-by-step plan, in which case forward would be better than reverse because it would help resolve anaphora better). Many of the discussion of the results seem too fixated on the details of this particular environment (e.g. the \"Study of the training strategy\", where a lot of discussion is devoted to explaining interactions of player and monster etc, how would we draw insights for other problems from this?).\n\nTo be fair though, the experimental results do seem impressive in the sense of the delta vs baseline (I am charitably assuming text2\\pi is a competitive sota model from the literature, the authors dont mention anything about it). In fact, for these environments, the problem is practically solved, which is notable for a challenge problem recently published.\n\nClarity: This is the biggest weakness. There are numerous grammatical and semantic mistakes, that make it impossible to understand the author's intent or requires multiple readings and putting peices in different places together to understand what's going on. Sec 2.2 took a while, part of the problem is that the problem setting was not described rigorously, so I had to figure out by implication what a \"object name\"  is and why we need to add d_n extra dimensions, what is meant by \"grid of word embeddings\" etc. I dont really understand what the reasoning for the precise operations in the  match module defined in eq 1 are, some explanation by the author could have been useful.  On the flip side, a lot of space is devoted to details that can be moved to the appendix like learning rates and an unnecessary subsection on machine comprehension related work (for example). This would have freed up space for moving the Messenger experiments into the main body, which would allow for a more comparative analysis of experiments leading to more generally applicable conclusions. A lot of work is needed to get the paper into a readable shape, and we can't trust that it will be done based on the current draft alone. \n", "summary_of_the_paper": "The paper applies a feudal RL algorithm to solve text-adventure type problems where information is available in text form about the environment. A manager agent reads the text and generates a sequence of subgoals, and worker agents execute the subgoals to reach the final goal. The models are described, experiments are described (mostly in the RTFM but a callout to Messenger is there in Appendix). ", "summary_of_the_review": "A nice natural idea for doing RL with access to text instructions, experiments on RTFM impressive but only 1 setting is a negative. Poor readability is the biggest negative.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RC6i9AvmY6b", "reviewer": "Reviewer_tKa7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- I find the application of Feudal Reinforcement Learning (FRL) interesting and novel in this setting.\n\nWeaknesses\n- the paper is quite hard to follow and it misses many details. On page 4, the authors mention the Bellman equation and \"a typical policy learning schedule\". The authors do not provide any formal definition of which exact RL method has been used, e.g. on/off policy etc. Moreover, it is very hard to read/understand Eq. 1. and 2. I suggest the authors rewrite it and better connect them to the figures. \n- [EDIT] Clarified in the rebuttal [EDIT]. The results reported in Table 1 doesn't match the one from txt2$\\pi$ (Zhong et al. 2020), which achieve 84±21 in the training environments,  83±21 in the 6x6 and 66±22 in 10x10. Could the authors' comment on the results in Table 1? is the setting comparable?\n- the paper claims, in the introduction, that the evaluation is done in two environments. However, the experiments in Messager are only preliminary and only shown in the appendix. I suggest explicitly mentioning this in the intro. ", "summary_of_the_paper": "In this paper, the authors propose a Feudal Reinforcement Learning (FRL) algorithm for improving the miss-match between high-level natural language commands (e.g., \"share my recent photo to my parents\") and the actual complex set of operations required by the systems (e.g., open the album --> select recent photo --> etc.). FRL (Dayan & Hinton, 1993) is a hierarchical RL algorithm with two agents: manager and worker. The manager issues a plan which is executed by the worker. To elaborate, the manager learns to generate a sequence of targets given $Q$ the goal description, $O$ the wiki paragraph which describes the game, $A$ the object names in the environment, and $H$ the subgoal history ($h_t$ refers to the target object at time $t$). The worker, instead, uses the plan from the manager (in terms of coordinate to the target ($X_target$)), the observation $E_{obs}$, and the position to the other player ($X_{pos}$) (not sure I understood correctly this) to interact with the environment. \n\nThe manager and the worker are trained separately using imitation learning (also here it is a bit confusing), from a 100K trajectory collected by a random walk in the map. \n\nThe authors benchmarked the FRL with two text-based interactive games such as RTFM (Zhong et al. 2020) and Messager (Wang & Narasimhan, et al 2021) (the results are only in the appendix, but this paper is very recent). The results show that FRL is better than txt2$\\pi$ without curriculum learning (here I am a bit unsure because the results in the original paper are different), in the RTFM.\n", "summary_of_the_review": "An interesting application of Feudal Reinforcement Learning (FRL) in text-guided RL environments (e.g., RTFM (Zhong et al. 2020)). Lack of details limits the significance of the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DHZzCru6V8K", "reviewer": "Reviewer_eXEz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written and easy to understand. The authors motivate the problem and introduce their model in relation to past works quite well. \n\n**Clarity in Model Specs:** While describing the model, the Co-match LSTM model. I found this part to be unclear. I would urge the authors to introduce the intuitions and notations used in equations (1) before presenting them to make that section clearer. Further, Figure 2 shows linear layers, but the authors say that \"Linear represents independent MLPs\". I then urge the authors to change the figure.\n\n**Collecting data for training the manager:** The authors state that the manager is trained by letting an agent walk randomly to generate a trajectory and using the successful trajectories as supervision for the manager. This seems like an inefficient way to collect successful trajectories by letting an agent execute random actions. I would have liked to see more details about how many random trajectories were needed before sufficient successful trajectories were collected. Moreover, collecting successful trajectories is not possible in more complex environments. I urge the authors to discuss how their manager model would learn in such scenarios.\n\n**Cases of failure of FRL:** The authors do an excellent job studying ablations and how different training strategies affect the policy. I would have liked to see the failure cases of the model and why the authors think that the model fails on these scenarios.\n\n**Results on Messenger:** The authors include the results of messenger in the appendix. The results are pretty good but are barely included in the main text. I would have liked to see some of them discussed (at least broadly, without specifics; I understand that there are space constraints). Moreover, I would have liked to see a description of the Messenger benchmark with a figure (like that of RTFM) and a description of the different splits used to evaluate (in the appendix). Finally, as the model fails on Stage 3 of Messenger, I would have liked to see a few failure cases and why the authors think the model fails, along with a speculative discussion of how it could be mitigated. \n\nTypos:\n\nPage 4: Our model need → Our model needs", "summary_of_the_paper": "The paper presents a hierarchical RL model to solve tasks that require reading and understanding instructions or manuals to solve complex goals (these goals usually require multiple steps of solving subgoals). The proposed model has two parts: a manager that uses the instructions/ manuals to design subgoals, and a worker that solves the subgoals and deals with low-level perceptions. The manager and the workers are trained separately. The Feudal model achieves SOTA scores on Messenger and RTFM without the use of a hand-designed curriculum.", "summary_of_the_review": "The FRL model shows impressive results on Messenger and RTFM. Still, there are a few improvements that the authors can make to improve the clarity, the analysis of results and also provide a more detailed discussion about cases of failure, especially on Messenger. One weakness of the model is how the data is sampled to train the manager, additional discussion on how this can be scaled to more complex environments is needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all raise critical issues with regard to both description and equations, and indicate that figures are not helpful. This is even after the revision. In response to KMX2, the authors suggested they will post additional experiments, but did not return to indicate that. This seems critical to answer empirical concerns about generality of the approach. We recommend to address these issues, if the authors decide to resubmit. 

The meta review and recommendation discount the review of RC7c. Unfortunately, the AC and other reviewers, weren't able to engage RC7c in the discussion and the review was extremely short.
Decision: Reject