Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper applies a feudal RL algorithm to solve text-adventure type problems where information is available in text form about the environment. A manager agent reads the text and generates a sequence of subgoals, and worker agents execute the subgoals to reach the final goal. The models are described, experiments are described (mostly in the RTFM but a callout to Messenger is there in Appendix). 
main_review: Using the Neurips review:

Originality: The key novelty I find is to use a heirarchical RL approach to segregate the transformation of the textual instructions into goals and the actual RL style execution of the goals. This is a natural approach and intuitively might be generalizable to other contexts. The design of the manager using the backward multi-hop generator is somewhat novel. Experiments are mostly straightforward though  there is one  minor novelty in the ablation to a forward generating manager. 

Quality: I like the basic idea of the approach, i think a well-executed version would be worthy of an ICLR paper. But there are some concerns about what is done so far in this paper:
a. The modeling of the language seems out of date, using LSTMs etc. Transformer-based would be more state of the art (and really easy to implement I imagine using off the shelf tools these days), and the cross-attention mechanism would be an elegant way to do what the (hacky-looking?) matching module seems to be doing.

b. Experiments seem hyper-specific to the RTFM environment. This is really concerning for being able to generalize conclusions of this work to other (esp. real world) applications. For example, I see it quite possible that the results of different modeling approaches are dependent on the way the instructions are written. For a different set of instructions, forward generation by the manager might be better (e.g if the instructions were more like a step-by-step plan, in which case forward would be better than reverse because it would help resolve anaphora better). Many of the discussion of the results seem too fixated on the details of this particular environment (e.g. the "Study of the training strategy", where a lot of discussion is devoted to explaining interactions of player and monster etc, how would we draw insights for other problems from this?).

To be fair though, the experimental results do seem impressive in the sense of the delta vs baseline (I am charitably assuming text2\pi is a competitive sota model from the literature, the authors dont mention anything about it). In fact, for these environments, the problem is practically solved, which is notable for a challenge problem recently published.

Clarity: This is the biggest weakness. There are numerous grammatical and semantic mistakes, that make it impossible to understand the author's intent or requires multiple readings and putting peices in different places together to understand what's going on. Sec 2.2 took a while, part of the problem is that the problem setting was not described rigorously, so I had to figure out by implication what a "object name"  is and why we need to add d_n extra dimensions, what is meant by "grid of word embeddings" etc. I dont really understand what the reasoning for the precise operations in the  match module defined in eq 1 are, some explanation by the author could have been useful.  On the flip side, a lot of space is devoted to details that can be moved to the appendix like learning rates and an unnecessary subsection on machine comprehension related work (for example). This would have freed up space for moving the Messenger experiments into the main body, which would allow for a more comparative analysis of experiments leading to more generally applicable conclusions. A lot of work is needed to get the paper into a readable shape, and we can't trust that it will be done based on the current draft alone. 

summary_of_the_review: A nice natural idea for doing RL with access to text instructions, experiments on RTFM impressive but only 1 setting is a negative. Poor readability is the biggest negative.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a feudal reinforcement learning model to solve the task of reasoning over textual (language) instructions that incorporate with low level state/action/environment information. This feudal model is composed of two agents, one of which is a "manager" that generates a multi-hop plan composed of several subgoals, while the other is a "worker" that takes in low-level information and executes actions to solve each sub-goal. They empirically evaluate on two challenging domains and show competitive performance to baselines.
main_review: Positive points:

1. There seems to be quite a large gain in performance with the FRL model that generates sub-goals in a backwards manner.
2. Although, this seems trivial, especially given that the "upperbound" i.e., a model that has perfect subgoal information gets nearly the same performance as the FRL-backward model.

Negative points:

1. There are several things in the paper that are not clearly explained at all that made this hard to follow. Example sentences are:
a) "It achieves a win rate about 12% instead of 1/4 · 1/3 = 1/12"
b) "In the fedual reinforcement learning (FRL) formulation, both the manager and the worker subject to Markov Decision Processes (MDP)."
2. There doesn't seem to be significant/novel introduction in terms of modelling changes for the model.
summary_of_the_review: Limited novelty and improvements but significant gains in performance---although unclear because the writing isn't clear.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a hierarchical RL model to solve tasks that require reading and understanding instructions or manuals to solve complex goals (these goals usually require multiple steps of solving subgoals). The proposed model has two parts: a manager that uses the instructions/ manuals to design subgoals, and a worker that solves the subgoals and deals with low-level perceptions. The manager and the workers are trained separately. The Feudal model achieves SOTA scores on Messenger and RTFM without the use of a hand-designed curriculum.
main_review: The paper is clearly written and easy to understand. The authors motivate the problem and introduce their model in relation to past works quite well. 

**Clarity in Model Specs:** While describing the model, the Co-match LSTM model. I found this part to be unclear. I would urge the authors to introduce the intuitions and notations used in equations (1) before presenting them to make that section clearer. Further, Figure 2 shows linear layers, but the authors say that "Linear represents independent MLPs". I then urge the authors to change the figure.

**Collecting data for training the manager:** The authors state that the manager is trained by letting an agent walk randomly to generate a trajectory and using the successful trajectories as supervision for the manager. This seems like an inefficient way to collect successful trajectories by letting an agent execute random actions. I would have liked to see more details about how many random trajectories were needed before sufficient successful trajectories were collected. Moreover, collecting successful trajectories is not possible in more complex environments. I urge the authors to discuss how their manager model would learn in such scenarios.

**Cases of failure of FRL:** The authors do an excellent job studying ablations and how different training strategies affect the policy. I would have liked to see the failure cases of the model and why the authors think that the model fails on these scenarios.

**Results on Messenger:** The authors include the results of messenger in the appendix. The results are pretty good but are barely included in the main text. I would have liked to see some of them discussed (at least broadly, without specifics; I understand that there are space constraints). Moreover, I would have liked to see a description of the Messenger benchmark with a figure (like that of RTFM) and a description of the different splits used to evaluate (in the appendix). Finally, as the model fails on Stage 3 of Messenger, I would have liked to see a few failure cases and why the authors think the model fails, along with a speculative discussion of how it could be mitigated. 

Typos:

Page 4: Our model need → Our model needs
summary_of_the_review: The FRL model shows impressive results on Messenger and RTFM. Still, there are a few improvements that the authors can make to improve the clarity, the analysis of results and also provide a more detailed discussion about cases of failure, especially on Messenger. One weakness of the model is how the data is sampled to train the manager, additional discussion on how this can be scaled to more complex environments is needed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a Feudal Reinforcement Learning (FRL) algorithm for improving the miss-match between high-level natural language commands (e.g., "share my recent photo to my parents") and the actual complex set of operations required by the systems (e.g., open the album --> select recent photo --> etc.). FRL (Dayan & Hinton, 1993) is a hierarchical RL algorithm with two agents: manager and worker. The manager issues a plan which is executed by the worker. To elaborate, the manager learns to generate a sequence of targets given $Q$ the goal description, $O$ the wiki paragraph which describes the game, $A$ the object names in the environment, and $H$ the subgoal history ($h_t$ refers to the target object at time $t$). The worker, instead, uses the plan from the manager (in terms of coordinate to the target ($X_target$)), the observation $E_{obs}$, and the position to the other player ($X_{pos}$) (not sure I understood correctly this) to interact with the environment. 

The manager and the worker are trained separately using imitation learning (also here it is a bit confusing), from a 100K trajectory collected by a random walk in the map. 

The authors benchmarked the FRL with two text-based interactive games such as RTFM (Zhong et al. 2020) and Messager (Wang & Narasimhan, et al 2021) (the results are only in the appendix, but this paper is very recent). The results show that FRL is better than txt2$\pi$ without curriculum learning (here I am a bit unsure because the results in the original paper are different), in the RTFM.

main_review: Strengths
- I find the application of Feudal Reinforcement Learning (FRL) interesting and novel in this setting.

Weaknesses
- the paper is quite hard to follow and it misses many details. On page 4, the authors mention the Bellman equation and "a typical policy learning schedule". The authors do not provide any formal definition of which exact RL method has been used, e.g. on/off policy etc. Moreover, it is very hard to read/understand Eq. 1. and 2. I suggest the authors rewrite it and better connect them to the figures. 
- [EDIT] Clarified in the rebuttal [EDIT]. The results reported in Table 1 doesn't match the one from txt2$\pi$ (Zhong et al. 2020), which achieve 84±21 in the training environments,  83±21 in the 6x6 and 66±22 in 10x10. Could the authors' comment on the results in Table 1? is the setting comparable?
- the paper claims, in the introduction, that the evaluation is done in two environments. However, the experiments in Messager are only preliminary and only shown in the appendix. I suggest explicitly mentioning this in the intro. 
summary_of_the_review: An interesting application of Feudal Reinforcement Learning (FRL) in text-guided RL environments (e.g., RTFM (Zhong et al. 2020)). Lack of details limits the significance of the paper.
review_title: null

Meta-review: The reviewers all raise critical issues with regard to both description and equations, and indicate that figures are not helpful. This is even after the revision. In response to KMX2, the authors suggested they will post additional experiments, but did not return to indicate that. This seems critical to answer empirical concerns about generality of the approach. We recommend to address these issues, if the authors decide to resubmit. 

The meta review and recommendation discount the review of RC7c. Unfortunately, the AC and other reviewers, weren't able to engage RC7c in the discussion and the review was extremely short.
Decision: Reject