Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new agent for Text Based Environment (TBG). The new agent is based on Linear Temporal Logic (LTL) and address the shortcomings of the existing SOTA agent GATA (Graph Aided Transformer Agent). Authors show that GATA agent is agnostic to instructions and are typically not successful in completing tasks in TBG. Authors incorporate formal representation of natural language instructions (via LTL) and this provides the agent with a measure of progress in task completion and helps the agent to complete tasks. 
strengths_and_weaknesses: Strengths:

1. Authors experimentally show that the SOTA agent GATA largely ignores initial instructions while playing the game. Further, authors show that the agent is ignorant of the final goal of the game (in this the agent does not know what it is suppose to cook as the final goal), the agent is solely driven by rewards alone. This is an interesting empirical insight by the paper. 

2. Authors develop and add a new component to the existing GATA architecture to overcome its limitation. The new component is a LTL Updater that generates LTL instructions from natural language observations and it also progresses the instructions.  



Weaknesses:

1. The use of formal structured language improves the performance but it is not generalisable to other domains. Authors in this case develop the specifications for cooking domain but if the agent were to be deployed for another domain new set of specifications would be required. LTL limits the possibility of deploying an agent to a real world setting where agent would have to deal with different tasks from varied domains. In such cases it is not possible to come up with specification for each possible domain as the number of such possibilities in principle is infinite. 

questions: None
limitations: The main limitation of the proposed model is that it is not generalizable to many (and possibly unknown) domains. See also limitation above.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper shows that the performance of the GATA agent is not affected by stripping the text instructions from the environment observations. This shows that the agent doesn’t consider the task instruction when making decisions. This paper proposes a method to improve task completion by exploiting progress monitoring in LTL. The resulting agent shows improvements in the Cooking domain at level 1 and 2. 
strengths_and_weaknesses: Strength
- This paper shows an analysis that reveals the weakness of the previously proposed model.
- The experiments at different levels and ablations show how incorporating LTL in text-based games can help task execution.

Weakness
- Novelty: Exploiting LTL progress monitoring to improve RL agents’ performance has been demonstrated in non-text-based games [Vaezipoor 2021]. It is good that the same method is applied to the text-based games but this shows limited novelty of the proposed method. It will be helpful if the authors can discuss how LTL can affect the training or design of the agents in text-based games compared to non-text-based games.
- Generalization: GATA was evaluated in 500+ games in TextWorld. However, the proposed method is only evaluated in the Cooking domain. It is unclear how the model can be extended to other domains. 
questions: - What’s the diversity of the converted LTL formulas? From the paper and the supplemental materials, the instructions are mainly concatenated with Eventually and Next. If the instructions are in simple forms, we may only need to co-train an event detector to match the events mentioned in the instructions without LTL formulas. Ablation of the LTL formula but keeping the event detector can show how much the text-based games require LTL understanding.
- The experiments show most improvements in level 1 and 2, but not 3 because navigation is still an issue. Is it possible that the progress of navigation can be encoded as the LTL formulas so the same method can help navigation as well?
limitations: While a simple LTL translator works for the Cooking domain, for more complex text-based instructions, the efficacy of the proposed method is limited by the quality of the semantic parser that parses natural language sentences into LTL formulas. It would be useful if the authors can discuss the impact of parser quality.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors first identity some important issues in recent text-based game agents that they tend to ignore instructions, these instructions can be dominant in solving text-based games. Then, the authors propose to equip agents with an internal structured representation of instructions, in the form of LTL, a formal language.

Designed for the cooking games in TextWorld, the authors use three types of instructions:
1. the agent needs to examine the cookbook to know the goal of the current game (which changes in every game);
2. the information written in the cookbook;
3. go to the kitchen if the agent is spawned elsewhere. 

The instructions can be used in multiple ways:
1. (main use) as an additional input being fed into the model, it can guide the agent to act in a favorable manner.
2. (additional use) to help shaping rewards, but as the authors discuss in Appendix H.4.2, this is not very helpful.

The proposed agent contains a minimal modification from prior work (the LPL updater), the authors show that equipped with the LPL module, their agent clearly outperforms baselines (GATA). 



strengths_and_weaknesses: - Strengths
1. The authors identify issues in existing (top performing) RL agents, and aim to mitigate these issues. The motivation is clear.
2. The authors show results that equipped with the proposed LPL module, GATA can obtain a performance gain on all the settings, this is especially obvious in level 2 games, where the tasks are the most difficult (ignoring navigation). The contribution is solid. 
2. The proposed system contains only minimal modification from prior work's model architecture, it is easy to compare the proposed method against baselines.
3. The presentation if clear, the idea is easy to follow.
4. Using success rate + game score as metric makes a lot of sense. 
5. The authors point out some implementation issues of prior works and provide bug fixes. 

- Weaknesses
1. The authors fails to sufficiently discuss the weaknesses of LPL-aided agents. For instance, it seems that for every domain (e.g., cooking, crafting, navigation), the proposed system requires to some level human annotated instructions. To what extend do the proposed instructions scale or transfer? Given a new domain (either a simulation environment or some real world tasks), what are ways to design good instructions? I am not against leveraging humans as domain experts, but these discussion can help the readers to better understand the bigger picture.
2. I suggest the authors to include numbers in the appendix, so future works can compare with their numbers. 

- Typo
1. L192, Table 3 --> Figure 3

questions: 1. Please see weakness 1.
2. Related to Q1, it would be nice if the authors could discuss some potentials using large pre-trained language models (e.g., GPT-3) for instruction generation, then using similar large LMs (e.g., Codex) to translate natural language instructions into formal language (e.g., LTL).
3. Regarding the first implementation issue discussed by the authors [HERE](https://github.com/xingdi-eric-yuan/GATA-public/blob/c1afc3c9ab38256f839b3e0ddf8243796df5bd77/dqn_memory_priortized_replay_buffer.py#L120-L123) and authors' code (experience_replay.py, L122-L129). I agree that length should be `n`, but the reward line should have `self._buffer[idx + j + i + 1]` instead of `self._buffer[idx + j + i]`? This is because that at a game step $t$, what's being pushed into the replay buffer is the reward $r_{t-1}$ resulted by the action at previous game step. Correct me if I'm wrong. 


limitations: The authors discussed broader impact including potential societal concerns in the appendix. I agree and I do not see major issue regarding this.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes to incorporate Linear Temporal Logic (LTL) into a TextWorld game agent (GATA) to (1) provide augmented rewards based on subgoal completion, (2) encode text instruction. Authors find that GATA cannot properly attend to task instructions, and the performance does not drop even if instruction information is not given. Instead, LTL-GATA improves test results in Level 1-2 of TextWorld, but still limited in Level 3 by navigation.
strengths_and_weaknesses: **Strengths**:
- The analysis of LTL failing to attend to instructions points to important problems in TextWorld agent design. 
- The introduction of LTL is novel in text games.

**Weaknesses**:
- The title is a bit misleading. Text-based games can mean synthetic games like TextWorld or harder human-designed games like Zork I, but apparently the method cannot apply to the latter. Maybe better to explicitly say "TextWorld" instead of "Text-based Games".

- The formula of LTL seems very simple in TextWorld, and I'm not sure if the approach can be useful beyond simple, synthetic domains with clean logic. Experiments on more complex or fuzzy (e.g. more natural language) domains will make the work much stronger.

- I feel the comparison of LTL-GATA and GATA is a bit unfair, as the former is given with more processed information (e.g. turning NL instructions into LTL synthetic language, and subtask completion maintenance) which assumes ground truth access to the "logic" of the instruction. It'd be better if the model can learn to turn raw game observations into the logic itself, instead of relying on gold annotations.

- The introduction of LTL and LTL progression is formal but not very intuitive. Maybe combine some examples from Section 4.1 is better.
questions: - In GATA, is a text instruction of the whole task always fed to the model (as part of text observation)? It seems so in Figure 2, just want to confirm. 
  - If so, is there any hypothesis why? 
  - Similarly, why GATA fails to attend to cookbook information?
  - Is cookbook information same as text instruction of the whole task?
- Line 37: Why "partial completion of tasks can be unsafe"?
- Between line 179-180: what is $\Phi$ in $R_{\Phi}$? 

limitations: The limitation about navigation is briefly discussed, but I think more limitations should be discussed (see weaknesses).
review_title: null

Meta-review: metareview: This paper proposes a linear temporal logic-based solutions to some of the common failure modes of existing agents addressing text-based games, stemming from the analysis of these failure modes. There was disagreement about whether the work is publishable based on the limited range of domains LTL supports. That said, from reading the discussion, it seems that the emerging consensus is that it is sufficiently novel and interesting for this work to prove the concept in text domains, and that further work may seek to extend such approaches (or draw inspiration from them) in a wider range of domains. I support acceptance.
confidence: Certain
Decision: Accept