Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper utilizes world models for text-based reinforcement learning agents to be aware of the change in the world based on one's own actions and what actions could be performed based on the current state of the world using Worldformer, a multi-task transformer for jointly learning to generate the knowledge graph differences based on the action taken and a set of valid action at the current state. Since the valid actions generated at the current step affect the action taken to change the world, these tasks are learned using the same loss function.

**Challenges**: Partial observability, combinatorial action space
main_review: **Major Comments:**
1. It would be helpful if the sample valid actions generated for some of the observations in the test set and the corresponding ground truth valid actions are shown side-by-side for comparison along with the graph additions and deletions. In addition to the numbers shown in Tables 2 and 3, qualitative examples could be very helpful in understanding the model.

2.  The results in the experiment section show how well the worldformer performs on the KG prediction and valid action prediction tasks for the held-out set. Does using this model for playing the games show any advantage over the other baselines in terms of scores/rewards? How will these learned world models be used for text-based games?

3. Does the checkmark in multitask mean the KG prediction and valid action task were learned independently?


**Minor Comments:**
1. Type in line 260 - table 2
2. Line 286 - 5 tokens
limitations_and_societal_impact: Yes, the broader statement explains the necessary details on the limitations and societal impacts.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a novel knowledge graph construction task based on attribute extraction from text game engines; and a set-of-sequence generation model that achieves good performance on the proposed tasks.
main_review: This paper proposes a novel knowledge graph construction task based on attribute extraction from text game engines. A set-of-sequence generation model, the Worldformer is proposed, which achieves good performance on the proposed tasks. I think this is a very clever way of creating knowledge graph construction tasks. And the two proposed tasks, on their own, are valid and interesting NLP tasks (though the human study might be needed, see my review at the end).

There are several major weaknesses this paper should addresses:

* Clarity

At first I did not find how the groundtruth graphs G_t and G_{t+1} are constructed. Then following the provided github repo I found this paper: https://openreview.net/pdf?id=Y1YtS9MZA75. Unfortunately, the dataset description paper still does not provide sufficient details. I guess the two papers are written by the same authors. If so, it would be good if more details about the dataset construction can be provided in the revised version. Specifically, how the tuples are extracted? Also why the average graph tuple number in a game is usually so small?

* The significance of the graphs prediction task

First, I did not see any theoretical or empirical justifications of the values of the proposed world model representation. Why this is the right world model for understanding the game texts? Is this world model better than simple sequence embedding of history? Why is it better than the model in [4]? The missing of the above answers makes the motivation of this paper not quite clear, as the addressed problem has not been well-verified yet according to the submission. 

Second, the proposed model does not show a clear advantage over the Q*BERT baseline. The baseline formulates the graph construction task as question answering, therefore it is not surprising the baseline does not perform well on graph metrics but works well on token metrics. This raises another problem of the task significance -- the used intermediate metrics cannot clearly show which model is the best. It would make more sense if the graph construction models were used to enhance the game-playing agents, and the agents' performance was reported.

* Empirical comparison

Related to the previous bullet, the compared Q*BERT baseline was proposed to enhance a game-playing agent and performed quite well. As a fair comparison, the proposed Worldformer should be compared in the same setting.

The submission cited the related work [39], which was also proposed for valid action prediction. I am wondering why the work [39] was not compared in the experiments? Did I miss anything?

* A minor question:

Are the textual inputs guaranteed to be sufficient for the two tasks? I think conducting a human study would be necessary to show the quality of the tasks and their upperbound performance.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method for modelling state (via graph prediction) and action distributions in text adventure games. To do the former, this work proposes predicting incremental graph updates instead of entire graphs. For both, the authors propose predicting sets of sequences as opposed to sequence generation and show that this improves the accuracy of predicted action sets and environment state.
main_review: This paper presents an original approach for modelling state and valid action spaces for text adventure games using pretrained language models. The work is clearly presented and well-motivated. My main concern has to do with the evaluation:

1. Is this estimation helpful for actual game-playing?
Given that the ultimate goal is to be more sample efficient due to having a good world model, I think it would be good to use this world-model to learn policies for these games, the the hypothesis being that using these models improve upon prior results. The numbers in Table 1 are all fairly low IMO and I am not confident that these results are actually helpful downstream.

2. I am unclear as to where the gains are coming from. Some ablations that I find missing include:
- QA & Seq2Seq with multitasking
- QA & Seq2Seq with set loss

3. What do the evaluation metrics mean? It is difficult to interpret F1 in the context of text adventure games. For example, what portion of the actions generated are actually parseable by the game parser?

Some questions for the authors:

- what are "lifted representations"? L38
- does this algorithm support removing nodes from the graph? some parts of the paper imply yes (L243) but the main exposition suggests no. L150
- is phrase-level masking actually better than normal LM masking? L165
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose a novel system that tries to learn world model --- represented as knowledge graphs --- for agents playing text-based adventure games. 

To learn the world model, the authors design two tasks:
- Knowledge graph prediction. More specifically, knowledge graph difference prediction. At a game step $t$, given the observation $O_t$, the set of valid actions $V_t$, the graph representation $G_t$, and an action $A \in V_t$, the model is required to predict the difference between $G_{t+1}$ and $G_t$, represented as a set of $<s, r, o>$ triplets. Using heuristics, the system deduces $G_{t} - G_{t+1}$ from $G_{t+1} - G_t$. Together, the set of graph difference (triplets) can be used to infer the full set of $G_{t+1}$, which represents the new game state $S_{t+1}$.
-Valid action prediction. Similarly, at a game step $t$, given $O_t$, $V_t$, $G_t$ and $A \in V_t$, the model is required to predict the set of valid actions $V_{t+1}$ at next game step. 

Since both of the above tasks are essentially set (of sequences) generation tasks, the authors propose a modification on the standard autoregressive NLG paradigm. Specifically, the decoder outputs a sequence of elements where each element is a multi-word chunk (either a  $<s, r, o>$ triplet or an action), the author propose to remove the dependency across elements so that when generating a word, the decoder is only conditioned on the previous tokens within the same element. From Figure 3, the authors show that such chunks are concatenated with special tokens (e.g., [ACT], [TRIPLE]). 

Another contribution is that the authors propose a multi-task architecture to jointly learn the aforementioned two tasks. Because the authors use triplets (sequences of tokens) to represent graphs, both the text encoder and graph encoder can be implemented as transformer-based encoders. The authors use an aggregator to combine the encodings from both encoders, denoted as $S_t$, which is later used to perform cross-attention with each of the individual encoding as the input to their corresponding decoder. 


main_review: Overall, this is a very interesting paper. The authors aim to learn a world model represents a model's understanding to the environment it is performing in. World model is a hot topic in model-based RL community, as cited by the authors, state prediction and action prediction seem to be a promising direction towards learning a "model". 

The authors tackle the problem by using a static dataset, JerichoWorld, a set of trajectories along with transitions at every game step collected from a set of text-based games. I like this choice because 1) it circumvents extra difficulties in the interactive settings so that they can focus on the world model learning part, 2) but it can also be seen as an offline-RL setting since the static dataset can be treated as what an online agent collected and cached into the replay buffer. 

I like how the authors formulate the world model as a graph --- there are prior works (such as Ammanabrolu and Riedl, 2019, Adhikari et al., 2020) suggest graph structures help text game agents as a memory and facilitate its planning. There is also a line of prior work focuses on representation learning in RL (e.g., Schwarzer et al., 2021), especially with contrastive predictive methods. This work in some sense brings the two lines of research together, explores how to learn a world model, in the format of a graph, by training with action and state prediction. 

I agree with the authors that the valid action prediction and state prediction tasks are set generation tasks. The authors attempts to modify the standard teacher-forcing paradigm to make it work better on such set generation tasks. 

The paper is clearly written, it is easy to follow. 

I will put my questions and concerns in two parts:
## General (maybe highlevel) questions:

**(Q1)** Are the inputs to the world model sufficient? To answer this question, please first check if my understanding to the model input/output are correct (as listed in the summary review). As mentioned by the authors (L29-L30), text-based game environments are often partially observable. If that is the case, how does one predict $S_{t+1}$ from $O_t$, $V_t$, $G_t$ and $A$? 

I can understand that within the current observation range, it is possible to predict the consequence of performing an action (e.g., take leaflet may cause the leaflet moving from mailbox to inventory). However, how does one predict something not in their observation range? For instance, in Figure 1, given all the information on left, plus the $A$ (go north), how to predict the right hand side information? How does the agent know what place is to the north, and what objects that place contains? I do expect a Seq2Seq model being able to memorize this from the training games, but I do not see how this can be generalized to unseen data points. 

**(Q2)** If the inputs are somehow sufficient, are they all necessary? In the knowledge graph generation task proposed by the authors, the model learns: $O_t, V_t, G_t, A \rightarrow \delta$ where $\delta = G_{t+1} - G_t$. I fail to understand why $O_t$ and $V_t$ is necessary here. Doesn't $G_t$ contain all information in $O_t$? Is this redundant information by design? Similarly, in the valid action generation task, can a model learn to deduce $V_{t+1}$ from $G_{t+1}$ only? If so, I could imagine a reasoning path of $G_t, A \rightarrow \delta  \rightarrow G_{t+1}  \rightarrow V_{t+1}$.

## Detailed questions:

**(Q3)** What reason is behind the choice of representing graphs as sequences of triples, compared to the (probably) more straightforward practice of adjacency matrix and Graph Neural Nets? Presumably, the model should learn an permutation invariant representation of set. However, there might be ordering biases introduced by the current way of encoding the set of triples? Did the authors do special modifications to the positional embeddings in their Transformer-based encoders (e.g., reset positions at the first token of each triple/action)?

**(Q4)** How is $A$ integrated in the model? I fail to find it from Figure 3. With Figure 3 alone, the inputs to outputs mapping seems impossible. 

**(Q5)** How does the proposed system learn to generalize to unseen objects and their affordances? Games in the Jericho collection can be quite different from each other, I wonder how do the proposed methods generalize to unseen and potentially very different games, where unseen objects have unseen affordances. It would be also nice to investigate deeper on the numbers shown in Table 1, what contributed to those numbers? What can be transferred across games and what not (i.e., what's an upperbound EM/F1 score for JerichoWorld)?

**(Q6)** I am curious to see a detailed analysis on the set of sequences (SOS) generation method. Ablation results in Table 2 and Table 3 suggest the SOS loss provide a non-trivial boost on performance, however, in experiments without SOS loss, do the authors sort the sub-sequences somehow? In Adhikari et al., they seem to suggest that sorting the sub-sequences are necessary in Seq2Seq learning. In addition, I am not intuitively convinced that cutting the dependencies between set members is the best way because there is arguably some kind of dependencies existing within a set (even to the least extent, set members should not be identical). 

That said, what we want is essentially permutation invariant, rather than removing dependency. Given how the SOS loss is designed, a simple failure mode I can imagine is that the model may have hard time generating the first token in each of the triples/actions, because they all condition on the same special token [TRIPLE]/[ACT], as a result, does this lead to more duplicates during inference? I suggest the authors to provide more information to make this more convincing. Also, Ye et al., 2021 seems quite related, might worth checking. 

**(Q7)** (minor point) It would be nice to see the learned world models are helpful in the interactive setting of text-based games rather than their static dataset version.

## Typos:
1. L47: that that.
2. L248: other other.
3. L260: Table 3 --> Table 2.

References:

1. Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning. Ammanabrolu and Riedl, NAACL 2019.
2. Learning Dynamic Belief Graphs to Generalize on Text-Based Games. Adhikari et al., NeurIPS 2020.
3. Data-Efficient Reinforcement Learning with Self-Predictive Representations. Schwarzer et al., ICLR 2021.
4. One2Set: Generating Diverse Keyphrases as a Set. Ye et al., ACL 2021. 
limitations_and_societal_impact: Yes, the authors have discussed limitations and potential negative societal impact. I do not see a problem here.
review_title: null

Meta-review: This paper jointly learns a policy (action distribution) and world model (updates to a graph representation of the environment) in a text adventure. While interesting, I'm not seeing the obvious application of this (despite having read the ethics impact section of the paper), and have some concerns about text adventure games as a domain — however the meta-review stage of the review process is not an appropriate point for me to bring these concerns to the authors' attention, so I will happily discount them and focus on the reviews and discussion.

The reviewer consensus leans towards acceptance, after some discussion. I would have liked to see a follow-up response from reviewer FHsa to the author rebuttal, and have no evidence that they have read it or considered updating their score despite my prompts, which is a bit disappointing. Less crucially, it would have been nice to get an author response to the follow-on comments by reviewer cdHj, but this is not as important since that reviewer updated their score to recommend acceptance.

Ultimately, when discounting the score of reviewer FHsa, the median and mean scores are in acceptance territory. I confess I do not fully understand reviewer FHsa's argument against the paper, and felt the author response was detailed. In the absence of this reviewer's willingness to defend their appraisal, I recommend acceptance.
Decision: Accept (Poster)