Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper considers the problem of interactive fiction games in which an agent interacts with the world purely through natural language. The problem is challenging because one has to map natural language as observations to appropriate representations to partially infer the state space of the world, and actions are also defined in vast space of natural language sentences. One of the main contribution in the paper is to represent the partial observations of the world as a knowledge graph, and so as to efficiently infer appropriate actions.

The paper is very well written, especially the introduction section, demonstrating novelty in the context of fictional games literature, and showing good empirical results.

However, I don't have any background in fictional games but dialog modeling. So it is hard for me to fairly assess how novel this work is. Ideas are simple and incremental, even if i rely upon literature overview provided by the authors in the related work section. Though, it can be effective.

The problem is challenging, and yet narrows down to fictional games. The proposed solution doesn't seem generic to be applied in other NLP or ML problems.

Authors argue that action space is super large even if generating sentences of length upto 5. Even though true, I think, this argument doesn't hold in the context of recent progress in NLP for problems like dialog modeling, where the action space of generating responses is even larger. I suggest the authors to relate their work to dialog models, as some of the ideas can be borrowed from there to simplify the solution for the considered problem setting.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper tackles the problem of developing agents to solve interactive fiction (IF) games. The authors propose an agent that builds a dynamic knowledge graph of each state from the textual observation provided by the games, while choosing actions from a template-based action space. While both these directions have been explored in prior work as pointed out, this paper combines them effectively to produce an agent that outperfoms existing methods on a benchmark of different IF games. 

Pros:
1. Writing is clear, method is easy to understand and design choices are clearly specified. 
2. Empirical results are good and presented on real IF games.

Cons:
1. (minor) While the authors test their method on a suite of human-made IF games, it would also be great to have a study on synthetic cases like the Microsoft TextWorld environments, if only to see which aspects of the method are crucial to making the jump from synthetic to real IF games. 


Other comments:
1. It looks like the knowledge graph is constructed for every state separately. The DRRN on the other hand incorporates more of the state history. From Figure 3(b), both DRRN and KG-A2C seem to do well - have you analyzed whether these methods are complementary? In general, it would be nice to have some more analysis on all the models across the different games rather than just Zork1.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a knowledge graph advantage actor critic (KG-A2C) model to allow an agent to do reinforcement learning in the interactive fiction game. Under the general framework of A2C, the core contribution of the paper is to apply a graph attention network on the knowledge graph to help learn better representation of the game state and reduce the action space. Experiments on Zork1 game environment are done to verify the effectiveness of the proposed method.

Overall, this paper presents a novel contribution to reinforcement learning with augmented memory/world-state. However, I do have a few concerns regarding the baselines and other details. Given these clarifications and or comparisons in an author response, I would be willing to increase the score.

Pros:

1, I like the idea of constructing the knowledge graph as the agent roll out. I think it is a better way to construct a structural representation of the world rather than assuming the agent gonna learn everything via single hidden state vector. It also permits more explainable policy in the future. Authors do make good progress along this line.

2, The paper is well written and the design of the proposed new model seems technically reasonable.

Cons & Questions:

1, The main concern I had is regarding to the baseline. I think it is more convincing to have a baseline which leverages the same entity extraction and template-action space. In particular, it should have the same model architecture except that it uses maybe a LSTM to decode the action rather than a GAT applied on knowledge graph. Note that the baseline I am referring to is different from the LSTM-A2C baseline reported in the paper as: (1) with entity extraction, although you may not get a graph mask, but you can still have a object-mask which also reduces the action space; (2) it is not clear to me that LSTM-A2C uses the same template-action mechanism as the KG-A2C, e.g., the valid action construction procedure described in section 4.1. Without such a baseline, it is hard to fully judge how helpful the knowledge graph is.

2, How do you test the generalization of the proposed models? In particular, do you use different maps during training and testing? If the model is merely trained on one map as shown in figure 5, it may just memorizes it in the knowledge graph and overfit to this map. 

3, The details of the interaction fiction problem setup are sparse. It would be very helpful to explain what exactly the observations are in the example of Figure 2. For example, what are the game description, game feedback are in this case? 

4, In the caption of figure 1, “Solid lines represent gradient flow” is misleading. If I understood correctly, solid lines refer to the computation flow which has gradient back-propagated in the backward pass.

5, Could you explain why KG-A2C converge slower than DRRN in figure 3?

6, Do you think having a fully differentiable mechanism of building knowledge graph would help or not? Why?

======================================================================================================

Most of my concerns are addressed by the authors' response. I increased my score.
review_title: Official Blind Review #1

Meta-review: This paper applies reinforcement learning to text adventure games by using knowledge graphs to constrain the action space. This is an exciting problem with relatively little work performed on it. Reviews agree that this is an interesting paper, well written, with good results. There are some concerns about novelty but general agreement that the paper should be accepted. I therefore recommend acceptance.
Decision: Accept (Poster)