Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies object-oriented text dynamics (OOTD) model and planning in text-based games. OOTD learns internal representation of object dynamics and uses transition layers to predict the belief of object states. Empirical results shows a performance boost compared to model-free baselines. Ablation studies are performed to explain the importance of OOTD components.
main_review: The motivation of the paper makes sense. The author starts with challenges in text-based games:
* text observation has variable length of sequence.
* text only partially describes the current state
* text could contain noisy information and it is challenging to identify salient terms.

General Significance: This is a difficult and interesting domain to study. The author proposes a reasonable extension based on OO POMDPs for text-based environment. The empirical results give strong evidence that this approach led to reasonable improvement in planning. 

Novelty: For my best knowledge, I think the method is a new contribution to the field. 

Technical Quality and Clarity: The paper is well-structured, with clear relationship between each section. The general ideas are clearly conveyed. I especially like Figure 1 and Figure 4. Figure 1 is illustrative and shows the high-level idea clearly. Figure 4 visualizes the object state embedding, which clearly shows that the OS-OOTD method was able to embed semantic similar objects (ingredients) in a near position.  But if the author can use more illustrative examples on explaining the technical details, it would be more readable for broader audience.

Typos:
* Figure 4: the title for the right-most image should be "SS-OOTD".
summary_of_the_review: marginally accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes Object-Oriented Text Dynamics for model-based RL in the TextWorld environment. OOTD ensembles existing methods such as ComplEx, R-GCN to build a memory graph, which contains object-level information about each object's attribute and its relations to others, from history observations and predicts states at the next time step. Therefore, the authors can combine the learned model with Dyn-Q or MCTS methods to select actions. If I am correct, under an object-supervised setting, it assumes the existence of a dataset that contains a ground truth memory graph. In contrast, the network has to discover graphs from predicting observation and rewards in the self-supervised setting. Experiments show that OOTD achieves superior performance than various baselines, including model-free RL and previously learned graph models.
main_review: Strength:

1. This paper studies the TextGames, a very interesting and exploratory direction, and sets up a new SOTA performance through the model-based RL methods. The success enriches the application of dynamic model learning and planning algorithms. It may encourage follow-up works along the direction of planning under high-level semantic information and inspire fields across visual language navigation and robotic manipulation.
2. Though there are existing works that train object-oriented dynamic models, applying them to text-domain requires non-trivial efforts. The paper demonstrates the importance of an object-oriented representation in text-model learning, which may help future research in learning better language models. If I am correct, the proposed method assembles various recent approaches, providing a very reasonable foundation for future researches.

Weakness

1. Though the paper does well in presenting the core idea of learning an object-oriented dynamic model, I met difficulties while parsing the architecture details. I believe that there is still a large room for improvement regarding paper writing. Here is my advice
    1. The overall structure is a little bit confusing. The paper seems to mix the problem setup with the proposed method. For example, the paper introduces the OO-MDP, graph representation, and model-based planning in Sec 2, and introduces the detailed transition model, reward model, and training methods, and additional modules in Sec 3. This causes the structure fragile. 
    2. For example, the paper introduces the concept of learning dynamic model in the section defining OOMDP and mentions a learned graph updater in the section defining memory graph. In section 2.3, it uses Table 1, which contains a lot of undefined terms, before introducing the planning methods. As a result, after reading Sec 2, I do not know what the graph updater is and whether the latent z is the ground truth state or a learned latent state. In fact, until now I do not understand the meaning of the graph updater as I can not find any strict definition about it.
    3. The training objectives section contains too much content. The authors put two different training setups, the way to construct the dataset, and several important network modules (observation encoder, decoder, and so on) in the last small section. This violates the logical flow and makes the section less coherent. The consequence is that I have to read back and forth to figure out what the networks actually learn and even go back to Sec 2 to get the meaning of variables. 
    4. While introducing the planning method and the transition model, the authors mention that "Given/define the states z". It is not proper, as z should be a learned latent variable and I think there is no ground truth information about z. In contrast, the authors should talk more about the observation, which is the real given input to the dynamic model. Currently, the observation encoder q_Epsilon is never formally defined and is missing in Figure 2. 
    5. In sec 2.2, the paper defines G_t, but it forgets to define h_t and oplus operator, making the graph updater unclear.
    6. It forgets to mention the distribution that the model uses. Are they all Gaussians in formula (5) and (7)?
    7. Section 3.1(c) lacks explanation. More importantly, it does not explain how it generates the two attention matrices Bs. Are they generated by the BIDAF networks? What is the input to that network?
2. It seems that the model requires knowing the object set before training the model. Even in the self-supervised setting, the model needs to use a pre-determined object set. I worry if introducing too much prior will hurt the generalizability of the proposed method in new language tasks.

Minor comments:

1. I don't agree that extending OO-MDP with partial observability is a significant contribution.
2. In Figure 4, there are two OS-OOTD in the title.
3. Regarding the performance of the OS and SS setup, the paper mentions that OS's inferiors are due to the inaccurate reward prediction. I am not fully convinced as the adding extra supervision should not hurt the performance too much. Instead, I suspect that the reward module is too simple. Only a single MLP may not be enough. I suggest authors do more study about the reward module to figure out the reason behind the bad reward prediction.
summary_of_the_review: I appreciate the author's effort in constructing such a complicated model to solve the challenging tasks, but the paper's writing has big flaws. Without adding enough details for a better evaluation, I do not recommend accepting this paper in its current form. I would be happy to increase my score if its writing can be improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the difficult problem of learning to play text-based games. This domain is particularly challenging, since the text observations received at each time-step are variable length, and will only provide a partial description of the world, from which the full state must be reconstructed. Additionally, the player must track the history of past states in order to make the correct decision.

This paper introduces a new framework for describing the time-evolution of a game: Object-Oriented Partially Observable Markov Decision Processes (OO POMDP). This work's main contribution is a learned Object-Oriented Text Dynamics (OOTD) model that learns the transition and reward function of a OO POMDP game.

Briefly, the presented method for predicting the transition function is: given a representation $z_t$ of the objects and $a_t$ of the action, create a graph, where the nodes are objects and the edges denote relationships. Perform message passing, and obtain new node representations $e_t$.

Then, (1) given $e_t$, predict a new representation $v_{t}$ using dual stream attention with the action $a_t$. Then, (2) use $v_t$ to predict the updated state $z_{t+1}$ of each object, using an independent set of parameters per object.

The dynamics model is trained using a Evidence lower bound objective (ELBo) in two varieties: one that relies on a deterministic extraction of the graph and one that learns directly from rewards and observations.

To show its effectiveness, the learned dynamics model is used to model-based training (Dyna-Q, MCTS) is used to learn a planner. The learned planner is shown to out-perform the baselines (DQN, DRQN, GATA), both on the test cases, and in terms of sample efficiency. Ablations of (1) and (2) find qualitative advantages of the proposed method, since the learned object representation are both more separable and better semantically clustered. Additionally, the ablations are shown to reduce the accuracy of the graphs and state recreated from the object representations.
main_review: # strengths and weaknesses
- strengths
  - Tackles a difficult problem
  - Presents a useful extension to OO POMDPs which can be used to model many text-based games
  - Presents good evidence that model-based training using their learned dynamics model is an improvement over the baseline (model-free learning)
  - Achievements in the domain of text-based games are important since they could lead to better conversational agents, which may be deployed in complex settings that require tracking conversation history, e.g., in IT, health
- weaknesses
  - Some technical details could use more explanation, which would improve reproducability and increase my confidence in the technical soundness of the work (see comments)
  - Although there are comparisons to model-free methods, it seems that a comparison with a model-based method is missing (table 2)? Could the authors comment on how they expect their OOTD approach to compare to, e.g., a simple RNN that takes text observations as input and outputs a list of object states. Are there no model-based baselines, since this work is the first in the text-based games domain? 

# Questions for clarification
- sec 2.2: Does $S$ contain all the information of $Z$? Is there a one-to-one correspondence between $S$ and $Z$? Is $Z$ completely recoverable from $S$?
- sec 2.2: What exactly is contained in the object level states $Z$? You extend Diuk et al 2019, which defines a set of object classes and attributes for each class. Are classes and attributes represented explictly anywhere in your framework? Or are the object states $Z$ hidden state vectors that are not constrained to take any particular form?
- sec 2.4: In equation 2, why isnt $\zeta$ indexed by the goal $g$? Shouldn't the tree-search nodes record the current goal?
- sec 3.1 (a): I think $Re(\omega_c, z_{i, t-1}, z_{j, t-1})$ needs something in between the parentheses. Perhaps something like $Re(f_{score}(\omega_c, z_{i,t-1}, z_{j,t-1}))$? Otherwise, do you mean that $Re()$ is being applied to the concatenation of three vectors?
- sec 3.1 (c): How big is the space of actions? It sounds like each action is identified by a (possibly lengthy) sentence. Did you have any trouble with exploration?
- sec 3.3 and A.3: What happens if the goal extractor finds multiple plausible goals in the memory graph? What if nothing valid is found?
- sec 4 table 2: why aren't the GATA baselines in the "model based planning category"? Don't they also model system dynamics?
- sec 3.3 equations 5 and 7: Can you explain how you arrive at your objective? I'm familiar with one way that evidence lower bound can appear: $\log P(X) - D_{KL}(Q || P)$. Can you give some insight into why the first term in equations 5 and 7 involves an expectation?
- sec 3.3 equations 5 and 7: Similarly, in the second term, can you explain the ordering of arguments in $D_{KL}(q_{\epsilon} || p_{\mathcal{T}})$ ? I would expect that since, $p_{\mathcal{T}}$ is the function being learned, it should be the first argument in the KL divergence, as is typically the case when using an ELBo objective. 

# Typos
- pg 2. typo? "to enable stochasticity in the dynamics" --> "to enable stochacisity in the training" 
- pg 6. typo: "is implement by" --> "is implemented by"
- pg. 7: "compassion" --> "comparison"
- pg. 7 bottom: sentence beginning with "3)" seems incomplete
- pg. 8: "object closed to each other" --> "close"
- pg. 8: figure 4 -- last two subfigures have same label
summary_of_the_review: Overall, I thought this paper was well written and presents a significant contribution in a novel area: modeling dynamics for text-based games. And the proposed extension to OO POMDPs will be generally useful for anyone working on text-based games. The given ablation studies and comparison to model-free baselines provides believable evidence for the effectiveness of their model-based learning approach. And while this work focused mainly on games, it's clear that their results could have useful applications in the field of conversational agents, which is growing in importance. I had some questions about the technical details of their method, especially the formulation of their objective. But these could probably be well answered in follow up discussion or in an appendix.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a novel approach for task-based games. The authors used an object-oriented POMDP formulation for the task. The key contribution is to learn the dynamics of the environment with a graph neural network (with either object supervision or self-supervision), and apply an online planning algorithm (e.g., MCTS) to solve the problem. The authors show strong improvements over other approaches in terms of sample efficiency and task scores.
main_review: Overall I found that the paper is clearly written, with helpful illustrations. My specific questions follow.

**Contributions**

1. I am not sure if "extension of OO-MDP to partial observability" is the contribution of this paper. For example http://h2r1.cs.brown.edu/wp-content/uploads/wandzel19.pdf
2. Abstract: dynamics models for image-based games are primarily on fully observable states -- I'm not sure if this is true. For example https://arxiv.org/pdf/1803.10122.pdf
3. Intro Paragraph 1: generalize to all states and actions -- I don't see how this is exhaustively evaluated in the paper, nor did I see why this is important. Technically, we only need to make predictions about states that we are interested to achieve (and this is also how the authors collected the data).

**Model**

It will be important to showcase or verify that the modeling of belief is important. I think the authors were slightly mixing two things together:
- The partial observability of an environment (e.g., the position of an object is unknown).
- The fact that at each step, the observation-only contains objects with state changes.

As the authors also stated, there is an implicit assumption about "other object states do not change." This is a very strong assumption and can make model learning much easier, as in the STRIPS representation. See also https://en.wikipedia.org/wiki/Frame_problem 
Actually, under this assumption, we don't have to treat it as a POMDP (instead, just an MDP). The authors should make the significance of their POMDP formulation more clear.

I would also suggest authors try an ablation study with the belief prediction part removed (i.e., do not use sampling in the transition model).

On the writing of Section 2 and 3, I suggest that authors include ALL important modules in the paper (for example, the goal encoder). Some of the details are just echoing existing work, for example, OO-POMDP, MCTS, and can be potentially shortened. The writing can also be improved by incorporating a concrete running example showing the dynamics and the decision made.
summary_of_the_review: The paper is well-written and the results are strong. I have a few comments on the formulation and presentation of the paper. I also suggested an ablation study.
review_title: null

Meta-review: This manuscript makes an interesting observation: there is no reason why planning-based methods like MDPs must be limited to physical or grounded environments. One can plan about more abstract textual domains. It adapts the standard methods from planning to such text domains in a fairly straightforward way. The fact that concepts from MDPs map to these problems directly is an asset: ideas could flow between these domains in the long term. While the original submission was lacking clarity and significant technical details, the authors engaged with the reviewers and resolved lingering concerns. Reviewers are unanimous that this a strong contribution.
Decision: Accept (Poster)