Learning Object-Oriented Dynamics for Planning from Text
Authors: Guiliang Liu
Ashutosh Adhikari
Amir-massoud Farahmand
Pascal Poupart
Venue: ICLR 2022 Poster (2022)
Keywords: Object Oriented Markov Decision Process
Reinforcement Learning
Model-Based Planning
Text-Based Games
Knowledge Extraction

Abstract: The advancement of dynamics models enables model-based planning in complex environments. Existing dynamics models commonly study image-based games with fully observable states. Generalizing these models to Text-Based Games (TBGs), which commonly describe the partially observable states with noisy text observations, is challenging. In this work, we propose an Object-Oriented Text Dynamics (OOTD) model that enables planning algorithms to solve decision-making problems in text domains. OOTD predicts a memory graph that dynamically remembers the history of object observations and filters object-irrelevant information.  To facilitate the robustness of dynamics, our OOTD model identifies the objects influenced by input actions and predicts the belief of object states with independently parameterized transition layers. We develop variational objectives under the object-supervised and self-supervised settings to model the stochasticity of predicted dynamics. Empirical results show OOTD-based planner significantly outperforms model-free baselines in terms of sample efficiency and running scores.

Published as a conference paper at ICLR 2022
LEARNING OBIECT-ORIENTED DYNAMICS FOR PLAN- NING FROM TEXT
Guiliang Ashutosh Ashutssh Adhikari' Amir-massoud Farahmand Pascal Poupart Uniivrsitty of Waterloo, UUniversity of Toronto, Vector Institute, *Microsoft
92331iu@wwaterloo ca, aadhikari weahhkkaimmccrooottt.coom com, Faralmand@recttoiisstte ai, ppt.ppptttttttterr....oo oapp://eeee..........oooppooeoo.
ABSTRACT
The advancement of dynamics models enables model-based planning in complex environments. Dynamics models mostly study image-based games with fully observable states. Generalizing these models to Text-Based Games (TBGs), which often include partially observable states with noisy text observations, is challenging. In this work, we propose Object- Oriented Text Dynamics (OOTD) model that enables planning algorithms to solve decision-mane maaiing problems in text domains. OOTD predicts memory graph that dynamically remembers the history of object observations and filters object-irreleant information. To improve the robustness of dynamics, our OOTD model identifies the objects influenced by imput actions and predicts beliefs of object states with independently parameterized transition layers. We develop variational objectives under the object-supervised and self-supervised settings to model the stochasticity of predicted dynamics. Empirical results show that our OOTD-based planner significantly outperforms model-free baselines in terms of sample efficiency and running scores.
INTRODUCTION
Planning algorithms typically leverage the environment dynamics to solve decision-making prob- lems (Sutton & Barto 2018). To plan in unknown environments, the agent must learn a dynamics model to predict future (belief) states and rewards by conditioning on an action. This dynamics model enables the implementation of intensive search for optimal actions, which can potentially increase both sample efficiency and cumulative rewards, compared to model-free methods (Hafner jet al. 2019 Wang et al., 2019 Kaiser al.) 2020). Despite the promising performance, learning dynamics model that accurately generalize at test time is still challenging, especially when handling a eigh--imensional state space for low-level features, c.g., pixels and text.
To facilitate dynamics learning in complex environments, Diuk et al. (008) proposed Object- Oriented Markov Decision Process (OO-MDP) that factorizes world states into object states. They showed that the agent can find optimal policies with better sample efficiency by modeling the dynamics at the object level. Some following works (Finn et al. [016)] Goel et al.) 2018 ZZu al. 2018 2020) extended OO-MDPs to image-based games. These methods typically assume fiall observability over game states and a fixed input size, which facilitates the use of object masks to decompose an image into different objects. On the other hand, in Text-Based Games (TBGs), the text observation, whose length runs from few words to an entire paragraph, is a partial description of the current game state, and thus each observation provides information about a limited number of objects (in fact, only an average of 4.51% % of the candidate objects are mentioned in an observation from Textworld (Côté et al. 2018)). The dynamics model must remember the history of observed objects to predict accurately their states. Moreover, an observation typically contains lots of noisy patterns that record iireeeatt information (c.g.. the non-bolded text from o, in Figure [T. While previous works (Ammanabrolu & Riedl] 2019; Ammanabrolu & Hausknecht (0200) designed rule-based heuristics to extract useful information, we expect that dynamically capturing object information from noisy observations helps to generalize more environments.
In this work, we design an Object-Oriented Text Dynamics (OOTD) model that integrates:
1) Graph Representation for Objects. OOTD predicts memory graph from the input states. The memory graph, whose nodes correspond to the objects to be modelled, captures the information about these objects from the beginning of game. By applying this graph as information bottleneck for

---

Published as a conference paper at ICLR 2022
dynamics prediction, OOTD filters the object-irrelevve information in the text inputs, which forms tighter information plane and facilitates generalization to unseen games (Tishby & Zaslavsky 2015).
2) Independent Transition Layers. Intuitively, input action influences the states of limited number of objects, for example, when the agent performs "take the carrot from the counter", only the states of "carrot" and "counter" should be updated. To capture which objects influenced and keep others invariant to this update, OOTD learns an action-aware representation for each object with bi-directional attention and predicts the belief states of objects with independently parameterized transition layers. This independent mechanism enhances the robustness for dynamics prediction (Goyal et al. 2019) (also see the ablation study in Section 4.2
During planning, we predict belief states without knowing rewards and observations. To learn a stochastic dynamics model, we propose supeec-su and self-supervised Evidence Lower Bound (ELBo) objectives for training our OOTD model. We evaluate how well the OOTD model supports planning by implementing planning algorithms (e.g.. Dyna-Q (Kuvayev & Sutton| 1996) and Monte Carlo Tree Search (MCTS) (Kocsis & Szepesvári (2066) based on learned dynamics. Empirical results show that these planning algorithms significantly outperform other baselines in terms of game performance and sample efficiency. To support the design of our OOTD model, we include an ablation study that visualizes the object states and quantifies the dynamics prediction performance.
Contributions: 1) We introduce our approach to implementing model-based planning algorithms for solving decision-making problems in the text-domain. 2) We propose the OOTD model that learns to predict beliefs about object dynamics.
approech (lrd ingredients: banana, block cheese, carret directions dics the banans, fry the banans,chap I'e black ol cheess, roast the block of cheess, alice the carrol fry the carrol, and prepare meal.
Welcooe
spen capy of "cooiiing gatheral fnlowing
direction
thes
take from the counter.
Examine cookbook
Take banana from countar
ac-1
enty
ar
Oc+1
Controlling TBG
Planning in object space
Planner
Oc
cheese
7,,--1
Zi,r
Zize1
carrot
Zz,1
22,041
banana Z3,t-1
Encode o, and a With (e2. 92 92)
Zz,t
Plan with Transilion (p). pP)
Z3,441
and reward Models
Figure 1: An example of planning from text. We show three objects (cheese, carrot and banana) in this example. When interacting with the environment (env), the agent encodes the information of observation o into object states with encoders (gg, 4.. gg). Given a goal 9t, the planner determines the action de by searching with the reward (p,) and transition models (pT PT pT).
OBJECT LEVEL PLANNING IN TEXT DOMAINS
In this work, the object-level dynamics are learned and evaluated in Text-Based Games (TBGs), for which we extend O0-MDPs (Diuk et al. 2008) to capture partial observability. We define graph representation to remember object information and introduce the corresponding planning algorithms.
2.1 TEXT-BASED GAMES
Text-Based Games (TBGs) are complex, interactive simulations where agent explores and plays game by processing text observations and responding with text commands. To study the generalization ability of this policy, many recent works (Yin & May 2019 Hausknecht al. 2019 Ecoffet et al. 2019; Adolphs & Hofmmann| 2020 Adhikari et al. 2020) proposed to learn policy by training with set of games and evaluating with games from hold-out test set. The distributions of rewards over state-action pairs are different in training and testing games because TBGs assign rewards by conditioning on specific goals (that are predefined, but unknown). In this setting, TBGs have different intermediate goals in each game, but they commonly share the same ultimate goals, and the underlying causal dependencies leading to this ultimate goal are consistent. For example, to solve the 'First Text- World Problems' (Côté al. 2018). an agent should always gather and process cooking ingredients (c.g.. "fry the potato") according to a recipe that discovers in the game, although the names and locations of the ingredients are different across different games. This property enables the generalization of knowledge learned from training games to solve testing games.
2

---

Published as a conference paper at ICLR 2022
2.2 REJECT-ORIENTE PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES
Since an observation reveals only partial information about the game state, we formulate TBGs as partially observable Markov decision processes (POMDPs). In this work, we extend the POMDP framework to represent object-level dynamics (Diuk et al. 2008). The Object-Oriented (OO) POMDP is a tuple (S, Z, 4, g, A, R, T,7). 7), where:
SS and O define the spaces of low-level states and observations from the TGB environments while Z and & are the spaces of object-level states and observations. In a TBG, o is a text message from the game environment (e.g., "Please fry the potato."), whereas ø & records the specific objects and their relations in this message (e.g., a triplet like ppotato-ned-ffr"); To model the object dynamics, the agent must extract o from by distilling objects' information from text sentences. In this work, we model object states 2 E 2 (instead of & E S) and learn latent representations for 2.
G and A are the spaces for goals and actions. We study choice-based games (Yin & May 2019),
where candidate actions A, E A are available at time step t. We include goal variable g to mark different tasks in each game. Following the Universal MDP (UMDP) (Schaul et al. 2015), the agent initializes a goal at the beginning of game and updates it when the task is finished.
R. and T define the spaces of reward and transition models. is a discount factor. We assume the real dynamics models (py and p,)) are unknown, so we learn the object-orientn transition model and the reward model, i.e., Ak E {1...., , K}, pt (2 1) E T and p- (re 1 E R.
In our OO-POMDP, transitions pT. observations o, and states are modelled for each object, whereas actions a and rewards r are defined for the entire environment. This generalizes to popular RL environments that accept an action and return reward at every time step. Unlike (Wandzel et al. 2019) that represented object states with symbolic attributes, we develop a latent object representation that can generalize to the complex environment with high-dimensional inputs (e.g., text).
2.3 GRAPH REPRESENTATION FOR OBJECTS
We utilize an object-relaae refectt-elationnojjeee triplet to represent object information. time step 1, de [or, dt.M] (M is the number of observed triplets at t). Given total of K candidate objects and C' candidate relations, these triplets can be mapped to a knowledge graph P2 E {0. [0x1cCxXK Each entry (c, i, E {0, 1} indicates whether there a relationship r between the ith and jth objects. This knowledge graph forms a natural representation of object relations since object information in most observations on corresponds to either entity attributes (e.g.. "potato-is-fried or to relational information about entities (e.g.. pptato-on-coonter" and "bedroom-north of of-kitchen"), which can be conveniently represented by triplets and the corresponding graph.
Memory Graph. We store the observations up to time step in memory graph h, S2g + Q t
... + 24 that captures object information observed by the agent since the beginning of a game. Similar to previous works Adhikari al. (2020): Ammanabrolu & Riedl (2019); Ammanabrolu & Hausknecht 2020), we summarize the object-level history with latent memory graph lr. To update t. we learn a graph updater @ and h4 = h @ 22, During updating. ① needs to resolve some semantically contradictory triplets, for example sedeyer-at-edroooe
the memory from
player-at-kitchens and (because the player cannot simultaneously be in two different locations). Our OOTD model is trained to automatically emulate such an operator. Our transition model (Section 3.1 trained to generate h4 from 2t, allowing the latent object states to capture object relations.
2.4 MODEL-BASED PLANNING IN TEXT-BASED GAMES
We introduce model- planning in TBGs. Based on the 10-POMDP (Sec- tion 2.2), at each time step f, we define
Table Object dynamics for planning, where the transi- tion models and observation encoders are independently parameterized for a total of K objects.
latent states Zt = [I., [ for a total of K objects, text observations or, action commands a,, and scalar rewards rr, that follow the stochastic dynamics in Table I The details of these dynamics models introduced in Section 3
[pt (e0,011 2x K
Transition models
k=]
Observation encoders Reward model Graph & Obs. decoder
[ (2M,041000411 at, 2x)] k=1
pr (re(reze 9r) pa(he (2), Po (or12z))
3

---

Published as a conference paper at ICLR 2022
Based on these dynamics, we implement planning algorithms select an action from candidate commands Ag for maximizing the expected sum of rewards E. (E y²rr), shown in Figure In this work, we study choice-based games (Yin & May 2019 where the candidate commands (or actions) Ag are available and the planner determines the action a A, be performed. The planning algorithms include Dyna-Q. Monte-Carlo Tree Search (MCTS) and their combinations:
Dyna-Q (Kuvayev & Sutton 1996) incorporates dynamics models and Q-learning. Dyna-Q inter- actively 1) updates the dynamics model with observed transitions and 2) trains the Q network to minimize the Temporal Difference (TD) loss (Equation IT based on both observed transitions from the environment and predicted transitions from the dynamics models. Compared to the model-free Deep Q-Network (DQN), Dyna-Q more sample efficient: by expanding the replay buffer with dynamics models, Dyna-Q converges faster with the same number of interactions from the environment.
LID = E [(re + y max Q a, - Q(z(-1 04-1, 9c-)) ag,r,
MCTS (Kocsis & Szepesvári 2006) is a heuristic search algorithm that builds and updates search tree based on environment dynamics. By performing Monte-Carke simulations organized in a tree structure, MCTS does an efficient search in environments with large action spaces (Silver et al. 2018). MCTS iteratively runs multiple playouts. At the jth playout, we implement 1) Selection: Traverse the tree from the root node to a leaf node (corresponding object state ) by selecting the action command ai to maximize the Upper Confidence Bound (UCB) (Coučtoux et al. 2011:
Vlog(i 1)
Oz,t arg max (Qi(zz, a. + Cpuct S1-1 a.g) + aeA,
where Cpuct controls the scale of exploration, and Si records the visit count at the 2th play. 2) Evaluation: Evaluate the selected leaf 2, with the reward model pr (r>llzz..))) gt). Expansion: Expand the leaf node by adding child nodes. 4) Back Up: Update the action-values: Qi (Qi-1G--1
(r)/((;-1 + 1) and increment the visit count: Si = Çi-1 + on all the traversed edges.
Dyna-Q + MCTS initializes Q values in MCTS by utilizing the Dyna-Q network. This combination enables MCTS to perform the tree search with more efficiency and efficacy (Silver et al.) (0166).
OBJECT-ORIENTED DYNAMICS MODELS
We introduce the Object-Oriintedd Text Dynamics (OOTD) models, including 1) a transition model that directly predicts the belief states of objects and 2) reward model that maps sampled object states rewards. Based on these models, the planner determines an action purely based on predicted beliefs and calibrates its belief with the feedback (observations and rewards) from the environment. We introduce the object-supervised and self-supervised objectives to train these models.
BIDAF
ar-1
Network
Attentions 8:t
MLP
(Hli o²,) nut
ComplEx Graph
RGCN Graph
Zg
h
MLP
Muhr Zus
Mean Pooting
Decader
Encoder
Zgt-1
MLP
x (ulcn zRn
9:
Object-Oriented Transition Function
Reward Function
Figure 2: Left: The Object-Oriented Transition Model. Right: The Reward Model.
3.1 OBJECT ORIENTED TRANSITION MODEL
Figure Dillustrates our transition model, which has three major components (a) graph decoder, (b) graph encoder, and (c) independent transition layers.
(a) Graph Decoder, Given the states of K objects 2,-1 = [2 1-11. ..... 2x, we implement a graph decoder with the ComplEx scoring function (Trouillon et al. 2016) that maps the object states to memory graph h E 00, 1] Kx The scoring function for an entry (c, j, (in the adjacency matrix of h4-1) implemented by Re( (We, 21,--1 2j,) where w. CE is complex vector for
4

---

Published as a conference paper at ICLR 2022
the relation and Re denotes the real part of the decomposition in complex space. The function can model a large variety of binary relations, including both symmetric and antisymmetric relations. Based this scoring function, predicting 1 is equivalent to estimating the probability of having relation e between object and j. which can be efficiently approximated with a low-ranked (E < K) matrix decomposition: h2 = [e ZW; z") ) ,..,, Re ReeZWcc Z 2 where Z E RxX is the matrix of object states and W. C Bx E is complex matrix. In practice, this decomposition enables to map efficiently from - to ft therefore scaling our method to environments with numerous objects.
(b) Graph Encoder. We encode the memory graph h2-1 into node representations 6t-1 (1.1-11- (x,-- with a Relational Graph Convolution Network (R-GCN) (Schlichtkrull et al. 2018). Nodes in feg correspond candidate objects and RGCN performs message passing between nodes, so Ek,I-1 captures the semantic information of the h object and its relations to other nodes (objects). Compared to 2t-1, Ct-1 denotes the object embeddings after taking into account relations to other objects in the graph. The graph can be thought as succinct latent representation that ignores irrelevant object information in text observations when predicting the next state žt.
(c) Independent Transition Layers. TBG environments typically contain a large population of objects, and each action can only affect a few underlying objects. In order to improve the robustness of the dynamics, we utilize 1) Bi-Directional Attention Flow (BIDAF) network (Sco et al. 2017) to identify the affected objects and 2) a group of transition layers to predict the belief state of objects by following the Independent Causal Mechanism (ICM) framework (Pearl 2009).
Given e and an action a = [01.t_1,...._, [Ja=1 (a sentence of J words), the BIDAF network learns an attention matrix Bue E [0, 1] where by.j measures the similarity between the represen- tation of the k-th object and the embedding of the j-th word. To learn action-aware representation of object, we compute 1) accion-to-objed attentions bo that indicate which object representations have the closest similarity to one of the action words and are hence critical for modelling the impact of the action words; 2) object-to-action attentions by that indicate which action words are most relevant to the 4tth object representation. The action-awaa aware object representation concatenates the attended action vector V the attended object representation v and the initial object representation
1 V Σ, by 66 (aj,0-1) where b softmar B
[0,
vf-1 E, bEEK11 where b softmoz (Be)) [0, 1]K,
Pf (2k.)aa--, 21 = (Iu,,, where [Ju,,t, [x,e] = v (, v ex,--1))
(3)
Here 1 is transformer-baseee text embedding function (Vaswani et al. 2017) for the input action, and vi is implemented by a MLP. The transition layers wi...., 1/K are parameterized independently.
3.2 OBJECT ORIENTED REWARD MODEL
We learn a reward model pr (ri (t) based on the predicted belief states. This reward model is important for learning goal-conditioned policies that can utilize the knowledge learned from task to solve similar tasks, which accelerates learning and facilitates knowledge generalization.
To implement the reward model, we build goal extractor (Appendix A.3) that extracts goals ge from object states 2, [21.s- ... 2.. The goal is described by sequence of words and symbols that follow the triplet format, e.g., "[potato, -, need, fry]" where "_" serves as separator between object and relation mentions. Although z, implicitly embeds 9r with latent values, we prefer to represent ge explicitly with sentence for the ease of model comprehension. Given the goal gr and object states z,, the reward model p, is defined by
(r(ri/=0,90 9t) v/ [=!"nr 16.2 (g6)) where 2n mean (2,, =K.]):
(4)
2h is transformer-based text embedding function for goals, and v² is implemented by MLP.
3.3 TRAINING OBJECTIVE FOR DYNAMICS MODELS
In this section, we introduce the objectives for learning the transition model and the reward model under the obpect-suerriise setting and the self-supervised setting.
Object-Superviset Setting, We learn the dynamics model under the supervision of memory graphs hy that record rich information about the observed objects and their relations from the beginning of
5

---

Published as a conference paper at ICLR 2022
game (Section) 2.3 The Object-Supervie Evidence Lower Bound (OS-ELBo) objective is:
Σ Rus log po(((a
Σ DEa (Ek, a1- -1-) 10-) (5)
where ps: is the graph decoder, pJ is the transition model and of is the observation encoder. We now provide more details about each component and how to obtain h4s Pq is approximated by re-sampling technique described in Appendix A.2 PJ is a Gaussian distribution as described in Equation dg shares similar structure to PT (see Appendix A.5 for more details).
The memory graph h, is not a direct output from the RL environment. To perform object-supervised training, we extract object information from o, and predict hn with a deterministic object extractor fe(or, Gc-1, fe 11- We learn f. during pre-training stage by utilizing FTWP dataset. The dataset records the trajectories [01. ........ Or, as} and corresponding [/a,..... he] by following walkthroughs in FTWP games. In practice, we can build such a dataset by extracting triplets (ω) from the text observations (o...... o₂) with an Open Information Extractor (OIE) (Angeli et al. 2015) and map the captured triplets into a knowledge graph by following (Ammanabrolu & Hausknecht, 2020). After training fe, we initialize ho - {0] [0)cxx* and update by h12 = felor, ar- /1 during training. The detailed introduction of fe and the FTWP dataset can be found in Appendix A.4 A.6
To train the reward model, we sample object states 2t = (21. =K,, from their belief states, extract goal gt, and minimize smooth L1 loss (Girshick 2015):
23 Pr (Tr F1, 9.) - r2]3, (re - -rel }, otherwise.
if (p.(re 96) - rzl < B,
(6)
Self-Supervieed Setting. In order to generalize our dynamics model to common RL environments where the predicted or ground truth memory graphs are unavailable, we develop a Self- Supervised Evidence Lower Bound (SS-ELBo) objective that directly learns from rewards and observations:
K
Eg logpo(o.(zz) + log pr (relze,, 9:)
Σ Dui (2 a1-1, 21 21 (*k, Z 21-1)
where PT. Pr and qe are the aforementioned transition model, reward model and observation encoder. The observation decoder po is implemented by a Sequencc-to-Sequee
(Seq2Seq) model (Sutskever a (014), which is trained by teacher-forcing technique with text observations Ot- This SS-EBLo objective enables training the reward and transition model together by maximizing one objective, which improves the training efficiency. Appendix C.T shows an example of a predicted graph.
4 EXPERIMENTS
Environments. We divide the games in the Text- World benchmark (Côté et al. 2018 into five subsets according to their difficulty levels. Each subset contains 100 training, 20 validation, and 20 testing games. These subsets are mutually exclusive. For the easier cooking games, the recipe requires a single ingredient and the world is limited to single location, whereas harder games require an agent to navigate a map of multiple locations to collect and appropriately process up to three ingredients. Table 4 summarizes the game statistics.
Running Settings. To efficiently collect samples from the games in the training dataset, we train our OOTD model by implementing the Dyna-Q algorithm (Algorithm Min Appendix) due to the fact that 1) --ntwwrrk supports batch input, which enables solving and gathering data from multiple games in parallel, and 2) the Q-based algorithms provide a fair comparison of sample efficiency with other model-free baselines. After training, we fix the parameters in dynamics models and predict belief states and rewards to solve the games in the test dataset. During testing, we study the options of applying different planning algorithms including Dyna-Q. MCTS, and their combination (Dyna-Q+ha) Q+MCTS) that initializes MCTS with values from Q networks (Algorithm) 2 in appendix). The model dyper-parameterr are tuned with the games in the validation set. (Appendix A.1
Baselines. we compare the state-od of the-an art models for solving TBGs. The baselines include Deep Q-Network (DON) (Narasimhan et al. 2015) and Deep Recurrent Q-Network (DRQN) (Yuan et al., 2018). DQN selects the action based on the current observation, whereas DRQN models the history of action and observations with an RNN. For fair comparisons, replace their LSTM--aased text

---

Published as a conference paper at ICLR 2022
encoders with transformers. We compare extension of DRQN by using episodic counting bonus to encourage exploration, which denoted as (DRQN+) Yuan et al. 2018). Our next baseline is KG- A2C (Ammanabrolu & Hausknecht 2020), which builds graph-based based state space and applies the Actor-Critic algorithm for learning the policy. We replace their action generator with an action selector for comparison. The last baseline is GATA (Adhikari et al.] 2020) that learns a graph-structured representation to model the environment dynamics and select actions based on this representation. We experiment with several options of GATA, including: 1) GATA-GTP that pre-trains the discrete graph updater with ground ground-truth graphs from the FTWP dataset (Appendix A.6). Note that we apply the same pre-training dataset for our OS-ELBo objective. 2) GATA-OG that learns continuous graph updater by reconstructing the text observation. 3) GATA-COC that trains the graph updater with a contrastive observation classification objective. Both GATA-OG and CATA-COC follow the self-supervised setting, which offers fair comparison to our SS-ELBo objective.
Table 2: Normalized testing scores and averaged improvement (f) over DQN in six difficulty levels (0 to 5). For each method, implement three independent (check random seeds in Appendix A.1 We select the pop-performing agents in validation games and report their testing scores.
Type
Model
2
DQN
90.0
62.5
32,0
38.3
17.7
34,6
DRQN
95.0
58.8
31.0
36.7
21.4
27.4
-0.8
Model-
DRQN+
95.0
58.8
33,0
33.3
19.5
30.6
-0.8
Free
KG-A2C
96.7
55.5
31.0
54.3
26.8
30.1
+3.2
Algorithm
GATA-GTP
95.0
62.5
32.0
51.7
21.8
23.5
+1.9
GATA-OG
100
66.2
36.0
58.3
14.1
45.0
+7.4
GATA-COC
96.7
62.5
33.0
46.7
25.9
33.4
+3.9
OOTD learned by the Objeet-Supervised (OS) ELBo Objective
OS-Dyna-Q
100
62.5
42.0
58.3
21.8
48.2
+9.6
OS-MCTS
95.0
77.5
56.0
63.3
24.9
42.9
+14.1
Model-
OS-Dyna-Q + MCTS
95.0
78.8
57.0
71.7
27.7
38.1
+15.5
Based
OOTD learned by the Sell-Supervised (SS) ELBo Objective
Planning
SS- SS-Dyna-Q SS-MCTS
100
62.5
48.0
53.3
30.5
47.0
+11.0
100
70.0
51.0
70.0
27.3
54.4
+16.3
SS- SS-Dyna-Qu + MCTS
100
81.3
56.9
75.0
31.4
58.4
+21.3
4.1 CONTROLLING PERFORMANCE IN TBG ENVIRONMENTS
Table 2 shows the agent's 's controlling performance at different difficulty levels. Equipped with the OOTD model, our agent implements model-based planning and achieves leading performance over the agents based on model-free algorithms. We find the SS-based OOTD models outperform the OS-based models by scoring more rewards at 4 (out of 6) difficulty levels. Although the object signals facilitate better prediction of dynamics, this advantage can not be directly transferred to controlling performance. This is commonly known the jnfeccive-mismatte
problem in model- based RL (Lambert et al. 2020). On the other hand, SS-based models are directly optimized toward generating better prediction of rewards, which are important signals for controlling. The reward- irrelevant information is filtered to derive a more compressed representation for better generalization ability. Another important finding that our OOTD-based agents perform better than other graph- based agents (e.g., GATA-GTP and KG-A2C). This is because OOTD is doing model-based RL by learning the dynamics the object level while other graph-based techniques are model free since they learn to extract object representations, but not to predict their dynamics.
Sample Efficiency. We empirically demonstrate the sample efficiency of our model-based agents by comparing their training performance with other model-free -reee agents. This experiment studies three variations of proposed methods, including 1) OS-Dyna-Q and 2) SS-Dyna-Q Q that implement the Q-Dyna algorithm and train the OOTD models with the OS-ELBo objective and the SS-ELBo objective respectively. 3) Object-Oriented (OO)-DQN that trains the OOTD model to predict object states, based on which we compute action values. As model-free method, OO-DQN does not expand the replay buffer with the samples generated by dynamics models. Figure shows the training curve for 6 difficulty levels. The learning speed of proposed model- algorithms (OS-Dyna- Q and SS- SS- SS-Dyna-QL Q) is generally faster than that of MF algorithms, but we observe an exception at the games of difficulty level 5, where GATA-OG converges faster. We find it generally takes more
7

---

Published as a conference paper at ICLR 2022
samples to learn the interactions among K objects and their relations action values. It explains why (MF)-Dyna-Q converges slower than others. However, the object- based value function performs better than that based raw observation in terms of normalized scores.
Officuity Lovel
Dfficclty Lovel
Diffcuity Lowel
Homakoedd Soore
Normat CARme Scoree
DON
0.40
GATA-GTP
0.33
GATA-OG
0.30
GATA-COC
OC-DON
S5-Uyna-4
13c
706
VM
NK
P2C
10g
70C
30C
10L
456
Ffx
T0c
R2K
OS-Dyna-Q
Epleode
Epeedd
Egledd
Officulty
Diffouty
0.25
urd Glate Scores
0.30
DRGN
GATA-GTP
044
0.20
GATA OG
GATA-COC
0:15
DC-DON
55-Dyna-Q
05 OS-Dyna Q
Episode
Episode
Episade
Figure 3: Training Curves: Agents' normalized scores for the games at different difficulty levels. The
plot shows mean + sld normalized scores computed with three independent runs.
4.2 ABLATION STUDIES FOR OBJECT-ORIENTED DYNAMICS
To understand the importance of OOTD design, we empirically study the options of 1) removing the BIDAF attentions by directly applying the action and node embeddings to predict object states (i.e.. No-Attentions Dynamics (NAt-Dyna)) and 2) removing independent transition layers by building single layer to predict states for all objects (i.e.. SingleLLayer Dynamics (SLa-Dyna)). We compare these models with our OOTD models trained by the OS-ELBo objective (OS-OOTD) and the SS- ELBo objective (SS-OOTD). To evaluate how well the modified model přy, captures object dynamics, we sample object states k,,t ~ (2t. 2t-1) for total of K objects, visualize these states with the t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm (Van der Maaten & Hinton] 2008), and quantify the dynamics prediction performance.
Object States Visualization. In this experiment, we randomly pick 20 games from the testing games of all difficulty levels. Within each game, py,, computes the states 28,1, , Ar,T with randomly selected actions 00..... ar-1. and t-SNE embeds 2k,t into a vector of 2 dimensions. Figure visualizes the embedded vectors labelled by 10 randomly selected objects. The object states computed by our OOTD model are distinguishable, and thus t-SNE embeds the states for the same object close each other. This phenomenon is not observable when apply single layer for computing object states (i.e., SLa-Dyna). Another important observation is that the OOTD model automatically learns generate similar states for similar objects. For example, OS-OOTD generates similar states for the object of ingredients (e.g.. parsley, apple, carrot, and potato marked by the dark frame). similar phenomenon, although less obvious, can observed for SS-OOTD, However, such a shrinkage effect disappears in NAt-Dyna, which explains the importance of bidirectional attentions.
SLA-Dynes
NAI-Dyna
05:00TD
SS-OOTD
player
kitchen
table
comkbenk
tiige
parsley
red apple
yel
Figure 4: Scatter plots for T-SNE-embedded object states. From left to right, the states are sampled from SLa-Dyna, NAnaDyna, OS-OOTD and SS-OOTD (from left to right).
Probing Experiment. This experiment studies how well our OOTD models can capture useful information from the TBG environments. We map the sampled object states 21,4 FK,t to memory
8

---

Published as a conference paper at ICLR 2022
graphs h₂ and rewards é with the additional graph decoder P/G and reward model p/.. The performance of the dynamics models is quantified by whether he and fg can match the ground-truth hi and rī in the FTWP dataset (Appendix A. During sampling, the input actions .1,.... Or follow the recorded trajectories in FTWP. The ground-trume -rouund-ruuthh/n rewards in FTWP games either one or zero, and we perform an entry-level evaluation (e.g., ht j. c) on the adjacency matrix of predicted graph. It allows us to evaluate whether the real rewards and triplets are captured with Fl scores. For a fair comparison, the games the FTWP dataset are divided into training (80%), validation (10%) and testing (10%) sets, based on which we first freeze the parameters in the dynamics models and then update the added P'g and p`, with data in training games.
Table 3 shows the testing performance. We study both 1) performance which measures the accu-
Table 3: Graph-generation (G-gen), graph updating (G-upt) and rewards prediction (7-pred) performance.
the graph-Generane racy of generating all the candidate triplets in graph and 2) the graph-Updlating performance which evaluates only the triplets that have been updated at the current step 4. We add a Real and a Random baseline that feed the ground- truth graph h and a randomly generated graph K into the graph encoder of transition models. They study the prob- ing performance when ground-truth graphs known and when the posterior collapses for the transition model (so pt. (zz,a)ar-1 12-1) p (zx,))) The results show that OS-OOTD achieves a better graph prediction performance by applying the abject-supervised objectives (OS-ELBo), but SS- OOTD has higher reward prediction accuracy, since SS-ELBo includes an end-to-end training to predict future rewards, which filters the reward-irrelevant object information. Another observation is that removing independent layers and directional attentions significantly degrade the graph prediction performance. demonstrates that they are important components of our OOTD model.
G-gen
G-upt
r-pred
Random Real
93.5
57.4
49.5
99.2
95.9
94.0
Ablated Dynamics modcls
SLa-Dyni
84.4
75.4
85.5
NAt-Dyna
90.9
75.2
88.7
OOTD models
OS-OOTD
99.0
83.8
92.5
SS-OOTD 77.2
65.3
94.2
5 RELATED WORK
DRL Agent for Text-Basee Based Games (TBGs). TBGs (e.g., Textworld (Côté et al. 2018) and inter- active fictional games (Hausknecht et al. 2020 interactive simulations with text observations and actions. To solve TBGs, previous DRL agents typically applied advanced neural models for
representing the state and action sentences, including recurrent models (Narasimhan et al. 2015 Dain
iet al. al. 2020 Ammanabrolu et al.) [2020] Urbanck et al.) 2201)) relevance networks (He et al. (016)). and convolutional neural networks Yii & May] 2019; Zahavy et al.) 2018 Yin & May] 2019)) Since states in the TBG environments are partially observable, many previous studies have formulated TBGs as POMDPs and modelled the history of observations with a Deep Recurrent Q-Network (DRON) (Yuan et al. 2018) or a knowledge graph (Hausknecht et al. 2019 Ammanabrolu & Riedl 2019 Ammanabrolu & Hausknecht] 2020; Adhikari et 20200 Ammanabrolu et al. (002). Previous works are commonly based on model-free Q learning while we explore model-based planning.
2020 |020:Zelinkka Zelinka|201 2011 8018 Madotto et al.) 2020; Adolphs & Hofmmann] (020). Transformers (Adhikari
Object Oriented Reinforcement Learning. Diuk al. (2008) proposed modelling the transition and reward models at the object level. Previous work (Mohan & Laird 201T Joshi et al. 2012
da Silva& & Costa 2018 Marom & Rosman 2018 da Silva et al.) 2019) explored training DRL agents with the object-level features. However, object features are not commonly available from the RL environments. Recent works have extracted object representations from states with optical flow (Goel let al. 2018), variational models (Burgess et al. 2019) and structured convolutional layers (Kipf et al. 2020 Finn et al.) 2016)) Zhu et al. [ 2020). Previous works commonly studied image-based Jully observable environments whereas we explore text-based partially observable environments.
CONCLUSION
We proposed an OOTD model that enables planning in text-based environments. The OOTD model learned a graph representation for capturing object dynamics and predicted the belief of object states with independent transition layers. Empirical results demonstrated that DOTD-based agents outperformed model-free baselines in terms of controlling performance and sample efficiency. We then provided ablation study to show the importance of OOTD components. A promising future direction will be expanding our OOTD model to parser-based games where the agent must generate the commands character by character without knowing candidate actions.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We acknowledge the funding from the Canada CIFAR AI Chairs program, and the support of the Natural Sciences and Engineering Research Council of Canada (NSERC). Resources used in this work were provided, in part, by the Province of Ontario, the Government Canada through CIFAR, and companies sponsoring the Vector Institute
/ https////eettr.sstttttt.eeeeee wttp::titiiteeeeettteteeeeee
ETHICS STATEMENT
In this work, we study model-based planning in simulated (TBG) environment. Although the task itself has limited consequences for society, take broader view of our work and discuss its potential influence on future research that might have considerable impact on our society. We believe developing autonomous RI agent for TBGs serves as the stepping stone toward real-worke applications where the machine cooperates or interacts with humans for some specific tasks. A concrete example of these applications will be the auto-dialogue systems, where the machine must respond to human concerns by conveying accurate information and solving the problem with humans. During this process, failing to communicate clearly, sensibly, or convincingly might also cause harm, for example, without sufficient calibration, the machine could generate improper words or phrases that might potentially cause a bias toward some minorities or underrepresented groups. How to eliminate such bias will be an important direction of future research.
REPRODUCIBILITY STATEMENT
This section introduces how we facilitate the reproducibility from the following perspectives:
Model Implementation. Section introduces our approach to dividing the training, validation, and testing datasets as well as the running settings based on the constructed datasets. We report the hyper-parameters for training OOTD model and the seeds for independent runs in Appendix A.I The computing resources and running time are discussed in Appendix A.7
Code and Dataset. Please find our implementation on Githut
REFERENCES
Ashutosh Adhikari, Xingdi Yuan, Alre-Alexandre Côté, Mikulás Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and William L.. Hamilton. Learning dynamic belief graphs to generalize on text-based games. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Leonard Adolphs and Thomas Hofmann. Ledeepchef deep reinforcement learning agent for families of text-based games. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7342-7349, AAAI Press, 2020.
Prithviraj Ammanabrolu and Matthew J. Hausknecht. Graph constrained reinforcement learning for natural language action spaces. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Prithviraj Ammanabrolu and Mark Riedl. Playing text-adventure games with graph-based deep reinforcement learning. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-ILLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 3557-3565. Association for Computational Linguistics, 2019.
Prithviraj Ammanabrolu, Ethan Tien. Matthew J. Hausknecht, and Mark O. Riedl. How to avoid being caten by grue: Structured exploration strategies for textual worlds. CoRR, abs/2006.07409, 2020.
10

---

Published as a conference paper at ICLR 2022
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACI. 2015, July 26-31, 2015, Beijing, China, Volume I: Long Papers, pp. 344 344-354. The Association for Computer Linguisties, 2015.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew Botvinick, and Alexander Lerehner. Monet: Unsupervised scene decomposition and representation. CoRR, abs/1901.11390, 2019.
Mare-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Workshop on Computer Games, pp. 41-75. Springer, 2018.
Adrien Couêtoux, Sean-Baptiste Hoock, Nataliya Sokolovska, Olivier Teytaud, and Nicolas Bonnard. Continuous upper confidence trees. In Carlos A. Coello Coello (ed.), Learning and Intelligent Optimization 5th International Conference, LION 5, Rome, Italy January 17-21, 2011. Selected Papers, volume 6683 of Lecture Notes in Computer Science, pp. 433-445. Springer, 2011.
Felipe Leno da Silva and Anna Helena Reali Costa. Object-oriented curriculum generation for reinforcement learning. In Elisabeth André, Sven Koenig, Mehdi Dastani, and Gita Sukthankar (eds.), Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, July 10-15, 2018, pp. 1026-10344 International Foundation for Autonomous Agents and Multiagent Systems Richland, SC, USA / ACM, 2018.
Felipe Leno da Silva, Ruben Glatt, and Anna Helena Reali Costa. MOO-MDP: an object-oriented representation for cooperative multiagent reinforcement learning. IEEE Trans. Cybern., 49(2): 567-579, 2019.
Carlos Diuk, Andre Cohen, and Michael L. Littman. An object-oriented representation for efficient reinforcement learning. In William W. Cohen, Andrew McCallum, and Sam T. Roweis (eds.), Ma- chine Learning. Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008, volume 307 of ACM International Conference Proceeding Series, pp. 240-247. ACM, 2008
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: a new approach for hard-expe explooralon problems. CoRR, abs/1901. 10995, 2019.
Chelsea Finn, Ian J. Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 64-72, 2016.
Ross B. Girshick. Fast R-CNN. In 2015 IEEE International Conference Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1440-1448. IEEE Computer Society, 2015.
Vikash Goel, Jameson Weng, and Pascal Poupart. Unsupervised video object segmentation for deep reinforcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchii and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 5688-5699, 2018
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. CoRR, Ttts/1999.10990 2019.
Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2555-2565. PMLR, 2019.

---

Published as a conference paper at ICLR 2022
Matthew J. Hausknecht, Ricky Loynd, Greg Yang, Adith Swaminathan, and Jason D. Williams. NAIL: A general interactive fiction agent. CoRR, abs/1902.002555 2019.
Matthew J. Hausknecht, Prithviraj Ammanabrolu, MlxcaAlexandre Côté, and Xingdi Yuan. Inter- active fiction games: A colossal adventure. In The Thirty-Fourhh AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7903-7910. AAAI Press, 2020.
He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng. and Mari Ostendorf. Deep reinforcement learning with natural language action space. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume I: Long Papers. The Association for Computer Linguistics, 2016.
Vishal Jain, William Fedus, Hugo Larochelle, Doina Precup, and Mare G. Bellemare. Algorithmic improvements for deep reinforcement learning applied interactive fiction. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 0928-4336. AAAI Press, 2020.
Mandar Joshi, Rakesh Khobragade, Saurabh Sarda, Umesh Deshpande, and Shiwali Mohan. Object- oriented representation and hierarchical reinforcement learning in infinite mario. In IEEE 24th International Conference on Tools with Artificial Intelligence, ICTAI 2012, Athens, Greece, Novem- ber 7-9, 2012, pp. 1076-1081. IEEE Computer Society, 2012.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Thomas N. Kipf, Elise van der Pol, and Max Welling, Contrastive learning of structured world models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou (eds.), Machine Learning: ECML 2006. 17th European Conference Machine Learning. Berlin, Germany, September 18-22, 2006, Proceedings, volume 4212 of Lecture Notes in Computer Science, pp. 282-293. Springer, 2006.
Leonid Kuvayev and Richard Sutton. Model-based reinforcement learning with an approximate, learned model. In Proceedings of the ninth Yale workshop on adaptive and learning systems, pp. 101-105, 1996.
Nathan O. Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. In Alexandre M. Bayen, Ali Jadbabaie, George J. Pappas, Pablo A. Parrilo, Benjamin Recht, Claire J. Tomlin, and Melanie N. Zeilinger (eds.), Proceedings of the 2nd Annual Conference on Learning for Dynamics and Control, L4DC 2020, Online Event, Berkeley, CA, USA, 11-12 June 2020, volume 120 of Proceedings of Machine Learning Research, pp. 761-770. PMLR, 2020.
Andrea Madotto, Mahdi Namazifar, Joost Huizinga, Piero Molino, Adrien Ecoffet, Huaixiu Zheng, Alexandros Papangelis, Dian Yu, Chandra Khatri, and Gökhan Tür. Exploration based language learning for text-based games. In Proceedings of the Twenty-Nind International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 1488-1494. ijcai.org, 2020.
Ofir Marom and Benjamin Rosman. Zero-shot transfer with deictic object-oriented representation in reinforcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 2297-2305, 2018
Shiwali Mohan and John E. Laird, An object-oriented approach to reinforcement learning in an action game. In Vadim Bulitko and Mark O. Riedl (eds.), Proceedings of the Seventh AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AlIDE 2011, October 10-14, 2011, Stanford, California, USA. The AAAI Press, 2011.
12

---

Published as a conference paper at ICLR 2022
Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. Language understanding for text- based games using deep reinforcement learning. In Lluís Märquez, Chris Callison-Buree Jian Su, Daniele Pighin, and Yuval Marton (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 1-11. The Association for Computational Linguistics, 2015.
Judea Pearl. Causality. Cambridge university press, 2009.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, 2015.
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. Aldo Gangemi, Roberto Navigli. Maria-Esther Vidal, Pascal Hitzler, Raphaël Troncy, Laura Hollink, Anna Tordai, and Mchwish Alam (eds.). The Semantic Web 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings, volume 10843 of Lecture Notes in Computer Science, pp. 593-607. Springer, 2018.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
David Silver. Aja Huang, Chris J. Maddison. Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Pannecrshelvam,, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587):484 489, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-1144, 2018.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence sequence learning with neural networks In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27: Annual Conference Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3104-3113 12, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop, ITW 2015, Jerusalem, Israel, April 26- May 1, 2015, pp. 1-5. IEEE, 2015.
Adam Trischler, Marc-Alexandre Côté, and Pedro Lima. First TextWorld Problems, the competition: Using text-based games to advance capabilities AI agents. 2019.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 2071-2080. JMLR.org, 2016.
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. Learning to speak and act in a fantasy text adventure game. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pp. 573-683. Association for Computational Linguistics, 2019.
13

---

Published as a conference paper at ICLR 2022
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017. December 4-9, 2017, Long Beach. CA, USA. pp.
2017.
Arthur Wandzel, Yoonseon Oh, Michael Fishman, Nishanth Kumar, Lawson L. S. Wong, and Stefanie Tellex. Multi-object search using object-oriented pomdps. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019, pp. 7194-7200. IEEE, 2019.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. CoRR, abs/ 00005700777 2019.
Xusen Yin and Jonathan May. Comprehensible context-driven text game playing. In IEEE Conference on Games, CoG 2019, London, United Kingdom, August 20-23, 2019, pp. 1-8. IEEE, 2019.
Xingdi Yuan, Mare-Alexandre Côté, Alessandro Sordoni, Romain Laroche, Remi Tachet des Combes, Matthew J. Hausknecht, and Adam Trischler. Counting to explore and generalize in text-based games. CoRR, abs/ tts//806.101 1525, 2018.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J. Mankowitz, and Shie Mannor. Learn what not to learn: Action elimination with deep reinforcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Desa-Biancchi, and Roman Garnett (eds.), Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 3566-3577, 2018.
Mikuláš Zelinka. Baselines for reinforcement learning in text games. Lefteri H. Tsoukalas, Éric Grégoire, and Miltiadis Alamaniotis (eds.), IEEE 30th International Conference Tools with Artificial Intelligence, ICTAI 2018, 5-7 November 2018, Volos, Greece, pp. 320-327. IEEE, 2018.
Guangxiang Zhu, Zhiao Huang, and Chongjie Zhang. Object-oriented dynamics predictor. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 9826-9837, 2018.
Guangxiang Zhu, Jianhao Wang, Zhizhou Ren, Zichuan Lin, and Chongjie Zhang. Object-oriented dynamics learning through multi-level abstraction. In The Chirty-Fourdh AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 6989-69999 AAAI Press, 2020.
14

---

Published as a conference paper at ICLR 2022
A MODEL IMPLEMENTATION DETAILS AND DATASETS
A.1 HYPER-PARAMETER
We report the key hyper-pane parameters that have been applied for model implementation. We set the number of candidate objects K to 99 and the number of candidate relations to 10 by following the settings in (Adhikari et al. 2020), which show that the agent can properly solve a game by modelling these objects and relations. The learning rate of policy training and dynamics model training is set to 0.0001. The discount factor ry is set to 0.9. The size of the hidden layers and the size of in our OOTD model are set to 32. The transition layers of different objects are parameterized independently, so the neural network predicts 32> x 99 = 3. 168 values at each step during dynamics modelling. The sizes of word embedding and node embedding are set 300 and 100 respectively. These numbers are determined experimentally, which could provide promising performance with a reasonable computational resource. The full list of typer-parameters is recorded in the yoml file in our submitted code. In our experiment, the random seeds we select are 123, 321, and 666.
A.2 RESAMPLING FOR COMPUTING THE GRAPH LIKELIHIOD
The graph likelihood Pn (2e] (1) computes the probability of generating h, with graph decoder. As it is discussed in Section 3.1 our graph decoder is implemented as a scoring function for entry-level prediction. However, h4 graph with significant sparsity, for example, the corresponding adjacency matrix assigns positive labels (i.e., 1) for an average 97.6 out of all 98,010 entries. As a result, directly optimizing toward h; creates large negative bias. In other words, the decoder achieve satisfying accuracy by labelling all the entries with Os, which does not capture any object information. To solve this problem, we implement a re-sampling technique that extracts all the positive samples (the entries labelled with one) from h and sample an equivalent number of negative samples from the rest of the entries. These positive and negative samples are used to estimate the graph likelihood.
A.3 GOAL EXTRACTOR
At a time step f, the goal extractor generates the goal Jt from the object states z; by optionally conditioning the previous goal dt-1 and observed text og. The goal can be represented by sequence of words and symbols that follows the triplet format. To generate this goal, we introduce two kinds of extractors that built for the OOTD learned by the OS-ELBo and the SS-ELBo objectives.
Goal Extractor with the OS-ELBo objective. Under this setting, the predicted memory graph hos captures the sparsity of object observations and forms direct prediction of the ground truth graph hg (check Figure 7). We can build a rule-based goal extractor that maps 2, his and checks whether h,s captures certain relations (e.g., 'need') between two objects. If there is such a relation, we directly output the sentence of corresponding triplets by following format "( object 1 $ relation $ object I". Such simple extractor can generalize well to training and testing games if i,s manages to capture correct relations.
Goal Extractor with the SS-ELBo objective. The memory graph i,s learned by the SS-ELBo objective captures latent representation of hz. This latent representation is not directly interpretable, and thus we build a word-by-word sequence generator whose output is the goal to be predicted (g:) and inputs are zt, 1t-1 and o,. To train this model, we manually build a goal learning dataset that collects pairs of model input/output from the training games. During testing, we predict the goal after interacting with the environment and update the current goal if valid goal sentence generated.
A.4 OBJЕCT EXTRACTOR
The object extractor fot, Gt- h4- predicts the memory graph hg by conditioning on current observations o, and action a, Figure showw shows the architecture of fe- It shares similar structure to our OOTD model, but the key differences are 1) fa is a deterministic model which only maximizes the likelihood of predicting he during training. 2) fa captures object information from o,, whereas
15

---

Published as a conference paper at ICLR 2022
Ot is not available for OOTD model, and it is the motivation why OOTD predicts the belief (distribution) states while fe directly generates the deterministic states.
O
a2-1
BIDAF
Attentions B B
MLP
Zi
RGCN Graph
Ck
ComplEx
-
MLP
Zy.
Graph Decoder
hr
Encoder
MLP
Zg.
Figure 5: The object extractor.
A.5 OBSERVATION ENCODER.
The observation encoder qe is defined by:
V 1 Σ, by 2/ (aja-1) where by softmar B
10,1,,
vs_ Σ Cx where b so voftmaz(max.. B Bae)) E [0, 1]
k = Σ, (a where b9 softmar (Bg
[0, 1]"
1 Σ ex where Docn softmar B (B)))) E [0, 1]
ek,611
(8)
de ( ( at-1, (k,0-1) = N o.)) where [4L o = (1V
V Vs cx,
d is transformer-bart text embedding function Vaswani al. 2017) for observations, and vil is implemented by a MLP. The transition layers vf..... Vi are parameterized independently.
Figure 6 shows the structure of an observation encoder of which shares a similar structure to transition (OOTD) model (Figure 3), but it includes an additional BIDAF network to learn an attention matrix 8 where bg, measures the similarity between the representation of the 4tth object and the 7th word in the current observation oz, It predicts the memory graph ht from 2f with ComplEE graph decoder (Section 3.1. predicts current observation ô, and reward rt with sequence decoder and the reward function (Section 3.2).
Observation BIDAF
O
Allentions n apee
Action BIDAF
ag-1
For the OS/SS ELBo
Graph/ sequence Decoder
2
Post MLP V.2
(1 og,)) =zf
0d/0,
7k.-1
Complex Decoder
h-
RGCN Encoder
Post MLP
Nih o9) ~ Rh
For the SS ELBo
Zg,111
Cj.c-1
Post MLP
N W ee off )eee
Mask Mean
Figure 6: The observation encoder.
A.6 FTWP DATASET
The FTWP dataset is public dataset that supports pre-training the dynamics model Trischler et al. 2019) provided the First TextWorld Problems (FTWP) dataset. The dataset consists of TextWorld
TThe dataset be downloaded https://t//aaaaaat httpss///aaaaaee/ffgaaaaaaaaaee
16

---

Published as a conference paper at ICLR 2022
games in cooking theme across wide range of difficulty levels. For each game, the expert trajectory and the memory graph h, at each time step are recorded. The original dataset has only 10 games for each difficulty level. This is insufficient for dynamics model pre-training, further expanded this dataset by adding more games into this dataset by utilizing the logic in the Textworld game engine. The specific generation rules described in (Adhikari al. 2020). After expansion, the dataset contains 4440 games for training, 222 games for validation, 514 games for lesting. To ensure fairness when using this dataset, we confirm that there no overlap between the FTWP and the games we used to train and evaluate our planning algorithms.
Adhikari et al. (2020)
A.7 COMPUTATIONAL RESOURCE AND RUNNING TIME
We run the experiment on cluster operated by the Slurm workload manager. The cluster has multiple kinds of GPUs, including Tesla T4 with 16 GB memory. Tesla P100 with 12 GB memory, and RTX 6000 with 24 GB memory. We used machines with 24 GB of memory for pre-training the object extractor and 64 GB for training the OOTD model. The number of running nodes is 1, and the number of CPUs requested per task is 32. Given the aforementioned resources, pre-training can be completed within 4 days (96 hours), and training the dynamics model takes 6 days (144 hours).
Table Game statistics at different difficulty levels.
Level
Recipe Size
#Locations
Max Scores
Need Cut
Need Cook
#Action Candidates
#Objects
0
1
3
X
10.5
15.4
1
1
4
V
X
11.5
17.1
11.8
17.5
3
1
9
3
X
X
7.2
34.1
4
3
6
11
V
28.4
33.4
5
Mixture of Levels| 1,2,3,4)
17

---

Published as a conference paper at ICLR 2022
B ALGORITHMS
Algorithm 1: Dyna-Q Training
Input
:Transition Model: (= [de; zz),
Object Encoder: qe (*t+1 01+1. ae. 21),
Reward Model: (r 1++1, ae. 9t),
Object Extraction function: fe (h,, Ce 9t+1),
Graph Decoder: (x.))
Training game ids Z.
Initialize Replay Buffers D;
while not converged do
Initialize belief state zo =random numbers, action do ="restart", goal go = "<pad>>" and memory graph ho = 00xXXX,
Randomly pick game id I, set = 0;
while not end do
x* Q learning with environment
0141,/t+1 +- env(a,, id);
Extract object information h(+1 flac: 2s, oc+1):
Update object belief z1+1 q8 (2(41 00-1: aes 21);
Extract goal: g++1 = EstactGoal 11 02+1- 9t):
Action selection: ut+1 EpsilonGreedy (2++11,121e))))) a++1, 1)+1);; Expand the replay buffer: {Z,, h2 a, Ir. Te+1s Ez41, h 11+1
1-1+1 and end Check End(env) end
for update step c=I....Cddo
/* Model fitting */
Sample transition: T= (zt, h4, Gt. 9t, f1,1, Zt+1, 11411,,411000 9c+1) from the dataset DD
Update the reward model to minimize the LISmooth loss (Equation 6 with T; Update the transition model to minimize the OS-ELBO (Equation objective or the SS-ELBO objective (Equation 7 with T;
Update the Q function to minimize the TD Loss (Equation) with T;
for explore step b= do
/ Explorat ion with the transition and reward models x/
Randomly select al, E A: 2,), ~ (p.((' 9r);
x++1
Update the transition T' = {Zt, 9t, a/,, T2+1, 1: 91+1
Update the Q function to minimize the TD Loss (Equation) with T:
end
end
end
18

---

Object Encoder: qe (#++1 0++1 ae: 2z), Published as a conference paper at ICLR 2022
Algorithm 2: MCTS Testing
Input
Transition Model: (z |ee zt),
Reward Model: pr D at; It), Action Value function: (2 Oe, 9t),
Testing game ids I
Initialize belief state 20 -random numbers, action on ="restart" and goal go ="<pad>"; Initialize reward buffer R;
for game id id E I do Initialize root No = InilTree((((z 04: 9x)), set / = 0 and R 0:
while not end do
01+1,"i+11 <- env(a, id) and R = + rz+1
Update object belief #t-11 ~ qE (Zr41 01-1: oz, 2:);
Extract goal: J++1 ExtactGoal #c+1. 0t+1, 9r);
Select action: 0c+1 = MCTS ( (Nr, 241 24. Q0);
Move the search node: N++1 ooveNN d(t1);
Expand ( = L + and end = Check End(eww))
end
Buffer total rewards R = UUU.
end
output: Reward buffer R
C COMPLEMENTARY RESULTS
C.I GRAPH VISUALIZATION
Graph Visualization. Figure illustrates examples of the ground-truth memory graph hū and the graphs learned by the Bo (A.S)) and SS-ELBo (h:53) objectives. Since the space of adjacency matrices is large (K x K x C') while the observed object information is limited, hद can be represent by a sparse matrix. Supervised by predicted graphs h4 OS-ELBo captures the sparsity, and h is direct approximation hç. However, graph information objective, so h. forms a dense latent representation of hü. Similar phenomenon can observed in GATA (Adhikari al., 2020). We study how well this latent representation and the corresponding object states can represent the object dynamics in Section 4.2
unavailable for the SS-ELBo
Ground Graph
80 leamed by 0S ELBo
leamed by ELBo
teamnd by GATA
05
30
B
02
30
50
00
30
80
000
50 50
Figure 7: Adjacency tensors corresponding the "is" relation in memory graphs hr From left to right: the ground-truth graph, and the graph learned by the OS-ELBo objective, 3) the SS-ELBo objective (Section 3.3). and 4) GATA (a graph-based dynamics model baseline) (Adhikari et al. 2020
C.2 PLANNING EFFICIENCY
We compare the efficiency of OOTD- based planning algorithms (including Dyna-Q, MCTS, and Dyna- Q+MCTS) by studying which method can gather more rewards with limited number of selected actions. Figure g shows results. Among the planning algorithms, Mena-Q+MCTS significantly outperforms others. It is because the Q network provides prior knowledge over the preference of
19

---

Published as a conference paper at ICLR 2022
actions, based which MCTS implements Monte-Carlo rollouts to generate and evaluate the belief states of objects. Together, they yield a more accurate look ahead future rewards.
Nommalinn Came Poetts
Uifically Lovel
Diffarly beed!
Lfficuly Level
Lifieckyy Leve
Uifeedyy Lovel
Difically terel
odve
D-yya
2.0
on 02
0.
24
B1--
23
101
10
Nommaloon Cere Poets
1.0
D-aya
06-
120
02
32
Figure 8: Planning Curve. The agent are based the dynamics model with the OS-ELBO objective (Upper) and the SS-ELBo objective (Lower). The plot shows mean normalized scores computed with three independent runs.
20


Reviews:
[{"review_id": "n5QxrLDFhQp", "reviewer": "Reviewer_cFdN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# strengths and weaknesses\n- strengths\n  - Tackles a difficult problem\n  - Presents a useful extension to OO POMDPs which can be used to model many text-based games\n  - Presents good evidence that model-based training using their learned dynamics model is an improvement over the baseline (model-free learning)\n  - Achievements in the domain of text-based games are important since they could lead to better conversational agents, which may be deployed in complex settings that require tracking conversation history, e.g., in IT, health\n- weaknesses\n  - Some technical details could use more explanation, which would improve reproducability and increase my confidence in the technical soundness of the work (see comments)\n  - Although there are comparisons to model-free methods, it seems that a comparison with a model-based method is missing (table 2)? Could the authors comment on how they expect their OOTD approach to compare to, e.g., a simple RNN that takes text observations as input and outputs a list of object states. Are there no model-based baselines, since this work is the first in the text-based games domain? \n\n# Questions for clarification\n- sec 2.2: Does $S$ contain all the information of $Z$? Is there a one-to-one correspondence between $S$ and $Z$? Is $Z$ completely recoverable from $S$?\n- sec 2.2: What exactly is contained in the object level states $Z$? You extend Diuk et al 2019, which defines a set of object classes and attributes for each class. Are classes and attributes represented explictly anywhere in your framework? Or are the object states $Z$ hidden state vectors that are not constrained to take any particular form?\n- sec 2.4: In equation 2, why isnt $\\zeta$ indexed by the goal $g$? Shouldn't the tree-search nodes record the current goal?\n- sec 3.1 (a): I think $Re(\\omega_c, z_{i, t-1}, z_{j, t-1})$ needs something in between the parentheses. Perhaps something like $Re(f_{score}(\\omega_c, z_{i,t-1}, z_{j,t-1}))$? Otherwise, do you mean that $Re()$ is being applied to the concatenation of three vectors?\n- sec 3.1 (c): How big is the space of actions? It sounds like each action is identified by a (possibly lengthy) sentence. Did you have any trouble with exploration?\n- sec 3.3 and A.3: What happens if the goal extractor finds multiple plausible goals in the memory graph? What if nothing valid is found?\n- sec 4 table 2: why aren't the GATA baselines in the \"model based planning category\"? Don't they also model system dynamics?\n- sec 3.3 equations 5 and 7: Can you explain how you arrive at your objective? I'm familiar with one way that evidence lower bound can appear: $\\log P(X) - D_{KL}(Q || P)$. Can you give some insight into why the first term in equations 5 and 7 involves an expectation?\n- sec 3.3 equations 5 and 7: Similarly, in the second term, can you explain the ordering of arguments in $D_{KL}(q_{\\epsilon} || p_{\\mathcal{T}})$ ? I would expect that since, $p_{\\mathcal{T}}$ is the function being learned, it should be the first argument in the KL divergence, as is typically the case when using an ELBo objective. \n\n# Typos\n- pg 2. typo? \"to enable stochasticity in the dynamics\" --> \"to enable stochacisity in the training\" \n- pg 6. typo: \"is implement by\" --> \"is implemented by\"\n- pg. 7: \"compassion\" --> \"comparison\"\n- pg. 7 bottom: sentence beginning with \"3)\" seems incomplete\n- pg. 8: \"object closed to each other\" --> \"close\"\n- pg. 8: figure 4 -- last two subfigures have same label", "summary_of_the_paper": "This paper tackles the difficult problem of learning to play text-based games. This domain is particularly challenging, since the text observations received at each time-step are variable length, and will only provide a partial description of the world, from which the full state must be reconstructed. Additionally, the player must track the history of past states in order to make the correct decision.\n\nThis paper introduces a new framework for describing the time-evolution of a game: Object-Oriented Partially Observable Markov Decision Processes (OO POMDP). This work's main contribution is a learned Object-Oriented Text Dynamics (OOTD) model that learns the transition and reward function of a OO POMDP game.\n\nBriefly, the presented method for predicting the transition function is: given a representation $z_t$ of the objects and $a_t$ of the action, create a graph, where the nodes are objects and the edges denote relationships. Perform message passing, and obtain new node representations $e_t$.\n\nThen, (1) given $e_t$, predict a new representation $v_{t}$ using dual stream attention with the action $a_t$. Then, (2) use $v_t$ to predict the updated state $z_{t+1}$ of each object, using an independent set of parameters per object.\n\nThe dynamics model is trained using a Evidence lower bound objective (ELBo) in two varieties: one that relies on a deterministic extraction of the graph and one that learns directly from rewards and observations.\n\nTo show its effectiveness, the learned dynamics model is used to model-based training (Dyna-Q, MCTS) is used to learn a planner. The learned planner is shown to out-perform the baselines (DQN, DRQN, GATA), both on the test cases, and in terms of sample efficiency. Ablations of (1) and (2) find qualitative advantages of the proposed method, since the learned object representation are both more separable and better semantically clustered. Additionally, the ablations are shown to reduce the accuracy of the graphs and state recreated from the object representations.", "summary_of_the_review": "Overall, I thought this paper was well written and presents a significant contribution in a novel area: modeling dynamics for text-based games. And the proposed extension to OO POMDPs will be generally useful for anyone working on text-based games. The given ablation studies and comparison to model-free baselines provides believable evidence for the effectiveness of their model-based learning approach. And while this work focused mainly on games, it's clear that their results could have useful applications in the field of conversational agents, which is growing in importance. I had some questions about the technical details of their method, especially the formulation of their objective. But these could probably be well answered in follow up discussion or in an appendix.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "d58ILDkrqO6", "reviewer": "Reviewer_enCT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I found that the paper is clearly written, with helpful illustrations. My specific questions follow.\n\n**Contributions**\n\n1. I am not sure if \"extension of OO-MDP to partial observability\" is the contribution of this paper. For example http://h2r1.cs.brown.edu/wp-content/uploads/wandzel19.pdf\n2. Abstract: dynamics models for image-based games are primarily on fully observable states -- I'm not sure if this is true. For example https://arxiv.org/pdf/1803.10122.pdf\n3. Intro Paragraph 1: generalize to all states and actions -- I don't see how this is exhaustively evaluated in the paper, nor did I see why this is important. Technically, we only need to make predictions about states that we are interested to achieve (and this is also how the authors collected the data).\n\n**Model**\n\nIt will be important to showcase or verify that the modeling of belief is important. I think the authors were slightly mixing two things together:\n- The partial observability of an environment (e.g., the position of an object is unknown).\n- The fact that at each step, the observation-only contains objects with state changes.\n\nAs the authors also stated, there is an implicit assumption about \"other object states do not change.\" This is a very strong assumption and can make model learning much easier, as in the STRIPS representation. See also https://en.wikipedia.org/wiki/Frame_problem \nActually, under this assumption, we don't have to treat it as a POMDP (instead, just an MDP). The authors should make the significance of their POMDP formulation more clear.\n\nI would also suggest authors try an ablation study with the belief prediction part removed (i.e., do not use sampling in the transition model).\n\nOn the writing of Section 2 and 3, I suggest that authors include ALL important modules in the paper (for example, the goal encoder). Some of the details are just echoing existing work, for example, OO-POMDP, MCTS, and can be potentially shortened. The writing can also be improved by incorporating a concrete running example showing the dynamics and the decision made.", "summary_of_the_paper": "This paper presents a novel approach for task-based games. The authors used an object-oriented POMDP formulation for the task. The key contribution is to learn the dynamics of the environment with a graph neural network (with either object supervision or self-supervision), and apply an online planning algorithm (e.g., MCTS) to solve the problem. The authors show strong improvements over other approaches in terms of sample efficiency and task scores.", "summary_of_the_review": "The paper is well-written and the results are strong. I have a few comments on the formulation and presentation of the paper. I also suggested an ablation study.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ERO-1L_-xqG", "reviewer": "Reviewer_W88S", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. This paper studies the TextGames, a very interesting and exploratory direction, and sets up a new SOTA performance through the model-based RL methods. The success enriches the application of dynamic model learning and planning algorithms. It may encourage follow-up works along the direction of planning under high-level semantic information and inspire fields across visual language navigation and robotic manipulation.\n2. Though there are existing works that train object-oriented dynamic models, applying them to text-domain requires non-trivial efforts. The paper demonstrates the importance of an object-oriented representation in text-model learning, which may help future research in learning better language models. If I am correct, the proposed method assembles various recent approaches, providing a very reasonable foundation for future researches.\n\nWeakness\n\n1. Though the paper does well in presenting the core idea of learning an object-oriented dynamic model, I met difficulties while parsing the architecture details. I believe that there is still a large room for improvement regarding paper writing. Here is my advice\n    1. The overall structure is a little bit confusing. The paper seems to mix the problem setup with the proposed method. For example, the paper introduces the OO-MDP, graph representation, and model-based planning in Sec 2, and introduces the detailed transition model, reward model, and training methods, and additional modules in Sec 3. This causes the structure fragile. \n    2. For example, the paper introduces the concept of learning dynamic model in the section defining OOMDP and mentions a learned graph updater in the section defining memory graph. In section 2.3, it uses Table 1, which contains a lot of undefined terms, before introducing the planning methods. As a result, after reading Sec 2, I do not know what the graph updater is and whether the latent z is the ground truth state or a learned latent state. In fact, until now I do not understand the meaning of the graph updater as I can not find any strict definition about it.\n    3. The training objectives section contains too much content. The authors put two different training setups, the way to construct the dataset, and several important network modules (observation encoder, decoder, and so on) in the last small section. This violates the logical flow and makes the section less coherent. The consequence is that I have to read back and forth to figure out what the networks actually learn and even go back to Sec 2 to get the meaning of variables. \n    4. While introducing the planning method and the transition model, the authors mention that \"Given/define the states z\". It is not proper, as z should be a learned latent variable and I think there is no ground truth information about z. In contrast, the authors should talk more about the observation, which is the real given input to the dynamic model. Currently, the observation encoder q_Epsilon is never formally defined and is missing in Figure 2. \n    5. In sec 2.2, the paper defines G_t, but it forgets to define h_t and oplus operator, making the graph updater unclear.\n    6. It forgets to mention the distribution that the model uses. Are they all Gaussians in formula (5) and (7)?\n    7. Section 3.1(c) lacks explanation. More importantly, it does not explain how it generates the two attention matrices Bs. Are they generated by the BIDAF networks? What is the input to that network?\n2. It seems that the model requires knowing the object set before training the model. Even in the self-supervised setting, the model needs to use a pre-determined object set. I worry if introducing too much prior will hurt the generalizability of the proposed method in new language tasks.\n\nMinor comments:\n\n1. I don't agree that extending OO-MDP with partial observability is a significant contribution.\n2. In Figure 4, there are two OS-OOTD in the title.\n3. Regarding the performance of the OS and SS setup, the paper mentions that OS's inferiors are due to the inaccurate reward prediction. I am not fully convinced as the adding extra supervision should not hurt the performance too much. Instead, I suspect that the reward module is too simple. Only a single MLP may not be enough. I suggest authors do more study about the reward module to figure out the reason behind the bad reward prediction.", "summary_of_the_paper": "This work proposes Object-Oriented Text Dynamics for model-based RL in the TextWorld environment. OOTD ensembles existing methods such as ComplEx, R-GCN to build a memory graph, which contains object-level information about each object's attribute and its relations to others, from history observations and predicts states at the next time step. Therefore, the authors can combine the learned model with Dyn-Q or MCTS methods to select actions. If I am correct, under an object-supervised setting, it assumes the existence of a dataset that contains a ground truth memory graph. In contrast, the network has to discover graphs from predicting observation and rewards in the self-supervised setting. Experiments show that OOTD achieves superior performance than various baselines, including model-free RL and previously learned graph models.", "summary_of_the_review": "I appreciate the author's effort in constructing such a complicated model to solve the challenging tasks, but the paper's writing has big flaws. Without adding enough details for a better evaluation, I do not recommend accepting this paper in its current form. I would be happy to increase my score if its writing can be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0oXDiEi4pZW", "reviewer": "Reviewer_DWsm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation of the paper makes sense. The author starts with challenges in text-based games:\n* text observation has variable length of sequence.\n* text only partially describes the current state\n* text could contain noisy information and it is challenging to identify salient terms.\n\nGeneral Significance: This is a difficult and interesting domain to study. The author proposes a reasonable extension based on OO POMDPs for text-based environment. The empirical results give strong evidence that this approach led to reasonable improvement in planning. \n\nNovelty: For my best knowledge, I think the method is a new contribution to the field. \n\nTechnical Quality and Clarity: The paper is well-structured, with clear relationship between each section. The general ideas are clearly conveyed. I especially like Figure 1 and Figure 4. Figure 1 is illustrative and shows the high-level idea clearly. Figure 4 visualizes the object state embedding, which clearly shows that the OS-OOTD method was able to embed semantic similar objects (ingredients) in a near position.  But if the author can use more illustrative examples on explaining the technical details, it would be more readable for broader audience.\n\nTypos:\n* Figure 4: the title for the right-most image should be \"SS-OOTD\".", "summary_of_the_paper": "This paper studies object-oriented text dynamics (OOTD) model and planning in text-based games. OOTD learns internal representation of object dynamics and uses transition layers to predict the belief of object states. Empirical results shows a performance boost compared to model-free baselines. Ablation studies are performed to explain the importance of OOTD components.", "summary_of_the_review": "marginally accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript makes an interesting observation: there is no reason why planning-based methods like MDPs must be limited to physical or grounded environments. One can plan about more abstract textual domains. It adapts the standard methods from planning to such text domains in a fairly straightforward way. The fact that concepts from MDPs map to these problems directly is an asset: ideas could flow between these domains in the long term. While the original submission was lacking clarity and significant technical details, the authors engaged with the reviewers and resolved lingering concerns. Reviewers are unanimous that this a strong contribution.
Decision: Accept (Poster)