Venue: NAACL 2021

# Reading and Acting while Blindfolded:

The Need for Semantics in Text Game Agents

Shunyu Yao\({}^{1*}\)  Karthik Narasimhan\({}^{\dagger}\)   Matthew Hausknecht\({}^{\ddagger}\)

\({}^{\dagger}\)Princeton University   \({}^{\ddagger}\)Microsoft Research

{shunyuyu, karthika}@princeton.edu

matthew.hausknecht@microsoft.com

Work partly done during internship at Microsoft Research. Project page: [https://blindfolded.cs](https://blindfolded.cs). princeton.edu.

###### Abstract

Text-based games simulate worlds and interact with players using natural language. Recent work has used them as a testbed for autonomous language-understanding agents, with the motivation being that understanding the meanings of words or _semantics_ is a key component of how humans understand, reason, and act in these worlds. However, it remains unclear to what extent artificial agents utilize semantic understanding of the text. To this end, we perform experiments to systematically reduce the amount of semantic information available to a learning agent. Surprisingly, we find that an agent is capable of achieving high scores even in the complete absence of language semantics, indicating that the currently popular experimental setup and models may be poorly designed to understand and leverage game texts. To remedy this deficiency, we propose an inverse dynamics decoder to regularize the representation space and encourage exploration, which shows improved performance on several games including Zork I. We discuss the implications of our findings for designing future agents with stronger semantic understanding.

## 1 Introduction

Text adventure games such as Zork I (Figure 1 (a)) have been a testbed for developing autonomous agents that operate using natural language. Since interactions in these games (input observations, action commands) are through text, the ability to understand and use language is deemed necessary and critical to progress through such games. Previous work has deployed a spectrum of methods for language processing in this domain, including word vectors Fulda et al. (2017), recurrent neural networks Narasimhan et al. (2015); Hausknecht et al. (2020), pre-trained language models Yao et al. (2020), open-domain question answering systems Ammanabrolu et al. (2020), knowledge graphs Ammanabrolu and Hausknecht (2020); Ammanabrolu et al. (2020); Adhikari et al. (2020), and reading comprehension systems Guo et al. (2020).

Meanwhile, most of these models operate under the reinforcement learning (RL) framework, where the agent explores the same environment in repeated episodes, learning a value function or policy to maximize game score. From this perspective, text games are just special instances of a partially observable Markov decision process (POMDP) \((S,T,A,O,R,\gamma)\), where players issue text actions \(a\in A\), receive text observations \(o\in O\) and scalar rewards \(r=R(s,a)\), and the underlying game state \(s\in S\) is updated by transition \(s^{\prime}=T(s,a)\).

However, what distinguishes these games from other POMDPs is the fact that the actions and observations are in language space \(L\). Therefore, a certain level of deciperable _semantics_ is attached to text observations \(o\in O\subset L\) and actions \(a\in A\subset L\). Ideally, these texts not only serve as observation or action _identifiers_, but also provide clues about the latent transition function \(T\) and reward function \(R\). For example, issuing an action "jump" based on an observation "on the cliff" would likely yield a subsequent observation such as "you are killed" along with a negative reward. Human players often rely on their understanding of language semantics to inform their choices, even on games they have never played before, while replacing texts with non-semantic identifiers such as their corresponding hashes (Figure 1 (c)) would likely render games unplayable for people. However, would this type of transformation affect current RL agents for such games? In this paper, we ask the following question: _To what extent do current reinforcement learning agents leverage semantics in text-based games?_

To shed light on this question, we investi

gate the Deep Reinforcement Relevance Network (DRRN) He et al. (2016), a top-performing RL model that uses gated recurrent units (GRU) Cho et al. (2014) to encode texts. We conduct three experiments on a set of interactive fiction games from the Jericho benchmark Hausknecht et al. (2020) to probe the effect of different semantic representations on the functioning of DRRN. These include (1) using just a location phrase as the input observation (Figure 1 (b)), (2) hashing text observations and actions (Figure 1 (c)), and (3) regularizing vector representations using an auxiliary inverse dynamics loss. While reducing observations to location phrases leads to decreased scores and enforcing inverse dynamics decoding leads to increased scores on some games, hashing texts to break semantics surprisingly matches or even outperforms the baseline DRRN on almost all games considered. This implies current RL agents for text-based games might not be sufficiently leveraging the semantic structure of game texts to learn good policies, and points to the need for developing better experiment setups and agents that have a finer grasp of natural language.

## 2 Models

DRRNBaselineOur baseline RL agent DRRN He et al. (2016) learns a Q-network \(Q_{\phi}(o,a)\) parametrized by \(\phi\). The model encodes the observation \(o\) and each action candidate \(a\) using two separate GRU encoders \(f_{o}\) and \(f_{a}\), and then aggregates the representations to derive the Q-value through a MLP decoder \(g\):

\[Q_{\phi}(o,a)=g(\mathrm{concat}(f_{o}(o),f_{a}(a))) \tag{1}\]

For learning \(\phi\), tuples \((o,a,r,o^{\prime})\) of observation, action, reward and the next observation are sampled from an experience replay buffer and the following temporal difference (TD) loss is minimized:

\[\mathcal{L}_{\text{TD}}(\phi)=(r+\gamma\max_{a^{\prime}\in A}Q_{\phi}(o^{ \prime},a^{\prime})-Q_{\phi}(o,a))^{2} \tag{2}\]

During gameplay, a softmax exploration policy is used to sample an action:

\[\pi_{\phi}(a|o)=\frac{\exp(Q_{\phi}(o,a))}{\sum_{a^{\prime}\in A}\exp(Q_{\phi }(o,a^{\prime}))} \tag{3}\]

Note that when the action space \(A\) is large, (2) and (3) become intractable. A valid action handicap Hausknecht et al. (2020) or a language model Yao et al. (2020) can be used to generate a reduced action space for efficient exploration. For all the modifications below, we use the DRRN with the valid action handicap as our base model.

Reducing Semantics via Minimizing Observation (min-ob)Unlike other RL domains such as video games or robotics control, at each step of text games the (valid) action space is constantly changing, and it reveals useful information about the current state. For example, knowing "unlock box" is valid leaks the existence of a locked box. Also, sometimes action semantics indicate its value even unconditional on the state, e.g. "pick gold" usually seems good. Given these, we minimize the observation to only a location phrase \(o\mapsto\mathrm{loc}(o)\) (Figure 1 (b)) to isolate the action semantics: \(Q_{\phi}^{\mathrm{loc}}(o,a)=g(f_{o}(\mathrm{loc}(o)),f_{a}(a)))\).

Breaking Semantics via Hashing (hash)GRU encoders \(f_{o}\) and \(f_{a}\) in the Q-network (1) generally ensure that similar texts (e.g. a single word change) are given similar representations, and therefore similar values. To study whether such a semantics continuity is useful, we break it by hashing observation and action texts. Specifically,

Figure 1: (a): Sample original gameplay from Zork I. (b) (c): Our proposed semantic ablations. (b) min-ob reduces observations to only the current location name, and (c) hash replaces observation and action texts by their string hash values.

 

given a hash function from strings to integers \(h:L\rightarrow\mathbb{Z}\), and a pseudo-random generator \(G:\mathbb{Z}\rightarrow\mathbb{R}^{d}\) that turns an integer seed to a random Gaussian vector, a hashing encoder \(\hat{f}=G\circ h:L\rightarrow\mathbb{R}^{d}\) can be composed. While \(f_{o}\) and \(f_{a}\) in (1) are trainable, \(\hat{f}\) is fixed throughout RL, and ensures two texts that only differ by a word would have completely different representations. In this sense, hashing breaks semantics and only serves to identify different observations and actions in an abstract MDP problem (Figure 1 (c)): \(Q_{\phi}^{\mathrm{hash}}(o,a)=g(\hat{f}(o),\hat{f}(a))\).

Regularizing Semantics via Inverse Dynamics Decoding (inv-dy)The GRU representations in DRRN \(f_{o}(o),f_{a}(a)\) are only optimized for the TD loss (2). As a result, text semantics can degenerate during encoding, and the text representation might arbitrarily overfit to the Q-values. To regularize and encourage more game-related semantics to be encoded, we take inspiration from Pathak et al. (2017) and propose an inverse dynamics auxiliary task during RL. Given representations of current and next observations \(f_{o}(o),f_{o}(o^{\prime})\), we use a MLP \(g_{inv}\) to predict the action representation, and a GRU decoder \(d\) to decode the action back to text*. The inverse dynamics loss is defined as

Footnote *: Directly defining an L1/L2 loss between \(f_{a}(a)\) and \(g_{inv}(\operatorname{concat}(f_{o}(o),f_{o}(o^{\prime})))\) in the representation space will collapse text representations together.

\[\mathcal{L}_{\text{inv}}(\phi,\theta)=-\log p_{d}(a|g_{inv}( \operatorname{concat}(f_{o}(o),f_{o}(o^{\prime}))))\]

where \(\theta\) denote weights of \(g_{inv}\) and \(d\), and \(p_{d}(a|x)\) is the probability of decoding token sequence \(a\) using GRU decoder \(d\) with initial hidden state as \(x\). To also regularize the action encoding, action reconstruction from \(f_{a}\) is also used as a loss term:

\[\mathcal{L}_{\text{dec}}(\phi,\theta)=-\log p_{d}(a|f_{a}(a))\]

And during experience replay, these two losses are optimized along with the TD loss:

\[\mathcal{L}(\phi,\theta)=\mathcal{L}_{\text{TD}}(\phi)+\lambda_{1}\mathcal{L }_{\text{inv}}(\phi,\theta)+\lambda_{2}\mathcal{L}_{\text{dec}}(\phi,\theta)\]

An intrinsic reward \(r^{+}=\mathcal{L}_{\text{inv}}(\phi,\theta)\) is also used to explore toward where the inverse dynamics is not learned well yet. All in all, the aim of inv-dy is threefold: (1) regularize both action and observation representations to avoid degeneration by decoding back to the textual domain, (2) encourage \(f_{o}\) to encode action-relevant parts of observations, and (3) provide intrinsic motivation for exploration.

## 3 Results

SetupWe train on 12 games++ from the Jericho benchmark Hausknecht et al. (2020). These human-written interactive fictions are rich, complex, and diverse in semantics.++ For each game, we train DRRN asynchronously on 8 parallel instances of the game environment for \(10^{5}\) steps, using a prioritized replay buffer. Following prior practice Hausknecht et al. (2020), we augment observations with location and inventory descriptions by issuing the 'look' and 'inventory' commands. We train three independent runs for each game and report their average score. For Hash, we use the Python built-in hash function to process text as a tuple of token IDs, and implement the random vector generator \(G\) by seeding PyTorch with the hash value. For inv-dy, we use \(\lambda_{1}=\lambda_{2}=1\).

ScoresTable 1 reports the final score (the average score of the final 100 episodes during training), and the maximum score seen in each game for different models. Average normalized score (raw score divided by game total score) over all games is also reported. Compared to the base DRRN, min-ob turns out to explore similar maximum scores on

\begin{table}
\begin{tabular}{r|l|l l l l|l} \hline \multicolumn{1}{c|}{**Game**} & \multicolumn{1}{c|}{**DRN**} & \multicolumn{1}{c}{**MIN-ob**} & \multicolumn{1}{c}{**Hash**} & \multicolumn{1}{c|}{**INV-by**} & \multicolumn{1}{c}{**Max**} \\ \hline \hline balances & 10 / 10 & 10 / 10 & 10 / 10 & 10 / 10 & 51 \\ deephone & 57 / 66 & 8.5 / 27 & 58 / 67 & 57.6 / 67 & 300 \\ detective & 290 / 337 & 86.3 / 350 & 290 / 317 & 290 / 323 & 360 \\ dragon & 5.0 / 6 & - 5.4 / 3 & - 5.0 / 7 & **-2.7** / 8 & **2** \\ enchanter & 20 / 20 & 20 / 40 & 20 / 30 & 20 / 30 & 40 \\ inhumane & 21.1 / 45 & 12.4 / 40 & 21.9 / 45 & 19.6 / 45 & 90 \\ library & 15.7 / 12 & 12.8 / 21 & **17** & 16.2 / 21 & 30 \\ ladder & 12.7 / 23 & 11.6 / 21 & **14.8**/ 23 & 13.5 / 23 & 150 \\ omnipast & 4.9 / 5 & 4.9 / 5 & 4.9 / 5 & 5.3 / 10 & 50 \\ pentari & 26.5 / 45 & 21.7 / 45 & **51.9**/ **60** & 37.2 / 50 & 70 \\ zork1 & 39.4 / 53 & 29.7 / 46 & 35.5 / 50 & **43.1**/ **87** & 350 \\ zork3 & 0.4 / 4.5 & 0.0 / 4 & 0.4 / 4 & 0.4 / 4 & 7 \\ \hline Avg. Norm & 21 / 38 & 12 / 35 & **.25** / 39 & 23 / **.40** & \\ \hline \end{tabular}
\end{table}
Table 1: Final/maximum score of different models.

Figure 2: Transfer results from Zork I.

 

most games (except Deephome and Dragon), but fails to memorize the good experience and reach high episodic scores, which suggests the importance of identifying different observations using language details. Most surprisingly, hash has a lower final score than DRRN on only one game (Zork I), while on Pentari it almost doubles the DRRN final score. It is also the model with the best average normalized final score across games, which indicates that the DRRN model can perform as well without leveraging any language semantics, but instead simply by identifying different observations and actions with random vectors and memorizing the Q-values. Lastly, we observe on some games (Dragon, Omniquest, Zork I) inv-dy can explore high scores that other models cannot. Most notably, on Zork I the maximum score seen is 87 (average of \(54,94,113\)), while any run of other models does not explore a score more than 55. This might indicate potential benefit of developing RL agents with more semantic representations.

TransferWe also investigate if representations of different models can transfer to a new language environment, which is a potential benefit of learning natural language semantics. So we consider the two most similar games in Jericho, Zork I and Zork III, fix the language encoders of different Zork I models, and re-train the Q-network on Zork III for 10,000 steps. As shown in Figure 2, inv-dy representations can achieve a score around 1, which surpasses the best result of models trained from scratch on Zork III for 100,000 steps (around 0.4), showing great promise in better gameplay by leveraging language understanding from other games. Hash transfer is equivalent to training from scratch as the representations are not learnt, and a score around 0.3 is achieved. Finally, DRRN representations transfer worse than hash, possibly due to overfitting to the TD loss [2].

VisualizationsFinally, we use t-SNE Maaten and Hinton (2008) to visualize representations of some zork I walkthrough states in Figure 3. The first 30 walkthrough states (red, score 0-45) are well experienced by the models during exploration, whereas the last 170 states (blue, score 157-350) are unseen6. We also encircle the subset of states at location 'living room' for their shared semantics.

Footnote 6: The rest 150 states in the middle (score 45-157) are omitted as they might be seen by some model but not others.

First, we note that the hash representations for living room states are scattered randomly, unlike the other two models with GRU language encoders. Further, the base DRRN overfits to the TD loss [2], representing unseen states (blue) in a different subspace to seen states (red) without regarding their semantic similarity. ind-dy is able to extrapolate to unseen states and represent them similarly to seen states for their shared semantics, which may explain its better performance on this game.

Game stochasticityAll the above experiments were performed using a fixed game random seed for each game, following prior work Hausknecht et al. (2020). To investigate if randomness in games affects our conclusions, we run one trial of each game with episode-varying random seeds7. We find the average normalized score for the base DRRN, hash, inv-dy to be all around 17%, with performance drop mainly on three stochastic games Dragan, Zork I, Zork III). Notably, the core finding that the base DRRN and hash perform similarly still holds. Intuitively, even though the Q-values would be lower overall with unexpected transitions, RL would still memorize observations and actions that lead to high Q-values.

Figure 3: t-SNE visualization of seen and unseen state observations of Zork I. DRRN (base) represents unseen states separated from seen states while inv-dy mixes them by semantic similarity.

 



## 4 Discussion

At a high level, RL agents for text-based games succeed by (1) exploring trajectories that lead to high scores, and (2) learning representations to stably reach high scores. Our experiments show that a semantics-regularized **inv-dy** model manages to explore higher scores on some games Dragon, Omniquest, Zork I), while the **hash** model manages to memorize scores better on other games Library, Ludicorp, Pentari) using just a fixed, random, non-semantic representation. This leads us to hypothesize two things. First, fixed, stable representations might make Q-learning easier. Second, it might be desirable to represent similar texts very differently for better gameplay, e.g. the Q-value can be much higher when a key object is mentioned, even if it only adds a few words to a long observation text. This motivates future thought into the structural vs. functional use of language semantics in these games.

Our findings also urge a re-thinking of the popular 'RL + valid action handicap' setup for these games. On one hand, RL sets training and evaluation in the same environment, with limited text corpora, and sparse, mostly deterministic rewards as the only optimization objective. Such a combination easily results in overfitting to the reward system of a specific game (Figure 2), or even just a specific stage of the game (Figure 3). On the other hand, the valid action handicap reduces the action set to a small size tractable for memorization, and reduces the language understanding challenge for the RL agent. Thus for future research on text-based games, we advocate for more attention towards alternative setups without RL or handicaps Hausknecht et al. (2019); Yao et al. (2020); Yu et al. (2020). Particularly, in a 'RL + no valid action handicap' setting, _generating_ action candidates rather than simply _choosing_ from a set entails more opportunities and challenges with respect to learning grounded language semantics Yao et al. (2020). Additionally, training agents on a distribution of games and evaluating them on a separate set of unseen games would require more general semantic understanding. Semantic evaluation of these proposed paradigms is outside the scope of this paper, but we hope it will spark a productive discussion on the next steps toward building agents with stronger semantic understanding.

### Ethical Considerations

Autonomous decision-making agents are potentially impactful in our society, and it is of great ethical consideration to make sure their understanding of the world and their objectives align with humans. Humans use natural language to convey and understand concepts as well as inform decisions, and in this work we investigate whether autonomous agents leverage language semantics similarly to humans in the environment of text-based games. Our findings suggest that the current generation of agents optimized for reinforcement learning objectives might not exhibit human-like language understanding, a phenomenon we should pay attention to and further study.

### Acknowledgements

We appreciate helpful suggestions from anonymous reviewers as well as members of the Princeton NLP Group and MSR RL Group.

## References

* Adhikari et al. [2020] Ashutosh Adhikari, Xingdi (Eric) Yuan, Marc-Alexandre Cote, Mikulas Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and William L. Hamilton . 2020. Learning dynamic knowledge graphs to generalize on text-based games. In NeurIPS 2020.
* Ammanabrolu and Hausknecht [2020] Prithviraj Ammanabrolu and Matthew J. Hausknecht. 2020. Graph constrained reinforcement learning for natural language action spaces. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
* Ammanabrolu et al. [2020] Prithviraj Ammanabrolu, Ethan Tien, Zhaochen Luo, and Mark O Riedl. 2020. How to avoid being eaten by a grue: Exploration strategies for text-adventure agents. _arXiv preprint arXiv:2002.08795_.
* Cho et al. [2014] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_, pages 103-111, Doha, Qatar. Association for Computational Linguistics.
* Fulda et al. [2017] Nancy Fulda, Daniel Ricks, Ben Murdoch, and David Wingate. 2017. What can you do with a rock? affordance extraction via word embeddings. In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017_, pages 1039-1045. ijcai.org

Xiaoxiao Guo, Mo Yu, Yupeng Gao, Chuang Gan, Murray Campbell, and Shiyu Chang. 2020. Interactive fiction game playing as multi-paragraph reading comprehension with reinforcement learning. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7755-7765, Online. Association for Computational Linguistics.
* Hausknecht et al. (2019) Matthew Hausknecht, Ricky Loyden, Greg Yang, Adith Swaminathan, and Jason D Williams. 2019. Nail: A general interactive fiction agent. _arXiv preprint arXiv:1902.04259_.
* Hausknecht et al. (2020) Matthew J. Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Cote, and Xingdi Yuan. 2020. Interactive fiction games: A colossal adventure. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_, pages 7903-7910. AAAI Press.
* He et al. (2016) Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. 2016. Deep reinforcement learning with a natural language action space. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1621-1630, Berlin, Germany. Association for Computational Linguistics.
* Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of machine learning research_, 9(Nov):2579-2605.
* Narasimhan et al. (2015) Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. 2015. Language understanding for text-based games using deep reinforcement learning. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1-11, Lisbon, Portugal. Association for Computational Linguistics.
* Pathak et al. (2017) Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. 2017. Curiosity-driven exploration by self-supervised prediction. In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_, volume 70 of _Proceedings of Machine Learning Research_, pages 2778-2787. PMLR.
* Yao et al. (2020) Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. 2020. Keep CALM and explore: Language models for action generation in text-based games. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8736-8754, Online. Association for Computational Linguistics.
* Yu et al. (2020) Mo Yu, Xiaoxiao Guo, Yufei Feng, Xiaodan Zhu, Michael Greenspan, and Murray Campbell. 2020. Deriving commonsense inference tasks from interactive fictions. _arXiv preprint arXiv:2010.09788_