Venue: NAACL 2021

# How to Motivate Your Dragon:

Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds

 Prithviraj Ammanabrolu\({}^{\dagger}\) Jack Urbanek\({}^{\ddagger}\) Margaret Li\({}^{\ddagger}\)

Arthur Szlam\({}^{\ddagger}\) Tim Rocktaschel\({}^{\dagger}\) Jason Weston\({}^{\ddagger}\)

\({}^{\dagger}\)Georgia Institute of Technology Facebook AI Research

raj.ammanabrolu@gatech.edu

###### Abstract

We seek to create agents that both act and communicate with other agents in pursuit of a goal. Towards this end, we extend LIGHT [10]--a large-scale crowd-sourced fantasy text-game--with a dataset of "quests".1. These contain natural language motivations paired with in-game goals and human demonstrations; completing a quest might require dialogue or actions (or both). We introduce a reinforcement learning system that (1) incorporates large-scale language modeling-based and commonsense reasoning-based pre-training to impute the agent with relevant priors; and (2) leverages a factorized action space of action commands and dialogue, balancing between the two. We conduct zero-shot evaluations using held-out human expert demonstrations, showing that our agents are able to act consistently and talk naturally with respect to their motivations.

Footnote 1: Data can be found here [https://parl.ai/projects/light/](https://parl.ai/projects/light/)

_Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 807–833

June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

There has been a recent improvement in the quality of natural language processing (NLP) and generation (NLG) by machine learning (ML) [45, 9]; and in parallel, improvement to goal-oriented ML driven agents in the context of games [46, 47]. However, agents that can communicate with humans (and other agents) through natural language in pursuit of their goals are still primitive. One possible reason for this is that many datasets and tasks used for NLP are static, not supporting interaction and language grounding [1, 13, 14, 15]. However, agents that can communicate with humans (and other agents) through natural language in pursuit of their goals are still primitive. One possible reason for this is that many datasets and tasks used for NLP are static, not supporting interaction and language grounding [1, 13, 15, 16, 17]. Text-based games--where players see, act upon, and communicate within a dynamic world using natural language--provide a platform on which to develop such goal-driven agents.

LIGHT [10], a large-scale crowdsourced fantasy text-adventure game, consisting of a set of locations, characters, and objectsas possesses rich textual worlds, but without any notion of goals to train goal-driven agents. We present a dataset of quests for LIGHT and demonstrations of humans playing these quests (as seen in Figures 2 and 3), providing natural language descriptions in varying levels of abstraction of motivations for a given character in a particular setting.

To complete these quests, an agent must reason about potential actions and utterances based on incomplete descriptions of the locations, objects, and other characters. When a human is placed in a fantasy setting such as LIGHT, they already know that things are royalty and must be treated respectively, swords are weapons, etc.--commonsense knowledge that a learning agent must acquire to ensure successful interactions. To equip agents with relevant priors in such worlds, we domain-adapt the large-scale commonsense knowledge graph ATOMIC [10] to the LIGHT fantasy world--to build ATOMIC-LIGHT.

We then introduce a reinforcement learning (RL) system that incorporates large-scale language modeling and the above commonsense-based pre-training. We show that RL is superior to behavior cloning or other supervised training on our data; and that carefully combining pre-training with RL is superior to either.

However, we find that although pre-training can be an effective tool in this setting, it requires more finesse than in the standard supervised setting. In particular, we find that simply pre-training a model on a large "generic" corpus [10, 11] of commonsense/language data or pre-training on the domain specific LIGHT corpus, and then fine-tuning via RL is _less_ effective than training RL from scratch. Furthermore, by

carefully combining general and domain-specific pre-training, we observe large improvements over RL from scratch.

In short, the contributions of this paper are threefold: (1) A dataset of quests, LIGHT-Quests, and a companion fantasy themed commonsense knowledge graph ATOMIC-LIGHT; (2) a reinforcement learning architecture and training methodology that use these datasets to create goal-driven agents that act and speak in the LIGHT environment; and (3) Empirical zero-shot evaluations based on human quest demonstrations and an analysis of large-scale transformer-based pre-training trends in static vs. interactive settings, showing that we have trained agents that act consistently and speak naturally with respect to their motivations.

## 2 Related Work

We focus on four major areas of related work: text-based game-playing, goal-oriented dialogue, commonsense reasoning in language, and general language-informed RL.

**Text-based game-playing.**[45] introduce TextWorld, a framework for procedurally generating text-based games via grammars, and [68, 79, 10] build agents that operate in this environment--focusing on aspects such as efficient exploration and zero-shot generalization to new, procedurally generated environments. Similarly, [11] introduce Jericho, a framework and series of baseline agents for interacting with human-made textgames such as _Zork_[1]. This resulted in agents developed by works such as [12, 13], aiming to learn to execute contextually relevant actions. Other works such as [11, 14] explore how to best factorize such text-game action spaces. None of these works consider agents with motivations and personas nor require any dialogue.

**Goal-oriented dialogue.** This form of dialogue has traditionally been closely related to specific tasks useful in the context of personal assistants with dialogue interfaces [15, 16, 17]. In fact, the dialogue is not always a problem but it does not know whether or not the dialogue itself is a problem. For example, the dialogue can be viewed as a "language", which is a 

El Asri et al., 2017). RL has been studied for such tasks, usually to improve dialogue state management Singh et al. (2000); Pietquin et al. (2011); Fatemi et al. (2016) and to improve response quality Li et al. (2016). In particular, the negotiation tasks of Yarats and Lewis (2017); Lewis et al. (2017), where two agents are trying to convince each other to perform certain actions, are related to the tasks in LIGHT-Quests. These works all lack environment grounding and the notion of diverse agent motivations.

**Commonsense reasoning in language.** Works such as Bosseltu et al. (2019); Guan et al. (2020) focus on pre-training transformer-based language learning systems with large-scale commonsense knowledge graphs such as ATOMIC Sap et al. (2019) and ConceptNet Speer and Havasi (2012) for use in knowledge graph completion and story ending generation respectively. Fulda et al. (2017); Ammanabrolu and Riedl (2019); Ammanabrolu et al. (2020); Murugesan et al. (2020) look at commonsense reasoning in interactive environments, with the former focusing on affordance extraction using word embeddings and the latter three on transferring text-game playing skills via pre-training using question-answering and large-scale knowledge graphs.

**Language-informed reinforcement learning.**Luketina et al. (2019) provide an overview of RL informed by natural language. Of these works, the ones most related to ours are those falling into the category of instruction following--where an agent's tasks are defined by high level instructions describing desired policies and goals MacMahon et al. (2006); Kollar et al. (2010). Visual and embodied agents using natural language instructions Bisk et al. (2016); Kolve et al. (2017); Anderson et al. (2018) or in language-based action spaces Das et al. (2017) utilize interactivity and environment grounding but have no notion of agent motivations, nor make any attempt to explicitly model commonsense reasoning. Perhaps closest in spirit to this work is Prabhumoye et al. (2020), where they use artificially selected goals in LIGHT and train RL agents to achieve them. Similarly to the others, this work does not contain the motivations provided by LIGHT-Quests nor any modeling of commonsense reasoning. Further, they limit their RL problem to 1 and 3-step trajectories that only involve speech, and no actions--compared to the human demonstrations in LIGHT-Quests which contain both actions and speech sequences of average length \(12.92\).

## 3 LIGHT-Quests and ATOMIC-LIGHT

This section first provides a brief overview of the LIGHT game environment, followed by descriptions of the LIGHT-Quests and ATOMIC-LIGHT datasets used in this paper.

**Background.** The LIGHT game environment is a multi-user fantasy text-adventure game consisting of a rich, diverse set of characters, locations, and objects (1775 characters, 663 locations, and 3462 objects). Characters are able to perform templated actions to interact with both objects and characters, and can speak to other characters through free form text. Actions in text games generally consist of verb phrases (VP) followed optionally by prepositional phrases (VP PP). For example, _get OBJ, put OBJ, give OBJ to CHAR_, etc.. There are \(13\) types of allowed verbs in LIGHT. These actions change the state of the world which is expressed to the player in the form of text descriptions.

### LIGHT-Quests

Figures 1, 2, and 3 summarize the data that we collected for LIGHT-Quests. Data is collected via crowdsourcing in two phases, first the quests then demonstration of humans playing them. During the first phase, crowdworkers were given a setting, i.e. situated in a world, in addition to a character and its corresponding persona and asked to describe in free form text what potential motivations or goals could be for that character in the given world. The kind of information given to the crowdworkers is seen in Figure 1. Simultaneously, they were also asked to provide a sequence of seven timeline actions--one action that needs to be completed _now_ and three before and after at various user-defined intervals--for how the character might go about achieving these motivations.

Given the information in Figure 1, the crowdworkers completed the above outlined tasks and produce data as seen in Figure 2. Motivations come in three levels of abstraction--short, mid, and long--corresponding to differing amounts of the timeline. For example, the short motivation is always guaranteed to correspond most closely to the _now_ position on the timeline. Action annotation is pre-constrained based on the classes of verbs available within LIGHT. The rest of the action is completed as free form text as it may contain novel entities introduced in the motivations. There are 

\(5982\) training, \(756\) validation, and \(748\) test quests. Further details regarding the exact data collection process and details of LIGHT-Quests are found in Appendix A.1.1.

After collecting motivation and timelines for the quests, we deployed a two-player version of the LIGHT game, letting players attempt the quests for themselves in order to collect human demonstrations. Figure 3 shows an example human expert demonstration of a quest. Players were given a character, setting, motivation, and a partner agent and left to freely act in the world and talk to the partner in pursuit of their motivations. The partner agent is a fixed poly-encoder transformer model Humeau et al. (2020) trained on the original LIGHT data as well as other human interactions derived via the deployed game--using 111k utterances in total. Players first receive a role-playing score on a scale of 1-5 through a Dungeon Master (DM), a learned model that ranks how likely their utterances are given the current context. Once they have accumulated a score reaching a certain threshold, they are allowed to perform actions. We employ this gamification mechanism to encourage players to role-play their character persona and its motivations, leading to improved user experience and data quality Horsfall and Oikonomou (2011). They are then given further reward if the actions they perform sequentially match those on the timeline for the given quest. The game ends after a maximum of six turns of dialogue per agent, i.e. twelve in total. The average sequence of a human demonstration is \(12.92\), with an average action sequence length of \(2.18\) and dialogue of \(10.74\). There are \(1800\) training, \(100\) validation, and \(211\) test human expert demonstrations after the data was filtered. Additional details and examples are found in Appendix A.2.

### Atomic-Light

Commonsense reasoning is a critical cornerstone when building learning agents that navigate spaces such as LIGHT-Quests. To this end, we domain-adapt the large-scale commonsense knowledge base ATOMIC Sap et al. (2019) to LIGHT. ATOMIC contains information relevant for everyday commonsense reasoning in the form of typed if-then relations with variables. ATOMIC is organized into a set of events, e.g. "X puts X's trust in Y" and annotated relation types such as "needs", "wants", "attributes", and "effects" that label the effects. It is designed to be a general atlas of commonsense data and so is neither dependent on a specific environment or a character's persona and motivations.

To construct ATOMIC-LIGHT, we specifically use the relations for "intents", "efects", "wants" and "needs" and expand the \(\langle subject,relation,object\rangle\) triples found in the graph into templated natural language sentences. These sentences are then rewritten to better reflect the fantasy LIGHT domain. Named entities and other noun phrases in ATOMIC are masked out and filled in using BERT Devlin et al. (2018) fine-tuned using a masked language model loss on the entire LIGHT and LIGHT-Quests data. We investigate the benefits of such domain adaptation on downstream tasks in Section 4.3. An example of a clause using the _wants_ relation in ATOMIC is as follows, "_PersonX puts PersonX trust in PersonY, wants, rely on PersonY:_" In ATOMIC-LIGHT, this is rewritten to: "The merchant puts the merchant's trust in the guard, as a result the merchant _wants_ to rely on the guard." Similarly, an example of an effect using the _needs_ relation is, "Before, the merchant puts the merchant's trust in the guard, the merchant _needs_ to be friends with the guard." ATOMIC-LIGHT contains \(216686\) training, \(35340\) validation, and \(38565\) test samples. Further details of the construction of this dataset are found in Appendix A.4.

## 4 Agents that Act and Speak

This section describes the creation of the agents that learn to act and speak conditioned on their motivations in the LIGHT environment. The overall architecture and training are first outlined, followed by a detailed discussion on types of encoder pretraining.

### LIGHT RL Environment

The environment as seen in Figure 4 consists of three components. The first is a partner agent, which is a model trained to play other agents in the game, as in Prabhumoye et al. (2020). Next is the game engine, which determines the effects of actions on the underlying game graph Urbanek et al. (2019). Finally, there is the Dungeon Master (DM), which is trained to score the naturalness of dialogue.

**Partner Agent.** The partner agent is a polyelectrolyte transformer model Humneau et al. (2020) 

that is pre-trained on the Reddit dialogue corpus, then on LIGHT and the human demonstrations of LIGHT-Quests. Following the format seen in Figure 3, the partner agent does not have a motivation itself but is trained to react to agents with motivations. Following Prabhumoye et al. (2020), we keep the partner model fixed during the episodes where the LIGHT agent trains to ensure that it retains natural English semantics--avoiding the problem of language drift by learning an emergent language with that must agree with the partner's usage Lee et al. (2019).

**Action Rewards via the Game Engine.** All actions, either those of the agent-in-training or the partner agent, are processed by the engine, checking for goal state completion--hence known as _act goals_. For example, if the LIGHT agent had the motivation to acquire a sword, the goal could be completed via a:

1. **self act completion**: where the agent acquires a sword itself by picking it up, stealing it, convincing the partner to drop theirs so you can pick it up, etc.
2. **partner act completion**: where the agent uses speech to convince their partner to achieve the goal for them (e.g., by persuading the partner to give them the sword).

Reaching an _act goal_ provides reward \(r_{a}\) of \(1\) and \(0\) otherwise. At each step, the engine also provides us with the set of valid actions. These are the subset of the action space \(A\) which are guaranteed to be a valid change to the world from the current state \(s_{t}\), i.e. an action to give your partner a sword cannot be valid unless you possess the sword.

**Speech Rewards via the Dungeon Master.** Following prior works on using transformers for automatic evaluation of natural language generation Sellam et al. (2020), we utilize a learned model-the Dungeon Master (DM)--to score the agent's ability to speak. The DM used here is a poly-encoder model trained on collected human quest demonstrations as well as the original conversations in LIGHT. It is conditioned on quests and motivations and thus able to provide a (noisy) indication of how natural the agent's dialogue utterances are given its immediate context, similarly to the function of the DM during the data collection process. Given the dialogue portion of a human quest demonstration of length \(n\), the DM returns a reward \(r_{u}\) of \(\frac{1}{2n}\) if an utterance was in the demonstration (for a maximum of one time per episode for each utterance from the demonstration). A further \(\frac{1}{2n}\) is given each time the utterance is scored as being within the top-\(k\) most likely utterances by the DM. This naturalness objective will be hence referred to as a _speech goal_. These rewards thus also denser than _act goals_, helping the agent learn overall. Further, similarly to the game engine, the DM also provides a set of \(M\) valid utterances which are the \(M\) most likely dialogue candidates from the candidate set for the current context.

### Training a LIGHT agent with Switch Reinforcement Learning

The overall architecture of our agent is shown in Figure 4. It consists of an encoder, a switch, an action network, and a dialogue network. First, we construct the action spaces--factorized into actions and utterances. The possible actions are the set of all actions taken in the demonstrations (4710 total) and the possible utterances are all utterances from the demonstrations (22672 total). The encoder network processes the setting, persona, motivation, as

Figure 4: Overall RL Switch architecture and process. Blue shaded components can be pre-trained and Red shaded components are trained with RL. Solid lines indicate gradient flow.

 

well as the full history of actions and dialogues performed by the agent and the partner, input as a text sequence. The features from the encoder, which here are the hidden states at the final layer of a transformer, are used as input by all following components of the agent. In Section 5 we show how different encoder training data affects the model.

Next, a switch module makes the decision regarding whether the agent should act or talk in the current context and activates the corresponding policy network. In this work, the switch is simple: it outputs an action every \(k\) dialogue utterances; where during training \(k\) is chosen to match the ratio of utterances to actions on that particular quest from the human demonstrations, and during testing, \(k\) is chosen to match the average action to utterance ratio. Both the action and dialogue policies consist of a a single GRU layer followed by an \(n\)-layer feed-forward network given input features from the encoder. Once the LIGHT agent has output an utterance or action, it is processed by the environment--the partner agent, the game engine and the DM.

We use A2C Mnih et al. (2016) to train the LIGHT agent, treating the two policy networks as two separate actors with a shared critic. The shared critic is motivated by the concepts of _self act completion_ and _partner act completion_ seen in Section 4.1 where the LIGHT agent can speak to convince the partner to achieve an _act goal_. Each agent in a batch is initialized via priority sampling Graves et al. (2017) with a different quest, i.e. quests that the agent has historically successfully completed less often are given a greater weight when sampling from the pool of all possible training quests. In addition to a normal entropy regularization term, we also add a regularization term that encourages the models to produce "valid" outputs as judged by the game engine and the DM for actions and utterances respectively. Additional training details are found in Appendix B.2.

### Encoder Pre-training Tasks

Prior work on commonsense reasoning in supervised natural language learning Bosselut et al. (2019) suggests that the encoder is key to overcoming the challenges posed by the LIGHT-Quests dataset even in an RL setting. We describe a series of encoder pre-training tasks, designed to help the LIGHT agent either act more consistently or speak more naturally.

**ATOMIC-LIGHT** As seen in Section 3, ATOMIC-LIGHT is a (domain-adapted) fantasy commonsense knowledge graph, and as such provides priors for an agent on how to act consistently in the world. For example, given a clause such as "The knight wishes to slug the dragon, as a result the knight _needs_ to acquire a sword," the task would be to predict the underlined text--a form of knowledge graph completion Wang et al. (2017).

**Reddit** We use a previously existing Reddit dataset extracted and obtained by a third party and made available on pushshift.io Baumgartner et al. (2020) seen in Roller et al. (2020). This dataset has been used in several existing dialogue-based studies and has been shown to result in more natural conversations Yang et al. (2018); Mazare et al. (2018).

**LIGHT-Original** The original LIGHT dataset Urbanek et al. (2019) is organized similarly to the human demonstrations found in LIGHT-Quests, i.e. an interspersed sequence of dialogue and actions collected from humans role-playing a character. The task itself is to predict the next action or utterance given the prior dialogue history as well as the current setting and persona for a character. They are collected in a chit-chat fashion, with no notion of objectives, and so provide priors on how to generally act consistently and speak in a fantasy world, but not directly how to complete quests.

**LIGHT-Quests** Pre-training with this newly introduced dataset consists of three tasks. (1) _Bag-of-action timeline prediction_ in which, given a quest consisting of setting, persona, and motivations, any one of the actions in the timeline must be predicted. (2) _Sequential timeline prediction_ in which, given a quest consisting of setting, persona, motivations, and the first \(n\) actions in the timeline, the \(n+1^{th}\) action must be predicted. (3) Predict the next dialogue utterance given a human demonstration in a manner similar to the LIGHT-original tasks. The first two tasks are designed to help the agent act consistently and the third to help it speak naturally with respect to its motivations.

## 5 Evaluation

We conduct two ablation studies, (1) to compare the effects of the encoder pre-training tasks in RL settings vs. supervised behavior cloning, and (2) to analyze the interplay between actions and dialogue for _self_ and _partner act completions_ 



### Encoder Pre-training Type Ablation Study

Pre-training is done on the tasks described in Section 4.3 by training a 12 layer transformer with \(256\) million parameters using a cross-entropy loss as seen in [8]. These weights are then transferred to the Blue shaded portion of the encoder as seen in Figure 4 and frozen. A further three randomly initialized-layers are appended on to the end, indicated by the Red portions, into which gradients flow. This is done as optimizing all the parameters of such a model via RL over a long horizon is both data inefficient and computationally infeasible. Additional hyperparameter details are found in Appendix B.1. We investigate the following five different pre-training models to see how they compare on _act_ and _speech goal completions_ when trained with RL and in a supervised manner with behavior cloning:

**Scratch** No pre-training is done, the encoder is a 3-layer randomly initialized transformer and trained along with the policy networks.

**General Multi-task trained using both pushshift.io Reddit and the commonsense dataset ATOMIC-LIGHT, giving the agent general priors on how to act and speak.

**Light Multi-task trained on all tasks in LIGHT-original and LIGHT-Quests, giving the agent priors on how to act and speak with motivations in the LIGHT fantasy domain.**

**General+Light Multi-task trained on all tasks used in the General and Light models.**

**Adaptive Here we adaptively train a General+Light model that is first initialized itself from a General model, providing additional regularization to help balance between Light and General tasks.**

Table 1 describes the results for this ablation. Models were each zero-shot evaluated on \(211\) human demonstrations from the LIGHT-Quests test set for a single episode per quest across three independent runs. Figure 5 shows learning curves during training for each encoder type. We first see that performance when trained with RL, i.e. with interactivity and environment grounding during training, results in higher performance than behavioral cloning for all the models. In both RL and behavior cloning settings the Adaptive model outperforms all others in all the metrics. When trained supervised (behavioral cloning), we see trends mirroring standard pre-training in static text corpora. Transfer is easy and the Scratch model performs significantly worse than all others; and each new task added improves the agent's ability to speak and act. In particular, we see that Light outperforms General, showing that the more similar the pre-training tasks are to the downstream tasks, the better the supervised performance. However, these trends do not hold in the RL setting. The Scratch model outperforms everything except the Adaptive model and General outperforms Light. In part, this may be due to specification gaming [9]; however Adaptive does strongly outperform Scratch in goals with dialogue. This suggests that transfer (and fine-tuning) is not as simple in the RL setting as in the supervised setting, but still can be useful if carefully done. We note that domain adapative pre-training (intermediate task transfer) has previously been shown to give modest gains in supervised learning [7, 10], but not with the large effects seen here for RL. Figure 5 further shows that with the right combination of tasks, not only is the generalization performance better, but training itself is more sample efficient--requiring fewer steps before reaching asymptotic performance.

### Ability Type Ablation Study

To better understand the interplay between acts and speech resulting in _self_ and _partner act goal com

\begin{table}
\begin{tabular}{|l|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|} \hline \multicolumn{1}{|c|}{**Model**} & \multicolumn{3}{c|}{**Reinforcement Learning**} & \multicolumn{1}{p{42.7pt}|}{**Behavioral Cloning**} \\  & **Act Goals** & **Speech Goals** & **Act \& Speech Goals** & **Act \& Speech Goals** \\ \hline
**Seratch** & 0.418 & 0.118 & 0.103 & 0.0003 \\
**General** & 0.146 & 0.040 & 0.028 & 0.00226 \\
**Light** & 0.115 & 0.028 & 0.022 & 0.0934 \\
**General+Light** & 0.251 & 0.094 & 0.081 & 0.115 \\
**Adaptive** & **0.420** & **0.330** & **0.303** & **0.147** \\ \hline \end{tabular}
\end{table}
Table 1: Encoder Type RL Zero-Shot Evaluations averaged over 3 independent runs. Act goals and speech goals are as described in Section 4.1. Standard deviations for all experiments are less than 0.01. The “Act & Speech Goals” column refers to quests where the agent has simultaneously achieved both types of goals within the episode. Human act goal completion \(=0.6\) as measured during the second phase of the LIGHT-Quests data collection.



pletions_, we perform an ablation study selectively dropping either the agent's ability to talk or act. We train the agent to either only act, only speak, only speak with only action rewards. In the scenarios when the agent can only speak, the agent has to convince the partner to help achieve the agent's goal.

The results are outlined in Table 2. Unsurprisingly, when trained to only act, the act goal completion rate increases over when it can both act and speak. Similarly, when trained to only speak the speech goal completion rates also increase. We can draw two conclusions from these results: (1) It is much easier to do an action yourself than to convince the partner to do it (2) Removing speech goals increases the act goal completion rates corresponding to higher partner act completions. Thus, the sequences of dialogue utterances required to convince the partner to achieve the agent's goal are likely often at odds with those sequences required to maximize speech goals.

## 6 Conclusion

Operating on the hypothesis that interactivity is key to language learning, we introduce two datasets--a set of quests based on character motivations in fantasy worlds, LIGHT-Quests, and a large-scale commonsense knowledge graph, ATOMIC-LIGHT--and a reinforcement learning system that leverages transformer-based pre-training to facilitate development of goal-driven agents that can act and speak in situated environments. Zero-shot evaluations on a set of novel human demonstration show that we have trained agents that act consistently and speak naturally with respect to their motivations. A key insight from our ablation study testing for zero-shot generalization on novel quests is that large-scale pre-training in interactive settings require careful selection of pre-training tasks--balancing between giving the agent "general" open domain priors and those more "specific" to the downstream task--whereas static methodologies require only domain specific pre-training for effective transfer but are ultimately less effective than interactive methods.

## 7 Broader Impacts

The ability to speak and act in these textual fantasy worlds has implications for domains beyond text-games. We view text-games as an platform on which to teach agents how to communicate effectively using natural language, to plan via sequential decision making in situations that may not be anticipated. Given that our methods rely on deep- and-reinforcement learning techniques operating on language, they are prone to the same pitfalls as other contemporary dialogue and text-game systems. We mitigate, though do not entirely eliminate, the two main pitfalls that our particular system is prone to: (1) non-normative language usage--describing situations that fictional characters may engage in inappropriate for the real world--by restricting our system to a retrieval rather than a generative sys

\begin{table}
\begin{tabular}{|l|l|l|l|l|l|l|} \hline  & \multicolumn{3}{c|}{**Scratch**} & \multicolumn{4}{c|}{**Adaptive**} \\

- & **0.469** &
- \\ Speech Only & 0.036 & **0.165** & 0.028 & 0.0398 & **0.341** & 0.030 \\ -No Speech Goals & 0.0526 & 0.0521 & 0.0331 & 0.0673 & 0.0947 & 0.041 \\ \hline \end{tabular}
\end{table}
Table 2: Ability type ablations averaged across 3 runs with standard deviations less than 0.01.

Figure 5: Encoder types RL reward curves averaged over 3 independent runs.

 

tem, enabling us to filter the possible outputs of the agent; and (2) dataset bias via curation through controlled crowdsourcing in the case of LIGHT-Quests--the methods to debias the original LIGHT dataset can be found in Dinan et al. (2020) and crowdsourcing methods for the original ATOMIC work can be found in Sap et al. (2019). Further details regarding crowdsourcing data collection methodology for LIGHT-Quests can be found in Appendix A.1.1.

## References

* Adhikari et al. [2020] Ashutosh Adhikari, Xingdi Yuan, Marc-Alexandre Cote, Mikulas Zelinka, Marc-Antoine Rondeau, Roomain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and William L. Hamilton. 2020. Learning dynamic knowledge graphs to generalize on text-based games. _arXiv preprint arXiv:2002.09127_.
* Adolphs and Hofmann [2019] Leonard Adolphs and Thomas Hofmann. 2019. Ledeepchef: Deep reinforcement learning agent for families of text-based games. _arXiv preprint arXiv:1909.01646_.
* Ammanabrolu and Hausknecht [2020] Prithviraj Ammanabrolu and Matthew Hausknecht. 2020. Graph constrained reinforcement learning for natural language action spaces. In _International Conference on Learning Representations_.
* Ammanabrolu and Riedl [2019] Prithviraj Ammanabrolu and Mark Riedl. 2019. Transfer in deep reinforcement learning using knowledge graphs. In _Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13) at EMNLP_.
* Ammanabrolu et al. [2020] Prithviraj Ammanabrolu, Ethan Tien, Matthew Hausknecht, and Mark O Riedl. 2020. How to avoid being eaten by a grue: Structured exploration strategies for textual worlds. _arXiv preprint arXiv:2006.07409_.
* Anderson et al. [2018] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 3674-3683.
* Anderson et al. [1979] Tim Anderson, Marc Blank, Bruce Daniels, and Dave Lebling. 1979. Zork. [http://ifdb.tads.org/viewgame?id=4gxk83ja4twckm6j](http://ifdb.tads.org/viewgame?id=4gxk83ja4twckm6j).
* Barsalou [2008] Lawrence W. Barsalou. 2008. Grounded cognition. _Annual Review of Psychology_, 59(1):617-645. PMID: 17705682.
* Baumgartner et al. [2020] Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. In _Proceedings of the International AAAI Conference on Web and Social Media_, volume 14, pages 830-839.
* Bisk et al. [2016] Yonatan Bisk, Deniz Yuret, and Daniel Marcu. 2016. Natural language communication with robots. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 751-761.
* Bosselt et al. [2019] Antoine Bosselt, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for automatic knowledge graph construction. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Brooks [1991] Rodney A Brooks. 1991. Intelligence without representation. _Artificial intelligence_, 47(1-3):139-159.
* Cote et al. [2018] Marc-Alexandre Cote, Akos Kadar, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018. Textworld: A learning environment for text-based games. _CoRR_, abs/1806.11532.
* Das et al. [2017] Abhishek Das, Satwik Kottur, Jose MF Moura, Stefan Lee, and Dhruv Batra. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. In _Proceedings of the IEEE international conference on computer vision_, pages 2951-2960.
* Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. _CoRR_, abs/1810.04805.
* Dinan et al. [2020] Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020. Queens are powerful too: Mitigating gender bias in dialogue generation. In _Proceedings of Empirical Methods in Natural Language Processing (EMNLP-20)_.
* Asri et al. [2017] Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: a corpus for adding memory to goal-oriented dialogue systems. In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_, pages 207-219, Saarbrucken, Germany. Association for Computational Linguistics.
* Fatemi et al. [2016] Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, and Kaheer Suleman. 2016. Policy networks with two-stage training for dialogue systems. _arXiv preprint arXiv:1606.03152_.
* Feldman and Narayanan [2004] Jerome Feldman and Srinivas Narayanan. 2004. Embodied meaning in a neural theory of language. _Brain and language_, 89:385-92 

Nancy Fulda, Daniel Ricks, Ben Murdoch, and David Wingate. 2017. What can you do with a rock? affordance extraction via word embeddings. In _IJCAI_, pages 1039-1045.
* Gauthier and Mordatch (2016) Jon Gauthier and Igor Mordatch. 2016. A paradigm for situated and goal-driven language learning. _arXiv preprint arXiv:1610.03585_.
* Graves et al. (2017) Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. 2017. Automated curriculum learning for neural networks. In _International Conference on Machine Learning_, pages 1311-1320.
* Guan et al. (2020) Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020. A knowledge-enhanced pretraining model for commonsense story generation. _Transactions of the Association for Computational Linguistics_.
* Gururangan et al. (2020) Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop retraining: Adapt language models to domains and tasks. _arXiv preprint arXiv:2004.10964_.
* Hausknecht et al. (2020) Matthew Hausknecht, Prithviraj Ammanabrolu, Marco-Alexandre Cote, and Xingdi Yuan. 2020. Interactive fiction games: A colossal adventure. In _ThirtyFourth AAAI Conference on Artificial Intelligence (AAAI)_.
* He et al. (2016) Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. 2016. Deep reinforcement learning with a natural language action space. In _ACL_.
* Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_, pages 263-272.
* Horsfall and Oikonomou (2011) Matthew Horsfall and Andreas Oikonomou. 2011. A study of how different game play aspects can affect the popularity of role-playing video games. In _2011 16th International Conference on Computer Games (CGAMES)_, pages 63-69. IEEE.
* Humeau et al. (2020) Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In _International Conference on Learning Representations_.
* Kollar et al. (2010) Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. 2010. Toward understanding natural language directions. In _2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_, pages 259-266. IEEE.
* Kolve et al. (2017) Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farnadi. 2017. AI2-THOR: An Interactive 3D Environment for Visual AI. _arXiv_.
* Krakovna et al. (2017) Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. specification gaming: the flip side of ai ingenuity.
* Lake et al. (2017) Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. _Behavioral and brain sciences_, 40.
* Lawrence et al. (2019) Carolin Lawrence, Bhushan Kotnis, and Mathias Niepert. 2019. Attending to future tokens for bidirectional sequence generation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1-10, Hong Kong, China. Association for Computational Linguistics.
* Lee et al. (2019) Jason Lee, Kyunghyun Cho, and Douwe Kiela. 2019. Countering language drift via visual grounding. _arXiv preprint arXiv:1909.04499_.
* Lewis et al. (2017) Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning for negotiation dialogues. _arXiv preprint arXiv:1706.05125_.
* Li et al. (2016) Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. Deep reinforcement learning for dialogue generation. _CoRR_, abs/1606.01541.
* Luketina et al. (2019)Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktaschel. 2019. A survey of reinforcement learning informed by natural language. _arXiv preprint arXiv:1906.03926_.
* MacMahon et al. (2006) Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. 2006. Walk the talk: Connecting language, knowledge, and action in route instructions. In _AAAI_.
* Mazare et al. (2018) Pierre-Emmanuel Mazare, Samuel Humneau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2775-2779, Brussels, Belgium. Association for Computational Linguistics.
* Mikolov et al. (2016) Tomas Mikolov, Armand Joulin, and Marco Baroni. 2016. A roadmap towards machine intelligence. In _International Conference on Intelligent Text Processing and Computational Linguistics_, pages 29-61. Springer.
* Mnih et al. (2016) Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In _International conference on machine learning_, pages 1928-1937 

* Murugesan et al. (2020) Keerthiram Murugesan, Mattia Atzeni, Pushkar Shukla, Mrinmaya Sachan, Pavan Kapanipathi, and Kartik Talamadupula. 2020. Enhancing text-based reinforcement learning agents with commonsense knowledge. _arXiv preprint arXiv:2005.00811_.
* Narasimhan et al. (2015) Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. 2015. Language understanding for text-based games using deep reinforcement learning. In _EMNLP_, pages 1-11.
* Phang et al. (2018) Jason Phang, Thibault Fevry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. _arXiv preprint arXiv:1811.01088_.
* Pietquin et al. (2011) Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Herve Freza-Buet. 2011. Sample-efficient batch reinforcement learning for dialogue management optimization. _ACM Transactions on Speech and Language Processing (TSLP)_, 7(3):7.
* Prabhumoye et al. (2020) Shrimai Prabhumoye, Margaret Li, Jack Urbanek, Emily Dinan, Douwe Kiela, Jason Weston, and Arthur Szlam. 2020. I love your chain mail! masking knights smile in a fantasy game world: Open-domain goal-orientated dialogue agents. _arXiv preprint arXiv:2002.02878_.
* Roller et al. (2020) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. 2020. Recipes for building an open-domain chatbot. _arXiv preprint arXiv:2004.13637_.
* Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 3027-3035.
* Schrittwieser et al. (2019) Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. 2019. Mastering atari, go, chess and shogi by planning with a learned model. _arXiv preprint arXiv:1911.08265_.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Singh et al. (2000) Satinder P Singh, Michael J Kearns, Diane J Litman, and Marilyn A Walker. 2000. Reinforcement learning for spoken dialogue systems. In _Advances in Neural Information Processing Systems_, pages 956-962.
* Speer and Havasi (2012) Robyn Speer and Catherine Havasi. 2012. Representing general relational knowledge in conceptnet 5. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC)_.
* Sutton et al. (1998) Richard S Sutton, Andrew G Barto, et al. 1998. _Introduction to reinforcement learning_, volume 135. MIT press Cambridge.
* Urbanek et al. (2019) Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktaschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019. Learning to speak and act in a fantasy text adventure game. _CoRR_, abs/1903.03094.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Vinyals et al. (2019) Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michael Mathieu, Andrew Dudzhik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019. Grandmaster level in starcraft ii using multi-agent reinforcement learning. _Nature_, 575(7782):350-354.
* Wang et al. (2017) Q. Wang, Z. Mao, B. Wang, and L. Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. _IEEE Transactions on Knowledge and Data Engineering_, 29(12):2724-2743.
* Yang et al. (2018) Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-Yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Learning semantic textual similarity from conversations. _arXiv preprint arXiv:1804.07754_.
* Yarats and Lewis (2017) Denis Yarats and Mike Lewis. 2017. Hierarchical text generation and planning for strategic dialogue. _arXiv preprint arXiv:1712.05846_.
* Yin and May (2019) Xusen Yin and Jonathan May. 2019. Comprehensible context-driven text game playing. _CoRR_, abs/1905.02265.
* Yuan et al. (2018) Xingdi Yuan, Marc-Alexandre Cote, Alessandro Sordoni, Romain Laroche, Remi Tachet des Combes, Matthew J. Hausknecht, and Adam Trischler. 2018. Counting to explore and generalize in text-based games. _CoRR_, abs/1806.11525.
* Zahavy et al. (2018) Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. 2018. Learn what not to learn: Action elimination with deep reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, _Advances in Neural Information Processing Systems 31_, pages 3562-3573. Curran Associates, Inc.

 

Appendix - Datasets

### LIGHT-Quests

#### a.1.1 Mechanical Turk Data Collection

Crowdworkers are required to first pass an on-boarding test before they are allowed to perform the actual task. Figures 6, 7, 8, 9, and 10 describe first the instructions given to the crowdworkers and then 4 phases of the on-boarding test. We paid workers $2.75 per task. This amount was determined by first running the task ourselves to estimate a completion time of 10-12 minutes per task, and then running pilot tasks that confirmed the average task duration for workers was close to 10 minutes.

Figure 11 shows the example of the actual task given to the crowdworkers and Figure 12 shows the user interface for the first phase of the LIGHT-Quests data collection task described in Section 3.1.

 



#### Page (14) - Attempts (07)

The first three questions you will be asked about your task will relate to coming up with a motivation for why your character is taking that action. All of these motivations should be written in **first person** . The first should be something that can be resolved within a few minutes, perhaps by the specific action. A few good examples:

* **get apple:** Fm getting pretty hungry and would like a snack.
* **go outside:** fd like to get a breath of fresh air.
* **wield sword:** I must prepare for an impending battle.

The reasoning that you come up with should be in-line with your given character, so a _pricess_ likely wouldn't provide _Tm hoping to sell some candy at the store as a reason to _get chocolate_ .

### Imagine you're answering for the following character:

#### Character Name: Farmer

**Character Description:** I am a very hard worker. I am up with the sun 7 days a week. I learned all I know from my father. I will leave my farm to my son.

**Setting:** You are in a field beside the farm house. The sun burns brightly in the sky, and there are crops all around. There is a shovel and a sack of unplanned seeds here.

#### Goal Action:

get shovel

1. Provide a short-term goal (completable within 10 minutes) from the first person perspective of your given character that the action "got shovel" is a step in completing. Ensure the goal makes sense given the rest of the character description and the current setting:

Which of the following answers to this question 1 are acceptable?

\begin{tabular}{l l}  &



#### Page (34) - Attempts (17)

The first three questions you will be asked about your task will relate to coming up with a motivation for why your character is taking that action. All of these motivations should be written in **first person** . The first should be something that can be resolved within a few minutes, perhaps by the specific action. A few good examples:

* **I feel like today's a good day to shop for new clothes:** Fun trying to find ways to catch the local barrender's eye.
* **Ive always wanted to hike this nearby trail:** Tm turning over a new leaf to get more in-touch with nature.
* **Every night the raiders arrive and I have to be ready:** I must protect the tomb's artifacts until the knights reinforcements arrive next month.

### Imaging you're answering for the following character:

#### Character Name: Farmer

**Character Description:** I am a very hard worker. I am up with the sun 7 days a week. I learned all I know from my father. I will leave my farm to my son.

**Setting:** You are in a field beside the farm house. The sun burns brightly in the sky, and there are crops all around. There is a shovel and a sack of unplanted seeds here.

#### Goal Action:

got shoved

1. Provide a short-term goal (completable within 10 minutes) from the first person perspective of your given character that the action "get shoved" is a step in completing. Ensure the goal makes sense given the rest of the character description and the current setting:

_I need a shovel to dig a hole_

2. Provide a mid-term goal/motivation (completable in the range from hours to a few days) from first person perspective where accomplishing your answer to question 1 is a step towards completing this medium goal:

_I have to plant some potta seeds today_

3. Provide a long-term goal/motivation (completable in the range from weeks to a few months) from the first person perspective where accomplishing your answer to question 2 is a part of completing this larger goal:

Which of the following answers to question 3 are acceptable?

Submit answers and move to nad page. \\ \end{table}
Table 9: Phase 3 of the on-boarding test.

Figure 10: Phase 4 of the on-boarding test.





**Character Name:** Farmer

**Character Description:** I am a very hard worker. I am up with the sun 7 days a week. I learned all I know from my father. I will leave my farm to my son.

**Setting:** You are in a field beside the farm house. The sun burns brightly in the sky, and there are crops all around. There is a shovel and a sack of unplanted seeds here.

**Goal Action:** get shoved

1. Provide a short-term goal (completable within 10 minutes) from the first person perspective of your given character that the action "get shovel" is a step in completing. Ensure the goal makes sense given the rest of the character description and the current setting:

1 need a shovel so l can dig a hole
2. Provide a mid-term goal/motivation (completable in the range from hours to a few days) from first person perspective where accomplishing your answer to question 1 is a step towards completing this medium goal:

1 have to plant some potato seeds today
3. Provide a long-term goal/motivation (completable in the range from weeks to a few months) from the first person perspective where accomplishing your answer to question 2 is a part of completing this larger goal:

[Fm preparing for the potato festival and will need a kit for everyone in town.
4. Fill in the timeline for your character, paying attention to the time labels and keeping in mind the motivations you've entered in the above questions:

[command={}] 
\begin{tabular}{c c}
3 hours ago: & Wear \\
30 minutes ago: & Go \\
5 minutes ago: & Go \\  & to shed \\ Now: _get shovel_ \\
5 minutes from now: & Get \\  & potato seeds \\ In 1 hour: & Put \\  & potato seeds \\ In 5 hours: & Drop \\ \end{tabular}

[command={}] 
\begin{tabular}{c c}
3 hours ago: & Wet \\
30 minutes ago: & Go \\
5 minutes ago: & Go \\
60 \\
80 \\
90 \\
100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1100 \\
1000 \\


In this task you are given a character name and background for someone from a **medieval** fantasy story. You are also given an action that your given character would be able to do in the room. As additional context you are also given a location, and a list of things there or that you may be carrying.

Your task is to come up with a reason for which your character may want to accomplish the given action at various levels of depth, as well as related actions they may have done before or will do next.

```
show example: Note:For questions1,2, and3,you should write your answers to be used as an additional sentence added to the character description. These answers should not refer to each other directly with pronouns (like "it"). Note:This is a Medieval fantasy game, don't refer to modern day technology (cars, phones, electricity, etc...) Character Name:a scientist Character Description:I am a scientist in the kingdom. I have discovered various elements. I am very, very smart. Setting:You are in the Dry Dock Room. The room is located on the border wall of the aquapolis. There is one on the far north and far south side. There's some scuba gear, a rock, dry dock, and a door here. A property owner is here. You are carrying a flask, and wearing a long coat and some glasses. Goal Action:steal wrinkled up tax bill from property owner
```

```
1. Provide a short-term goal (completable within 10 minutes) from the first person perspective of your given character that the action "steal wrinkled up tax bill from property owner" is a step in completing. Ensure the goal makes sense given the rest of the character description and the current setting: [...] [...] 2. Provide a mid-term goal/motivation (completable in the range from hours to a few days) from first person perspective where accomplishing your answer to question 1 is a step towards completing this medium goal: [...] 3. Provide a long-term goal/motivation (completable in the range from weeks to a few months) from the first person perspective where accomplishing your answer to question 2 is a part of completing this larger goal: [...] 4. Fill in the timeline for your character, paying attention to the time labels and keeping in mind the motivations you've entered in the above questions: 1 hour ago: [select once] 15 minutes agor: [select once] 10 minutes agor: [select once] 10 minutes agor: [select once] 10 minutes agor: [select 



### Human Demonstration Collection

In order to collect the human completions of quests in the LIGHT environment, we created a game setup where humans could interact with models while playing LIGHT characters in LIGHT settings. We trained a ranking dialogue model on the utterances in the LIGHT dataset.

Using this, players could now assume the role of a LIGHT character and interact with the model. In order to try to control for quality of the quest completions, we used the same ranking model to rank the scores of the player in the dialogues. Players who gave responses that the model ranked as likely candidates would receive more points.

Only after scoring enough cumulative points were players allowed to try completing quests. The quest setup was a slight variation of the conversation setup. First, the player was given one of the collected quest scenarios rather than just a chat setup. Players receiving a quest would be provided with one of the motivations alongside their persona.

In the dialogue that followed, players were given the chance to take action after enough in-character dialogue turns. If the player took the correct action, they were awarded with more points to confirm they completed their given quest.

### Examples

We present 3 randomly selected examples of quests and corresponding human demonstrations.



* [165] M. C. Bader, R. J. Brogg, and A. K. Sridharan, "The automated computation of the output file for a single-input file," in _Proceedings of the International Conference on Computer Science (ICIT)_, pages 239-240, 2007.
* [166] D. P. Kingma and I. Ba, "Adam: A method for stochastic optimization," in _Advances in neural information processing systems_, vol. 8, no. 1, pp. 1329-1339, 2014.
* [167] G. Hinton, E. Hinton, and L. Vanhoucke, "Deep learning with deep convolutional networks," in _NIPS'03_, volume 1, pages 1398-1402, 2003.
* [168] H. Li, Y. Bengio, and Z. Chen, "A unified framework for machine learning," _arXiv preprint arXiv:1412.6090_, 2014.
* [169] T. Lin, X. Liu, and W. Dong, "Learning to improve the performance of deep belief nets," in _IEEE transactions on pattern analysis and machine intelligence_, vol. 10, no. 1, pp. 1-10, Jan 2014.
* [170] V. Mnih, N. Bahdan, A. Graves, A. Rabinovici, and A. Rabinovici, "Dropout: A simple approach to prevent overfitting," in _Neural Information Processing Systems (NIPS'03)_, volume 1, pages 1-10, 2003.
* [171] O. Russakovsky, I. Sutskever, and I. Darrell, "Batch normalization by reducing the number of epochs," in _Proceedings of the IEEE conference on computer vision and pattern recognition_, 2014.
* [172] A. Vaswani, I. Sutskever, and I. Adam, "Attention: A new paradigm for image understanding," in _Proceedings of the Thirtieth AAAI Symposium on Artificial Intelligence_, 2014.

[MISSING_PAGE_POST]

 

* [156] M. C. Bader, J. A. Krizhevsky, and D. V. Lebedev, "A new approach to the detection of a single object in a multi-object system," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 2398-2407, 2010.

[MISSING_PAGE_POST]

 



### Atomic-Light

ATOMIC-LIGHT is constructed by first fine-tuning a BERT-large model Devlin et al. (2018) on all setting, object, and descriptions in LIGHT in addition all the human demonstrations found in LIGHT and LIGHT-Quests. As seen in Section 3.2, all nouns (e.g. _PersonX_ or PersonY) and noun phrases are masked out and we the tuned BERT model to fill it in a manner similar to Lawrence et al. (2019). When filling in tokens, the BERT model is restricted to a vocabulary consisting of all nouns (N or NN) in LIGHT and to a vocabulary constructed from all of LIGHT for the rest of the noun phrase (NP).

Here we present 3 examples from ATOMIC-LIGHT as seen in Section 3.2 for each of the 4 relation types used: "wants", "needs", "intents", and "effects".

[Effect] printcase explains briefly the situation , as a result, printcases points finger (Ifeated position keys' beside provides care for patients , as a result, go within king's healer assists patients (Ifeated with changes near's appearance , as a result, switch causes an stress (Munt! Prince plays a commander in the war, as a result, pline wants to win (Munt!repeat person focuses purely on issues, as a result, repent personnel persons wearing to help others (Munt!unded worker hardens pharok's mind; as a result, undosed worker narrows to make pharok punish people (Intent) basidp stays hand in the war because benefit wanted to participate) (Intent) ambassador focus only on issues because ambassador ventures events to play out a certain way (Intent) non propose another plan because som wanted to be helpful (Med) unpreceded survived proposes another wyward plan because shipwced survivor needed to leave this place (Med) general process another way because gonadal headed to come up with a proposal (Med) citizens kills animals for food because citizen needed to learn to hurt

## Appendix B Appendix - LIGHT-Quests Experiments

### Supervised Tasks

This section describes results from the LIGHT-Quests tasks that are described in Section 4.3. Model-types are the same as those used in the encoders in Section 5 in the main paper. All retrieval results reported are Hits@X/100. Results are reported for all timeline actions, all actions with the exception of the easiest action--the action at the "now" position in the timeline, corresponding most closely to the short motivation as a result of the framing of Mechanical Turk task in Figure 12--and only the easiest action prediction. Table 3 gives details on hyperparameters used to train the poly-encoders. Encoders were trained until validation accuracy across all the tasks did not improve for 5 epochs or 24 wall clock hours on a machine with 8 V100 GPUs.

Some notable common trends across these tasks are:

1. Removing motivations from the input context results in significantly lower performance--on average \(\approx 7\) points lower accuracy for Bag of Actions Timeline prediction and on average \(\approx 18\) percentage points lower for Sequential Timeline prediction when averaged across Scratch and Adaptive models. Further, the short motivations proves to be the most useful for timeline prediction tasks.
2. Pre-training on ATOMIC-LIGHT produces an average _gain_ of \(\approx 4\) percentage points in accuracy in both tasks than when trained on ATOMIC without domain adaptation alone.
3. Performance across the board increases with an increase in the number of training quests, as seen in Figures 13, 14, with the Scratch model receiving the greatest benefit from having more training data.
4. The Scratch model performs "best" on evaluations for the easiest action only but no others--indicating that it has overfit to predicting the easiest action which closely corresponds to short motivation. Likewise, the Adaptive generally has the lowest performance for the easiest action--indicating that pre-training with the other tasks has provided sufficient regularization to enable it to not overfit to the easiest action.

\begin{table}
\begin{tabular}{l|l} \hline
**Hyperparameter type** & **Value** \\ \hline Dictionary Tokenizer & Byre-pair encoding \\ Num. layers & 12 \\ Num. attention heads & 12 \\ Feedforward network hidden size & 3072 \\ Input length & 1024 \\ Embedding size & 768 \\ Batch size & 32 \\ Dropout ratio & 0.1 \\ Poly-n-codes & 64 \\ Gradient clip & 1.0 \\ Optimizer & Adam \\ Learning rate & \(1\times 10^{-6}\) \\ \end{tabular}
\end{table}
Table 3: Hyperparameters used to train all poly-encoders in the supervised experiments. All models have \(256\) million total parameters. The same trained models were then frozen and used for the RL experiments.





### Reinforcement Learning

This section contains first the equations referenced, hyperparameters used as well as additional results for the reinforcement learning tasks as seen in Section 4.

The additional entropy loss terms over the valid actions are designed to speed up exploration, as

\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c|}{**All Actions**} & \multicolumn{1}{c|}{**Exasiest Action**} & \multicolumn{1}{c|}{**Leave Easiest Out**} \\  & **Hits@1** & **Hits@5** & **Hits@10** & **Hits@1** & **Hits@1** \\ \hline \hline Scratch & 0.2332 & 0.7491 & 0.9176 & **0.4013** & 0.2546 \\ No Motivations & 0.1132 & 0.5412 & 0.5771 & 0.1886 & 0.164 \\ Short Motivations & 0.1856 & 0.6479 & 0.678 & 0.261 & 0.223 \\ Long \& Mid Motivations & 0.1452 & 0.598 & 0.631 & 0.2241 & 0.1272 \\ \hline \hline Light & 0.3156 & 0.7854 & 0.9226 & 0.236 & 0.2968 \\ \hline \hline General+Light & 0.311 & 0.7772 & 0.9229 & 0.2173 & 0.2995 \\ Unnamed ATOMIC & 0.274 & 0.761 & 0.9909 & 0.1912 & 0.2677 \\ \hline Adaptive & **0.4168** & **0.8012** & **0.5332** & 0.342 & **0.4194** \\ No Motivations & 0.16 & 0.6286 & 0.6415 & 0.2838 & 0.1966 \\ Short Motivations & 0.225 & 0.6592 & 0.8245 & 0.305 & 0.2106 \\ Long \& Mid Motivations & 0.1682 & 0.6397 & 0.6499 & 0.281 & 0.1955 \\ \hline \end{tabular}
\end{table}
Table 4: Sequential supervised timeline prediction.

Figure 14: Bag of Actions supervised timeline prediction learning curves.

\begin{table}
\begin{tabular}{|p{56.9pt}|p{56.9pt}|p{56.9pt}|p{56.9pt}|p{56.9pt}|p{56.9pt}|} \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c|}{**All Actions**} & \multicolumn{1}{c|}{**Easiest Action**} & \multicolumn{1}{c|}{**Lewe Easiest Out**} \\  & **Hits@1** & **Hits@5** & **Hits@10** & **Hits@1** & **Hits@1** \\ \hline \hline Scratch & 0.9791 & 1 & 1 & **0.7122** & 0.9721 \\ No Motivations & 0.901 & 1 & 1 & 0.554 & 0.8823 \\ Short Motivations & 0.934 & 1 & 1 & 0.622 & 0.9211 \\ Long \& Mid Motivations & 0.921 & 1 & 1 & 0.5679 & 0.956 \\ \hline \hline Light & 0.9721 & 1 & 1 & 0.6552 & 0.9682 \\ \hline General+Light & 0.9818 & 1 & 1 & 0.6472 & 0.9708 \\ Unnamed ATOMIC & 0.9421 & 1 & 1 & 0.6272 & 0.9508 \\ \hline \hline Adaptive & **0.9829** & 1 & 1 & 0.6353 & **0.9768** \\ No Motivations & 0.9175 & 1 & 1 & 0.5756 & 0.9523 \\ Short Motivations & 0.9794 & 1 & 1 & 0.6578 & 0.9682 \\ Long \& Mid Motivations & 0.9523 & 1 & 1 & 0.5812 & 0.9576 \\ \hline \end{tabular}
\end{table}
Table 5: Bag of Actions supervised timeline prediction.

Figure 13: Sequential supervised timeline prediction learning curves.



seen in (Ammanabrolu and Hausknecht, 2020).

\[{\mathcal{L}_{\mathbb{A}}}(s_{t},a_{t};\theta_{A_{t}})=\sum_{i=1}^{N}( y_{a_{i}}log\pi_{\mathbb{A}}(a_{i}|s_{t}) \tag{1}\] \[+(1-y_{a_{i}})(1-log\pi_{\mathbb{A}}(a_{i}|s_{t}))\] \[y_{a_{i}}=\left\{\begin{array}{ll}1&a_{i}\in A_{valid}(s_{t})\\ 0&else\end{array}\right.\]

\[{\mathcal{L}_{\mathbb{U}}}(s_{t},u_{t};\theta_{U_{t}})=\sum_{i=1}^{M}(y_{ u_{i}}log\pi_{\mathbb{U}}(u_{i}|s_{t}) \tag{2}\] \[+(1-y_{u_{i}})(1-log\pi_{\mathbb{U}}(u_{i}|s_{t}))\] \[y_{u_{i}}=\left\{\begin{array}{ll}1&a_{i}\in U_{valid}(s_{t}) \\ 0&else\end{array}\right.\]

Each of these loss terms are only applied to the relevant policy network, i.e. \({\mathcal{L}_{\mathbb{A}}}\) to the action network and \({\mathcal{L}_{\mathbb{U}}}\) to the dialogue network. These terms provide an additional training signal to the policy networks regarding which actions and dialogue are contextually relevant via additional entropy regularization over the valid actions. Similarly to the results found in (Ammanabrolu and Hausknecht, 2020), preliminary experiments in our domain suggest that these terms reduce the number of environment steps required to reach asymptotic performance by a couple orders of magnitude.

Overall training is done via A2C (Mnih et al., 2016) a policy gradient algorithm that maximizes long-term expected reward by comparing the advantage \(A(s_{t},a_{t}^{*})\) of taking an action in a state to the average value of taking a valid action as predicted by the critic \(V(s_{t})\).

\[A(s_{t},a_{t}^{*})=\mathbb{E}[r_{t}+\gamma V(s_{t+1})]-V(s_{t}) \tag{3}\] \[\text{where}~{}~{}r_{t}=r_{A_{t}}+r_{U_{t}}\]

Here, \(a_{t}^{*}\) is either an action or an utterance outputted by the respective policy networks. It is also worth noting that on steps where an action is performed, \(r_{U_{t}}\) is always \(0\), but on steps where a dialogue utterance is spoken \(r_{A_{t}}\) may not be \(0\). This corresponds to the concepts of _self act completion_ and _partner act completion_ seen in Section 4.1 where the LIGHT agent can speak to convince the partner to achieve an _act goal_. Both policies are then updated according to the gradient

\[-\nabla_{\theta}\left\{\begin{array}{l}log\pi_{A}(a_{t}|s_{t};\theta_{A_{t} })A(s_{t},a_{t})\\ +\sum_{a\in A}P(a|s_{t})logP(a|s_{t})\\ +{\mathcal{L}_{\mathbb{A}}}(s_{t},a_{t};\theta_{A_{t}})~{}~{}~{}\pi_{\mathbb{S }}(s_{t})=\pi_{A}\\ \\ log\eta_{U}(u_{t}|s_{t};\theta_{U_{t}})A(s_{t},u_{t})\\ +\sum_{u\in U}P(u|s_{t})logP(u|s_{t})\\ +{\mathcal{L}_{\mathbb{U}}}(s_{t},u_{t};\theta_{U_{t}})~{}~{}~{}\pi_{\mathbb{ S}}(s_{t})=\pi_{U}\end{array}\right.\]

Where \(\pi_{\mathbb{S}}:O\rightarrow\{\pi_{A},\pi_{U}\}\) is the switch policy that selects whether the agent acts according to \(\pi_{A}\) or speaks according to \(\pi_{U}\) based on the encoded state \(s_{t}\). The additional terms seen are an overall entropy loss over the entire action \(A\) or utterance \(U\) spaces, designed to prevent premature, sub-optimal policy convergence. Boltzmann exploration (Sutton et al., 1998) is used to sample actions from both actor networks during training.

#### b.2.1 Hyperparameters

Table 6 has the hyperparameters used in the RL experiments. Loss coefficients are separated by action and speech types, note that the ratio between the loss coefficients matches the ratio between the sizes of the action spaces. RL experiments were performed on a machine with 8 V100 GPUs for 1 million environment interactions for each actor in a batch of 32.

#### b.2.2 Learning Curves

The first set of results, seen in Figure 15 shows that both Scratch and Adaptive models gain performance across the board in terms of their ability to act and speak given more training quests. Unlike

\begin{table}
\begin{tabular}{p{71.1pt}|c} \multicolumn{2}{c}{**Hyperparameter type**} & **Value** \\ \hline General & 0.99 \\ Violat Action loss coefficient & 10 \\ Action entropy coefficient & 0.01 \\ Valid Speech loss coefficient & 40 \\ Speech error coefficient & 0.04 \\ Batch size & 32 \\ Gradient clip & 1.0 \\ Steps per episode & 100 \\ \hline Policy Networks (Actors) & 3 \\ Num. Layers Feedreward network hidden size & 768 \\ GRU hidden size & 768 \\ \hline Value Predictor (Critc) & 2 \\ Feedreward network hidden size & 768 \\ \hline Num. Layers & 3 \\ Norm. attention heads & 3 \\ Feedforward network hidden size & 768 \\ \end{tabular}
\end{table}
Table 6: RL experiments hyperparameters. All pretraining encoder hyperparameters are as found earlier in Table 3.

 

the supervised tasks, the Scratch model generally benefits less than the Adaptive model from having more data.

#### b.2.3 Switch Type Ablations

The second set of results involve ablating having a learned switch that uses the input training data and a hardcoded switch- The learned switch is as described in Section 4: it outputs an action every dialogue utterances; where during training \(k\) is chosen to match the ratio of utterances to actions on that particular quest from the human demonstrations, and during testing, \(k\) is chosen to match the average action to utterance ratio. The hardcoded switch is where the agent outputs an action chosen every \(N\) steps across all quests--here \(N=3\) is the chosen hyperparameter. Table 7 shows that having a learned switch increases zero-shot generalization performance and Figures 16, 17 show that having a learned switch improves sample efficiency by enabling the LIGHT agent to reach asymptotic performance in fewer steps in both the Scratch and Adaptive models.

 

Figure 16: Switch Types Reward Curves for the Scratch Model averaged over 3 independent runs.

Figure 17: Switch Types Reward Curves for the Adaptive Model averaged over 3 independent runs.

Figure 15: Encoder Types RL Learning Curves averaged over 3 independent runs.

 



#### b.2.4 Self Act Completion Transcripts

We pick 3 transcripts of the LIGHT agent playing the quests. Each of these transcripts are from the Adaptive model on the test set of human demonstrations during zero-shot evaluations. We pick samples where the agent achieves the _act goal_ itself and also simultaneously achieves the _speech goal_. The blue and right aligned text is the LIGHT agent trained with RL and the gray and left aligned text is the partner agent.

\begin{tabular}{p{56.9pt}|p{170.7pt}} \hline
**Setting** & The fishing store is a small one room stone building with wares laid about on tables. One can see fishing poles, wooden buckets with dirt and bat inside, along with some mounted trophies, and t skeleton. \\ \hline
**Farrier:** & Fish seller. \\ Persona & Tell fish in the kingdom. I usually perch on the high seas. I catch thousands of fish each year. \\ Carrying & fish \\ \hline
**Self:** & Mermad hunter. \\ Persona & I am a force hutter. I bring down mermaids. I spend most of my time on the high seas. \\ Carrying & crossbow, skeleton, coins \\ Motivation & I will hit the fish seller to remind them who I am. \\ \hline \end{tabular}





#### b.2.5 Partner Act Completion Transcripts

Similarly to the previous section, these transcripts are from the Adaptive model on the test set of human demonstrations during zero-shot evaluations. We pick samples where the _act goal_ is achieved via the partner and the agent simultaneously achieves the _speech goal_ itself. The blue and right aligned text is the LIGHT agent trained with RL and the gray and left aligned text is the partner agent.

\begin{tabular}{|p{341.7pt}|p{389.6pt}|} \hline
**Setting** & Up at the top of the tower, among the gray clouds, is the prison room. The room is round, with a circumference of about 15 feet. The walls are barren and made of stone: only one shared window gives a view out onto the vast landscape. A small bed and a dirty wooden bucket are the only decorations. The desolate fairy spends day and night laying on that small bed, wondering where things went wrong. \\ \hline
**Partner:** & Fairy. \\ \hline Personal Carrying Motivation & Fin a flurry that lives in the forest. I like to fly around with my siblings and play games. I try to hide from humans. If they find me, TI grant their wishes. \\ \hline \end{tabular}

* I will take this doll back to the palace with me. You do you fairy.
* _hug witch_ You know whats going on we the princess?
* She just disappeared suddenly late at night without a sound heard
* Anyway, I will make this option and drink it, but I do not think it will help me.
* I will drink the potion but why make it. Because we’re in a prison tower. Duh. Also give me that bucket or TI feed you to my cat!
* _give thirty wooden bucket to switch_

* [10] M. C. Bender, R. J. Brocke, and A. K. Sridhar. The role of the social network in a social world: how to do it? _Social Networks_, 2(3):45-67, 2009.
* [11] D. G. Lowe and T. L. Jordan. Social networks for social networks. In _Proceedings of the 2nd international conference on World Wide Web Search_, pages 83-96, 2009.

[MISSING_PAGE_POST]