SlowGuess/NAACL_2021_OCR huggingface dataset:
Prithviraj Ammanabrolu† Jack Urbanek‡ Margaret Li‡
Arthur Szlam† Tim Rocktäschel‡ Jason Weston‡
$^{\dagger}$ Georgia Institute of Technology  $\ddagger$ Facebook AI Research
raj.ammanabrolu@gatech.edu
We seek to create agents that both act and communicate with other agents in pursuit of a goal. Towards this end, we extend LIGHT (Urbanek et al., 2019)—a large-scale crowd-sourced fantasy text-game—with a dataset of “quests”. These contain natural language motivations paired with in-game goals and human demonstrations; completing a quest might require dialogue or actions (or both). We introduce a reinforcement learning system that (1) incorporates large-scale language modeling-based and commonsense reasoning-based pre-training to imbue the agent with relevant priors; and (2) leverages a factorized action space of action commands and dialogue, balancing between the two. We conduct zero-shot evaluations using held-out human expert demonstrations, showing that our agents are able to act consistently and talk naturally with respect to their motivations.
There has been a recent improvement in the quality of natural language processing (NLP) and generation (NLG) by machine learning (ML) (Vaswani et al., 2017; Devlin et al., 2018); and in parallel, improvement to goal-oriented ML driven agents in the context of games (Vinyals et al., 2019; Schrittwieser et al., 2019). However, agents that can communicate with humans (and other agents) through natural language in pursuit of their goals are still primitive. One possible reason for this is that many datasets and tasks used for NLP are static, not supporting interaction and language grounding (Brooks, 1991; Feldman and Narayanan, 2004; Barsalou, 2008; Mikolov et al., 2016; Gauthier and Mordatch, 2016; Lake et al., 2017). Text-based games—where players see, act upon, and communicate within a dynamic world using natural
language—provide a platform on which to develop such goal-driven agents.
LIGHT (Urbanek et al., 2019), a large-scale crowdsourced fantasy text-adventure game, consisting of a set of locations, characters, and objectsa possesses rich textual worlds, but without any notion of goals to train goal-driven agents. We present a dataset of quests for LIGHT and demonstrations of humans playing these quests (as seen in Figures 2 and 3), providing natural language descriptions in varying levels of abstraction of motivations for a given character in a particular setting.
To complete these quests, an agent must reason about potential actions and utterances based on incomplete descriptions of the locations, objects, and other characters. When a human is placed in a fantasy setting such as LIGHT, they already know that kings are royalty and must be treated respectfully, swords are weapons, etc.—commonsense knowledge that a learning agent must acquire to ensure successful interactions. To equip agents with relevant priors in such worlds, we domain-adapt the large-scale commonsense knowledge graph ATOMIC (Sap et al., 2019) to the LIGHT fantasy world—to build ATOMIC-LIGHT.
We then introduce a reinforcement learning (RL) system that incorporates large-scale language modeling and the above commonsense-based pretraining. We show that RL is superior to behavior cloning or other supervised training on our data; and that carefully combining pre-training with RL is superior to either.
However, we find that although pre-training can be an effective tool in this setting, it requires more finesse than in the standard supervised setting. In particular, we find that simply pre-training a model on a large "generic" corpus (Sap et al., 2019; Baumgartner et al., 2020) of commonsense/language data or pre-training on the domain specific LIGHT corpus, and then fine-tuning via RL is less effective than training RL from scratch. Furthermore, by
1Data can be found here https://parl.ai/projects/light/
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 807-833 June 6-11, 2021. ©2021 Association for Computational Linguistics
Setting	You are in the Dangerous Precipice. The dangerous precipice overlooks the valley below. The ground slopes down to the edge here. Dirt crumbles down to the edge of the cliff. There's a dragon crescent, a knight's armor, a golden dragon egg, and a knight's fighting gear here. A knight is here. You are carrying nothing.
Partner:
Persona
Carrying	Knight.
I am a knight. I come from a lower-ranking noble family. I serve under the king, as my father did before me. In times of war, I fight on horseback. knight's armor, golden dragon egg, knight's fighting gear
Self:
Persona
Carrying	A dragon.
I am a dragon living in the mountains. I enjoy hoarding treasure. I terrorize the local populace for fun. Nothing.
Figure 1: Setting and character information for both self and partner characters as taken from LIGHT.
Motivations:	Timeline:
Short	I need to recover the dragon egg that was stolen and punish the knight.	-4 hours	go to dangerous precinctive
-15 min	get knights armor from knight
Mid	I need to return the golden dragon egg to my treasure hoard.	-10 min	get golden dragon egg
Now	hit knight
+5 min	put dragon egg on back
Long	I need to build the largest hoard ever attained by any one dragon.	+15 min	eat the knight
+2 hours	go to the mountains
Figure 2: Motivations with different levels of abstractions and corresponding sequence of timeline actions in chronological order for the self character in LIGHT-Quests. There are 7486 quests in total.
Insssssolent pessst! I should immolate you for this tresssspasss.
And why is that, dragon?
Ssstealing my preccciouss golden egg! I'll tell you what, I'll give you 10 sssseconds to amusse me with your sssstory and THEN I'll burn you alive!
You said you wanted to attack me, dragon, did you not?
Go ahead, I'm lissssstening.
get golden dragon egg
Now now! I would have given you that had you asked!
Assssssk for my own property back? What a riduculousss notion
Look here, I told you to watch your mouth and you didn't, so leave or I'll make you leave.
And now threatsss! Thisss is proving to be a mossst engaging conversssation.
hit knight
Give my regasdsss to the valley floor below!
Figure 3: Example of a demonstration of a human (blue shaded) completing the above quest while role-playing as the self character with a partner agent (grey shaded). There are 2111 such human demonstrations of average sequence length 12.92, consisting of 22672 dialogues in total.
carefully combining general and domain-specific pre-training, we observe large improvements over RL from scratch.
In short, the contributions of this paper are three-fold: (1) A dataset of quests, LIGHT-Quests, and a companion fantasy themed commonsense knowledge graph ATOMIC-LIGHT; (2) a reinforcement learning architecture and training methodology that use these datasets to create goal-driven agents that act and speak in the LIGHT environment; and (3) Empirical zero-shot evaluations based on human quest demonstrations and an analysis of large-scale transformer-based pre-training trends in static vs. interactive settings, showing that we have trained agents that act consistently and speak naturally with respect to their motivations.
We focus on four major areas of related work: text-based game-playing, goal-oriented dialogue, commonsense reasoning in language, and general language-informed RL.
Text-based game-playing. (Côté et al., 2018)
introduce TextWorld, a framework for procedurally generating text-based games via grammars, and (Yuan et al., 2018; Yin and May, 2019; Adolphs and Hofmann, 2019; Adhikari et al., 2020) build agents that operate in this environment—focusing on aspects such as efficient exploration and zero-shot generalization to new, procedurally generated environments. Similarly, (Hausknecht et al., 2020) introduce Jericho, a framework and series of baseline agents for interacting with human-made text-games such as Zork (Anderson et al., 1979). This resulted in agents developed by works such as (Zahavy et al., 2018; Ammanabrolu and Hausknecht, 2020), aiming to learn to execute contextually relevant actions. Other works such as (Narasimhan et al., 2015; He et al., 2016) explore how to best factorize such text-game action spaces. None of these works consider agents with motivations and personas nor require any dialogue.
Goal-oriented dialogue. This form of dialogue has traditionally been closely related to specific tasks useful in the context of personal assistants with dialogue interfaces (Henderson et al., 2014;
El Asri et al., 2017). RL has been studied for such tasks, usually to improve dialogue state management (Singh et al., 2000; Pietquin et al., 2011; Fatemi et al., 2016) and to improve response quality (Li et al., 2016). In particular, the negotiation tasks of (Yarats and Lewis, 2017; Lewis et al., 2017), where two agents are trying to convince each other to perform certain actions, are related to the tasks in LIGHT-Quests. These works all lack environment grounding and the notion of diverse agent motivations.
Commonsense reasoning in language. Works such as (Bosselut et al., 2019; Guan et al., 2020) focus on pre-training transformer-based language learning systems with large-scale commonsense knowledge graphs such as ATOMIC (Sap et al., 2019) and ConceptNet (Speer and Havasi, 2012) for use in knowledge graph completion and story ending generation respectively. (Fulda et al., 2017; Ammanabrolu and Riedl, 2019; Ammanabrolu et al., 2020; Murugesan et al., 2020) look at commonsense reasoning in interactive environments, with the former focusing on affordance extraction using word embeddings and the latter three on transferring text-game playing skills via pretraining using question-answering and large-scale knowledge graphs.
Language-informed reinforcement learning. (Luketina et al., 2019) provide an overview of RL informed by natural language. Of these works, the ones most related to ours are those falling into the category of instruction following—where an agent's tasks are defined by high level instructions describing desired policies and goals (MacMahon et al., 2006; Kollar et al., 2010). Visual and embodied agents using natural language instructions (Bisk et al., 2016; Kolve et al., 2017; Anderson et al., 2018) or in language-based action spaces (Das et al., 2017) utilize interactivity and environment grounding but have no notion of agent motivations, nor make any attempt to explicitly model commonsense reasoning. Perhaps closest in spirit to this work is (Prabhumoye et al., 2020), where they use artificially selected goals in LIGHT and train RL agents to achieve them. Similarly to the others, this work does not contain the motivations provided by LIGHT-Quests nor any modeling of commonsense reasoning. Further, they limit their RL problem to 1 and 3-step trajectories that only involve speech, and no actions—compared to the human demonstrations in LIGHT-Quests which contain both actions
and speech sequences of average length 12.92.
3 LIGHT-Quests and ATOMIC-LIGHT
This section first provides a brief overview of the LIGHT game environment, followed by descriptions of the LIGHT-Quests and ATOMIC-LIGHT datasets used in this paper.
Background. The LIGHT game environment is a multi-user fantasy text-adventure game consisting of a rich, diverse set of characters, locations, and objects (1775 characters, 663 locations, and 3462 objects). Characters are able to perform templated actions to interact with both objects and characters, and can speak to other characters through free form text. Actions in text games generally consist of verb phrases (VP) followed optionally by prepositional phrases (VP PP). For example, get OBJ, put OBJ, give OBJ to CHAR, etc.. There are 13 types of allowed verbs in LIGHT. These actions change the state of the world which is expressed to the player in the form of text descriptions.
3.1 LIGHT-Quests
Figures 1, 2, and 3 summarize the data that we collected for LIGHT-Quests. Data is collected via crowdsourcing in two phases, first the quests then demonstration of humans playing them. During the first phase, crowdworkers were given a setting, i.e. situated in a world, in addition to a character and its corresponding persona and asked to describe in free form text what potential motivations or goals could be for that character in the given world. The kind of information given to the crowdworkers is seen in Figure 1. Simultaneously, they were also asked to provide a sequence of seven timeline actions—one action that needs to be completed now and three before and after at various user-defined intervals—for how the character might go about achieving these motivations.
Given the information in Figure 1, the crowd-workers completed the above outlined tasks and produce data as seen in Figure 2. Motivations come in three levels of abstraction—short, mid, and long—corresponding to differing amounts of the timeline. For example, the short motivation is always guaranteed to correspond most closely to the now position on the timeline. Action annotation is pre-constrained based on the classes of verbs available within LIGHT. The rest of the action is completed as free form text as it may contain novel entities introduced in the motivations. There are
5982 training, 756 validation, and 748 test quests. Further details regarding the exact data collection process and details of LIGHT-Quests are found in Appendix A.1.1.
After collecting motivation and timelines for the quests, we deployed a two-player version of the LIGHT game, letting players attempt the quests for themselves in order to collect human demonstrations. Figure 3 shows an example human expert demonstration of a quest. Players were given a character, setting, motivation, and a partner agent and left to freely act in the world and talk to the partner in pursuit of their motivations. The partner agent is a fixed poly-encoder transformer model (Humeau et al., 2020) trained on the original LIGHT data as well as other human interactions derived via the deployed game—using 11k utterances in total. Players first receive a role-playing score on a scale of 1-5 through a Dungeon Master (DM), a learned model that ranks how likely their utterances are given the current context. Once they have accumulated a score reaching a certain threshold, they are allowed to perform actions. We employ this gamification mechanism to encourage players to role-play their character persona and its motivations, leading to improved user experience and data quality (Horsfall and Oikonomou, 2011). They are then given further reward if the actions they perform sequentially match those on the timeline for the given quest. The game ends after a maximum of six turns of dialogue per agent, i.e. twelve in total. The average sequence of a human demonstration is 12.92, with an average action sequence length of 2.18 and dialogue of 10.74. There are 1800 training, 100 validation, and 211 test human expert demonstrations after the data was filtered. Additional details and examples are found in Appendix A.2.
3.2 ATOMIC-LIGHT
Commonsense reasoning is a critical cornerstone when building learning agents that navigate spaces such as LIGHT-Quests. To this end, we domain-adapt the large-scale commonsense knowledge base ATOMIC (Sap et al., 2019) to LIGHT. ATOMIC contains information relevant for everyday commonsense reasoning in the form of typed if-then relations with variables. ATOMIC is organized into a set of events, e.g. "X puts X's trust in Y" and annotated relation types such as "needs", "wants", "attributes", and "effects" that label the effects. It is designed to be a general atlas of com
monsense data and so is neither dependent on a specific environment or a character's persona and motivations.
To construct ATOMIC-LIGHT, we specifically use the relations for "intent", "effects", "wants" and "needs" and expand the  $\langle$  subject, relation, object  $\rangle$  triples found in the graph into templated natural language sentences. These sentences are then rewritten to better reflect the fantasy LIGHT domain. Named entities and other noun phrases in ATOMIC are masked out and filled in using BERT (Devlin et al., 2018) fine-tuned using a masked language model loss on the entire LIGHT and LIGHT-Quests data. We investigate the benefits of such domain adaptation on downstream tasks in Section 4.3. An example of a clause using the wants relation in ATOMIC is as follows, "PersonX puts PersonX trust in PersonY, wants, rely on PersonY." In ATOMIC-LIGHT, this is rewritten to: "The merchant puts the merchant's trust in the guard, as a result the merchant wants to rely on the guard." Similarly, an example of an effect using the needs relation is, "Before, the merchant puts the merchant's trust in the
guard, the merchant needs to be friends with the guard." ATOMIC-LIGHT contains 216686 training, 35340 validation, and 38565 test samples. Further details of the construction of this dataset are found in Appendix A.4.
4 Agents that Act and Speak
This section describes the creation of the agents that learn to act and speak conditioned on their motivations in the LIGHT environment. The overall architecture and training are first outlined, followed by a detailed discussion on types of encoder pretraining.
4.1 LIGHT RL Environment
The environment as seen in Figure 4 consists of three components. The first is a partner agent, which is a model trained to play other agents in the game, as in (Prabhumoye et al., 2020). Next is the game engine, which determines the effects of actions on the underlying game graph (Urbanek et al., 2019). Finally, there is the Dungeon Master (DM), which is trained to score the naturalness of dialogue.
Partner Agent. The partner agent is a poly-encoder transformer model (Humeau et al., 2020)
Figure 4: Overall RL Switch architecture and process. Blue shaded components can be pre-trained and Red shaded components are trained with RL. Solid lines indicate gradient flow.
that is pre-trained on the Reddit dialogue corpus, then on LIGHT and the human demonstrations of LIGHT-Quests. Following the format seen in Figure 3, the partner agent does not have a motivation itself but is trained to react to agents with motivations. Following (Prabhumoye et al., 2020), we keep the partner model fixed during the episodes where the LIGHT agent trains to ensure that it retains natural English semantics—avoiding the problem of language drift by learning an emergent language with that must agree with the partner's usage (Lee et al., 2019).
Action Rewards via the Game Engine. All actions, either those of the agent-in-training or the partner agent, are processed by the engine, checking for goal state completion—hence known as act goals. For example, if the LIGHT agent had the motivation to acquire a sword, the goal could be completed via a:
1. self act completion: where the agent acquires a sword itself by picking it up, stealing it, convincing the partner to drop theirs so you can pick it up, etc.
2. partner act completion: where the agent uses speech to convince their partner to achieve the goal for them (e.g., by persuading the partner to give them the sword).
Reaching an act goal provides reward  $r_a$  of 1 and 0 otherwise. At each step, the engine also provides us with the set of valid actions. These are the subset of the action space  $A$  which are guaranteed to be a valid change to the world from the current state  $s_t$ , i.e., an action to give your partner a sword cannot be valid unless you possess the sword.
Speech Rewards via the Dungeon Master. Following prior works on using transformers for
automatic evaluation of natural language generation (Sellam et al., 2020), we utilize a learned model—the Dungeon Master (DM)—to score the agent's ability to speak. The DM used here is a poly-encoder model trained on collected human quest demonstrations as well as the original conversations in LIGHT. It is conditioned on quests and motivations and thus able to provide a (noisy) indication of how natural the agent's dialogue utterances are given its immediate context, similarly to the function of the DM during the data collection process. Given the dialogue portion of a human quest demonstration of length  $n$ , the DM returns a reward  $r_u$  of  $\frac{1}{2n}$  if an utterance was in the demonstration (for a maximum of one time per episode for each utterance from the demonstration). A further  $\frac{1}{2n}$  is given each time the utterance is scored as being within the top- $k$  most likely utterances by the DM. This naturalness objective will be hence referred to as a speech goal. These rewards thus also denser than act goals, helping the agent learn overall. Further, similarly to the game engine, the DM also provides a set of  $M$  valid utterances which are the  $M$  most likely dialogue candidates from the candidate set for the current context.
4.2 Training a LIGHT agent with Switch Reinforcement Learning
The overall architecture of our agent is shown in Figure 4. It consists of an encoder, a switch, an action network, and a dialogue network. First, we construct the action spaces—factorized into actions and utterances. The possible actions are the set of all actions taken in the demonstrations (4710 total) and the possible utterances are all utterances from the demonstrations (22672 total). The encoder network processes the setting, persona, motivation, as
well as the full history of actions and dialogues performed by the agent and the partner, input as a text sequence. The features from the encoder, which here are the hidden states at the final layer of a transformer, are used as input by all following components of the agent. In Section 5 we show how different encoder training data affects the model.
Next, a switch module makes the decision regarding whether the agent should act or talk in the current context and activates the corresponding policy network. In this work, the switch is simple: it outputs an action every  $k$  dialogue utterances; where during training  $k$  is chosen to match the ratio of utterances to actions on that particular quest from the human demonstrations, and during testing,  $k$  is chosen to match the average action to utterance ratio. Both the action and dialogue policies consist of a single GRU layer followed by an  $n$ -layer feed-forward network given input features from the encoder. Once the LIGHT agent has output an utterance or action, it is processed by the environment—the partner agent, the game engine and the DM.
We use A2C (Mnih et al., 2016) to train the LIGHT agent, treating the two policy networks as two separate actors with a shared critic. The shared critic is motivated by the concepts of self act completion and partner act completion seen in Section 4.1 where the LIGHT agent can speak to convince the partner to achieve an act goal. Each agent in a batch is initialized via priority sampling (Graves et al., 2017) with a different quest, i.e. quests that the agent has historically successfully completed less often are given a greater weight when sampling from the pool of all possible training quests. In addition to a normal entropy regularization term, we also add a regularization term that encourages the models to produce "valid" outputs as judged by the game engine and the DM for actions and utterances respectively. Additional training details are found in Appendix B.2.
4.3 Encoder Pre-training Tasks
Prior work on commonsense reasoning in supervised natural language learning (Bosselut et al., 2019) suggests that the encoder is key to overcoming the challenges posed by the LIGHT-Quests dataset even in an RL setting. We describe a series of encoder pre-training tasks, designed to help the LIGHT agent either act more consistently or speak more naturally.
ATOMIC-LIGHT As seen in Section 3, ATOMIC-LIGHT is a (domain-adapted) fantasy commonsense knowledge graph, and as such provides priors for an agent on how to act consistently in the world. For example, given a clause such as "The knight wishes to slay the dragon, as a result the knight needs to acquire a sword," the task would be to predict the underlined text—a form of knowledge graph completion (Wang et al., 2017).