Title: I LOVE YOUR CHAIN MAIL! MAKING KNIGHTS SMILE IN A FANTASY GAME WORLD

Abstract: Dialogue research tends to distinguish between chit-chat and goal-oriented tasks. While the former is arguably more naturalistic and has a wider use of language, the latter has clearer metrics and a more straightforward learning signal. Humans effortlessly combine the two, and engage in chit-chat for example with the goal of exchanging information or eliciting a specific response. Here, we bridge the divide between these two domains in the setting of a rich multi-player text-based fantasy environment where agents and humans engage in both actions and dialogue. Specifically, we train a goal-oriented model with reinforcement learning via selfplay against an imitation-learned "chit-chat" model with two new approaches: the policy either learns to pick a topic or learns to pick an utterance given the top-K utterances. We show that both models outperform a strong inverse model baseline and can converse naturally with their dialogue partner in order to achieve goals.

1. INTRODUCTION:

In the literature on artificial dialogue agents, a distinction is often made between "goal-oriented" dialogue, where an agent is tasked with filling slots or otherwise obtaining or disseminating specified information from the user to help complete a task, and "chit-chat", where an agent should imitate human small talk. Modeling goal-oriented dialogue can have advantages over chit-chat imitation as it gives clearer metrics of success and perhaps more meaningful learning signals; but goal-oriented dialogue data is often more specialized, covering only a narrow slice of natural language. Current goal-oriented datasets study setting like booking restaurants or airline tickets, or obtaining weather information, as standalone tasks (Raux et al., 2005; Henderson et al., 2014; Bordes et al., 2017; El Asri et al., 2017; Budzianowski et al., 2018) . Chit-chat agents, by contrast, might focus on coarse statistical regularities of dialogue data without accurately modeling the underlying "meaning"; but the data often covers a much wider space of natural language. For example, Twitter or Reddit chitchat tasks (Li et al., 2016a; Yang et al., 2018; Mazaré et al., 2018) cover a huge spectrum of language and diverse topics. Chit-chat and goal-oriented dialogue are not mutually exclusive: when humans engage in chit-chat, their aim is to exchange information, or to elicit specific responses from their partners. Modeling such goals, however, is made difficult by the fact that it requires large amounts of world knowledge, and that goals in real life are implicit.

In this work, we study goal-oriented dialogue agents in the setting of a multi-player text-based fantasy environment (Urbanek et al., 2019) . The environment is built on top of a game engine that grounds actions and reference objects, and thus codifies a body of world-knowledge. Although the interactions between objects and characters are simulated, the choice and types of interactions, the text used to describe them, and the dialogues between characters, are "natural" and wide-ranging, having been collected from human crowdworkers. We define the general task of, given a particular character in a particular scenario (location, set of objects and other characters to interact with) to conduct open-ended dialogue such that a given action is executed in the future by their dialogue partner. The given action could be an emote action (smile, laugh, ponder, . . . ) , or a game action (wear chain mail, drink mead, put glass on table, . . . ). The richness of the environment means that there are a huge set of possible tasks and scenarios in which to achieve a wide range of actions. Thus, this task is ideally suited for bridging the divide between goal-oriented and chit-chat dialogue, combining clearer metrics and learning signals on the one hand, with the richness and complexity of situated but open-domain natural language on the other.

Persona 1: Village youth Persona 2: Horse I am a youth who lives in the village. I am adventurous and I live on a farm. I work for humans. curious about everything. I get bored easily, so I will always try I like hay. to slip away from mundane daily tasks to find something more interesting to do. I am smart for my age and a bit smug.

Setting: Royal stables, Outside Castle -The royal stables hold all of the horses of the noble people living in the castle. They are very well constructed, consisting of expert mason work and fixtures of some sort of heavy, forest-scented wood. Each horse has a large stall, lined with fresh hay replenished by stable hands daily. On the walls of stable, there are tools used to care for and groom the horses, hung on hooks. The stables are lit by lanterns hung throughout the stables.

Figure 1 : Example episode from the LIGHT dataset, consisting of an environment (location setting, characters with given personas, objects), utterances and game actions. There are 10,777 such humanhuman gameplay episodes, and a rich world of 663 locations, 1755 characters and 3462 objects.

We train models to achieve these tasks using reinforcement learning (RL) and a type of self-play between two agents. The first agent, which we call the environment agent, is trained with imitation learning on human-human interactions (game actions, utterances and emotes) and subsequently kept fixed. The second agent, the RL agent, is trained to conduct dialogue given the goal, and the two agents interact within a given environment until the goal is either reached or a given number of turns has expired. At that point, rewards are given, and the RL agent is updated. We compare agents that have been trained to imitate human actions given a goal (an "inverse model") to two different RL approaches: optimizing actions with latent discrete variables (topics), or via rewarding actions sampled from the model (via the top-K outputs). We show that both types of RL agent are able to learn effectively, outperforming the inverse model approach or a vanilla chit-chat imitation baseline, and can converse naturally with their dialogue partner to achieve goals.

2. GAME ENVIRONMENT:

We work in the LIGHT game environment (Urbanek et al., 2019) , which is a multi-user text-based game, involving many characters playing the game at once. Characters (either played by humans or run by models) can speak to to each other via free text, send emote actions like applaud, nod or pout (22 emote types in total), and take actions to move to different locations and interact with objects (e.g. get cutlery, put cutlery in drawer, etc.), see Appendix A for a full list of game actions.

LIGHT at its core has a game engine which can formally be defined as a graph, where each location, object and character is a node, and they are connected by labeled edges representing relationships, for example contained-in, path-to or has-property. Actions in the game result in changes in state of the graph. To a player (agent) a local view of the graph can be seen and this is expressed in text, as are the game actions and changes of state. This text then naturally interleaves with the dialogue utterances of the speakers as well to form a input context sequence from which a character can base their subsequent actions. See Fig. 1 for an example.

To make the world and its textual descriptions, LIGHT consists of a large set of human-written game locations, characters, and objects, all based within a fantasy medieval setting. Their names, descriptions and properties were crowd-sourced, yielding a total of 663 locations, 1755 characters, and 3462 objects. They range from beaches with crabs and seaweed to crypts with archaeologists and coffins, yielding an extremely rich environment for agents to learn within.

An additional set of crowdworkers were then asked to play the role of characters (randomly picked from the set of 1755) within the created world as rendered by the game engine. This involved them making utterances, game actions and emotes, while interacting with each other (in pairs). The resulting gameplay data consists of 10,777 episodes with an average of 18.3 actions each (game actions, emotes and utterances) of rich human play. These are split into train (8538), validation (500) and test (1739) portions, the latter being split into new episodes in existing settings (test seen, 1000) and completely new settings (test unseen, 739) . This gameplay data can be used for training models using imitation learning, as well as for obtaining "common sense" knowledge about how the world works, i.e., what kinds of things certain characters say; what actions they use with certain objects; what they say and how they act in certain environments or while interacting with certain other characters. The whole environment is thus intended as a proxy for learning about the world within a rich simulation, while avoiding the complexities and bandwidth of rendering (3D) computer graphics. While players were not given specific goals, but instead asked to play the role convincingly of the character given, during play some of them effectively defined their own goals during the interactions, see Fig. 1 .

3. TASK:

The tasks we consider in this work involve interaction between two agents in a given LIGHT scenario. One of the agents, which we will call M env , together with the game engine, effectively functions as an environment for the other agent, which we will call M RL . Because we will formulate our tasks as a reinforcement learning problem, we will also refer to M env as the "environment agent" and M RL as the "RL agent". We assume that the environment agent is fixed; in this work it will be a model trained via behavioral cloning from human-human interaction data. The RL agent must conduct open-ended dialogue such that a given goal action is executed in the future by the environment agent.

Our task is formally defined as follows. The two agents M env and M RL are given their views of the scenario (D env and D RL respectively). These consist of the setting name, scenario description, character names, and their own persona, all described as a sequence of text (see Fig 2) . Note that each agent can only access their own persona but not the persona of the partner with whom they are conversing, but they do know the name of their partner. Denote by t the time-step of the environment, U RL t and U env t the utterances of the agents M RL and M env respectively, and denote by A env t the environment actions by M env . Hence the interaction sequence looks like

S t = [U RL 0 , (U env 0 , A env 0 ), U RL 1 , (U env 1 , A env 1 ), . . . , U RL n , (U env n , A env n )].

Note that there is an inversion from the usual reinforcement literature language, as the "actions" of the RL agent are its utterances U RL t ; the actions A env t of the environment agent should be considered as internal mechanics of the environment. The agent M RL is additionally given a goal g to achieve, which consists of an action which must be executed by the other agent. That is, the objective of M RL is for M env to take the action g. An episode ends when A env t == g or when n becomes larger than a set number of turns. The RL agent only speaks, but does not perform game or emote actions. This was chosen for simplicity, but also to guarantee that the RL agent cannot help force the goal to be reached by performing actions itself -it has to pick the appropriate utterances U RL such that M env eventually takes the action g.

Goals:

We experiment separately with two different types of goals: game actions and emote actions. We use the same train, valid, test (seen and unseen) split of the original human-human LIGHT episodes, assign roles M RL and M env randomly, and randomly pick an action by M env that occurs in the episode as the goal. We can then present the corresponding setting to our agents in order to form a new interaction, but within the same scenario and with a goal that was naturally desirable and achievable within that setting. Observations The state observation O t = (D RL , S t-1 , g) at time t given to an RL model consists of the RL agent's setting description (D RL ), the utterance and action history up to that time step (S t-1 ), and the agent's goal (g). Our RL agent models consume O t as a flattened sequence of tokens, and return a dialogue utterance U RL t . Each structured component is represented in the flattened sequenced separated by a special token denoting the types, e.g. names, settings, etc., see Fig. 2 . Note that because the entire history and goal is given to the RL agent, the environment is Markovian.

Reward We have a terminal reward of +1 only if the goal g is achieved and 0 otherwise, i.e, it is +1 if the environment agent takes the goal action g. The episode ends after n steps. In our experiments we consider n = 1 and n = 3.

4. MODELS:

In this section we describe the models for M env and M RL . In this work these are retrieval models, using the LIGHT dialogue corpus as candidates. We leave generative models to future work.

Base Agent Architecture For all our models we adopt the same base architecture, which is a 12layer bidirectional transformer (Vaswani et al., 2017) pre-trained on a large dialogue corpus (Reddit, 174M examples), and then fine-tuned on our task1 . To score retrieval candidates, we use a biencoder as in (Humeau et al., 2019; Urbanek et al., 2019) . That is, two transformers are used, one to encode the context, and another to encoder a candidate dialogue, and a dot product between the first output vector of each scores the match. To produce a dialogue utterance one then takes the utterance with the largest output from the training set candidates (111k in this case). For emotes and actions, the same procedure is used, but with those candidate sets instead. For actions, the candidates are the set of admissible actions at that game state, which are provided by the game engine, for example get apple is only available in the candidate set if it is a valid action (an apple is present in the room). For emotes, all 22 candidates are always available. To train the model, a cross entropy loss is used. Similar to Mazaré et al. (2018) , during training we consider the other elements of the batch as negatives.

Environment agent:

The environment agent is the base agent described above, and stays fixed during episodes where the RL agent is trained. This helps guarantee that our RL models stick to using the semantics of natural language (English) rather than so-called language drift, of learning a new emergent language with the same tokens (Lee et al., 2019) .

RL agents:

We design two RL approaches for our tasks -learn to pick the right latent discrete variables (topics) that lead to the correct U RL i ; and learn to pick the correct U RL i from the top K candidates. These are described in more detail in Sections 4.2 and 4.3. We also discuss a baseline "inverse" model trained via behavioral cloning on the human-human data.

4.1. INVERSE MODEL:

We consider an inverse model, trained to imitate human actions given a goal, as both a baseline for comparing to RL models, and for producing weights form which we can fine-tune. The inverse model consists of a Bi-encoder, as described above, which takes as input an observation O t similar to our RL models, and outputs an utterance. We train it by extracting from the human-human game logs training set (which does not have goals) every instance where a game action occurs at time t in S t , that is where

S t = [(U RL 1 , A RL 1 ), (U env 1 , A env 1 ), . . . , (U RL t , A RL t ), (U env t , A env t )],

and where A env t is not null (but A RL i for 0 < i ≤ t or A env i for 0 < i < t might be null). We then construct a training example for the inverse model with observation (D RL , g = A env t , S t-1 ). i.e. setting the goal g to be A env t , and with the desired action to be taken by the agent as U RL t . Here we use the subscripts "RL" and "env" just to mark the relative positions in the sequence, as all actions and utterances come from the human logs. Note also that unlike the RL agents we train, the human in the RL agent "position" can take game actions.

We can thus train this model in a supervised manner using a cross entropy loss as described before. This model does not learn a policy interactively, and hence might not learn to plan or strategize optimally for goal completion. The data distribution it is trained on is different than the data distribution seen by the RL agents. Nevertheless, it can serve as a strong baseline. Further, when training our RL agents, we initialize their weights to the weights of this model, and then fine-tune from that point.

4.2. LATENT DISCRETE VARIABLE (TOPIC) MODEL:

Optimizing all the parameters of a large transformer architecture by RL is both incredibly costly in data efficiency and computing time, and is also known to have the problem of language drift (Lee et al., 2019) -that is, there is no guarantee after training with self-chat that the models will output recognizable natural language utterances. A solution to both problems is to train most of the parameters of the model with human-human language data, and then to either disentangle or only optimize some of the parameters with model self-chat (Yarats & Lewis, 2017) .

Here, we propose a straight-forward model for that purpose. We assume an RL agent that consists of two components.

The first component F c (O) = P (T c (O)) maps from an observation to a discrete variable with C possible values. It consists of a chain of two functions: a transformer T s that takes in the observation, and outputs a state representation s, and a policy chooser c = P (s) ∈ (1, . . . , C) which takes in the state representation and outputs the value of the discrete latent variable.

The second component T u (O, c) is an additional transformer that takes as input the observation as well as the output of the first component, and outputs a dialogue utterance. That is, the entire model is the chain u = T u (O, P (T s (O))). We make this explicit decomposition so that we can train only part of the model with RL; note that the "action" trained via RL is choosing c, not outputting the final utterance.

Initial topics:

We first pre-train the transformer T s using the inverse model described in Section 4.1, which produces a vectorial representation of a given observation. We then run K-means over the vectorial representations of all observations from the training set to provide the mapping to one of C values, which represent dialogue topics, which we use as our initial function P (s). These two functions together give us our initialization of F c . Table 1 shows the cluster ID and the topic denoted by that cluster along with the most representative sentences (closest to the center) for that cluster for 50 topics. As we can see, the clusters learnt can be coherent about a topic. We use these 50 topics as a set of actions A for our RL setup.

From c to A Given our initial choice of F c , we can also pre-train T u . We simply take our initial human-human training data, and for each observation append the topic computed by F c to it. This allows our model to be able to generate an action (utterance) conditional on both an input and a topic. We can now train a policy by RL that optimizes the topic at any given point in the episode.

Policy training:

We keep the pre-trained portions of the model T u and T s fixed and during fine-tuning only optimize P . The cluster chooser P is redefined (from the initial K-means) to be an MLP network consisting of 2 layers. A discrete action is sampled from a categorical probability distribution over the possible topics, given by c t ∼ Categorical(h 2 t ), where h 2 t = tanh(W 2 tanh(W 1 s t + b 1 ) + b 2 ). The state vector s t also encodes the goal g and hence, the policy is conditioned on the goal g of the agent. Hence, the policy can learn strategies that will result in picking actions at each time step t that will help the agent to achieve its goal g. As our RL agent can only choose topics, it cannnot redefine easily the meaning of words to cause language drift. 

4.3. TOP-K MODEL:

The Top-K model is another approach to keeping the number of trainable parameters small. It uses the inverse model to get a context embedding v context from the observation, and a list of K candidate utterance embeddings v 1 , ...v K . These are the encodings by the inverse model of the K utterances it considers most likely given the context and goal. We then train a small (2-layer) transformer model that takes as input the set {v context , v 1 , ...v K }. We use the attention above weights of v context against the candidates at the last layer of the transformer as the distribution over the candidates for sampling an utterance. We use K = 50 in the experiments.

4.4. RL TRAINING:

We use the Advantage Actor-Critic implementation (A2C; Kostrikov, 2018) to train the policy and the value function for both the latent-variable and top-K models.

5. RELATED WORK:

Chit-chat dialogue There is an increasing body of work in the domain of chit-chat, where the primary approaches being currently tried are end-to-end neural approaches. They are typically large pre-trained and then fine-tuned transformers, either generative or retrieval, where currently retrieval models work best on a number of tasks (Zhang et al., 2018; Dinan et al., 2018; Li et al., 2019) . Our work shares a commonality with these approaches in that the original LIGHT dialogue data we use has no specified goals, and humans chit-chat together (and act). Thus, the conversations cover a rich number of diverse topics. In Urbanek et al. (2019) models were trained in a similar fashion to chit-chat task models, and we adopt similar architectures here, but instead adapt them to learn to pursue goals.

Goal-oriented dialogue Traditional goal-oriented dialogue has focused on narrow tasks that would typically be useful for a dialogue-based assistant, for example restaurant (Henderson et al., 2014) , taxi, train, and hotel (Budzianowski et al., 2018) or trip (El Asri et al., 2017) booking. Hence, each task typically focuses on a narrow slice of natural language and world knowledge for a specialized domain. Earlier work focused on labeled state representations, slot filling mechanisms and dialogue managers (Rieser & Lemon, 2011) , and more recent work has shifted to an end-to-end approach (Bordes et al., 2017) , in line with chit-chat models, but still the two sets of tasks are rarely considered together, or by using the same methods.

RL for dialogue:

The classical goal-oriented dialogue literature studies RL extensively (Singh et al., 2000) . Typically, they used RL to improve dialogue managers, which manage transitions between dialogue states (Singh et al., 2002; Pietquin et al., 2011; Rieser & Lemon, 2011; Gasic et al., 2013; Fatemi et al., 2016) . Recent works have focused more on end-to-end learning. Some works have focused on self-play type mechanisms for end-to-end reinforcement learning, where the reward is derived from goal. A related approach to ours is the negotation tasks of Lewis et al. (2017) ; Yarats & Lewis (2017) , which require two agents to swap 3 item types (hats, balls, books) where the value of the items is different for the two agents, and derives their personal reward. In contrast, our setup encompasses a rich world of settings and characters -with 3462 object types, and a corresponding large number of actions. This is reflected in the vocabulary size itself (∼32,000 versus ∼2,000 in the negotation tasks). Other notable uses of RL in dialogue include within visual question answering (Das et al., 2017) , in the domain of chit-chat where RL has been used to decrease repetitive and generic responses through the the use of self-play (Li et al., 2016b) , and through human-bot conversation (Sankar & Ravi, 2019) .

RL for language and games RL is used extensively for learning to play games, one of the most well known examples being AlphaGo (Silver et al., 2016) . Since then, language in games has started to be more deeply explored, for example in graphical games such as Minecraft (Oh et al., 2017) , Real-time strategy war games (Hu et al., 2019) , or in text adventure games (Narasimhan et al., 2015; Côté et al., 2018) . The latter are related to our setting. However, those approaches use RL to optimize the set of actions given feedback in a single-player rather than multi-player game, so the text only refers to the environment, and there is no dialogue or actions from other agents. Our work focuses specifically on the latter.

6. EXPERIMENTS:

We compare our various models on the game action and emote action tasks. We experiment with differing number of steps n allowed to complete the goal, n = 1 and n = 3. Our main results for both seen and unseen test environments are given in Table 2 . We report the average reward and for n = 3 the average number of turns before completion. The results show clear improvements for our topic RL ( §4.2) and top-K RL ( §4.3) compared to the inverse model baseline for all values of n, and both types of actions (game actions and emotes).

We show the training curves for topic RL in Fig. 3 , reporting rewards averaged over the batch (512 for n = 1, and 128 for n = 3). They show relatively smooth improvements over time, with clear gains over the baseline. As a sanity check we also tried, after training, to replace the topic RL policy with random topic prediction, which yielded poor results, e.g. 0.217 reward for n = 1 test seen game actions. Our model is clearly learning appropriate topic acts. We show examples of successful utterances, achieving goal actions in Fig. 3 for a diverse range of scenarios, actions and language.

Test Seen Test Unseen 

(n = 1) (n = 3) (n = 1) (n = 3) Model Goal

7. CONCLUSION:

In this paper, we investigate agents that can interact (speak or act) and can achieve goals in a rich world with diverse language, bridging the gap between chit-chat and goal-oriented dialogue. We achieve this by defining a task for an agent, where the goal is for the other player to execute a particular action. We explore two reinforcement learning based approaches to solve this task: the policy either learns to pick a topic or learns to pick an utterance given the top K utterances, and compare them against a strong baseline trained to imitate chit-chat. We show that these approaches effectively learn dialogue strategies that lead to successful completion of goals, while producing natural chat. Future work should explore further RL algorithms for agents that can act and speak in natural language at scale in our proposed rich task environment, and we expect further advancements. 

A GAME ACTIONS WITHIN LIGHT:



:

This arrangement outperforms BERT-based pre-training in dialogue settings, see(Humeau et al.,

2019)

Figures and Tables:

Figure 2: Example interaction in the described task setup (single turn). Here the RL agent M RL would receive a reward as the environment agent M env took the desired action g.

Figure 3: Topic RL model training for n = 1 and n = 3 step action goals for game actions (left) and emotes (right), comparing to the inverse model baseline. Darker lines indicate smoothed plots.

Figure 4: Emote actions within the LIGHT platform from Urbanek et al. (2019)

Then your poor life is a sign from God for you to join us in the church God and serve him!', 'If you say so priest. From now I will pray every night. . . for wealth and good food!', 'Continue to love, worship, and serve Him.' 45 ask favor 'Yes but do you mind doing me a favor?', 'Since I have helped you, could you do me a favor?', 'If I offer to solve your problem, what will. . .

Table content: <table><tr><td colspan="2">#C Topic</td><td>Representative Sentences</td></tr><tr><td>19</td><td colspan="2">animal sounds 'Meow! Purr!', 'Bah-Buk! Tasty!', 'Woof! Very!', 'Bock! Bock!'</td></tr><tr><td>12</td><td>find the cost</td><td>'I would love some fruit. What are your prices?', 'They are beautiful.</td></tr><tr><td/><td/><td>How much do the cost?', 'It flows easily, how much are you selling it for?'</td></tr><tr><td>28</td><td>prayer,</td><td>'</td></tr></table>

Type Reward Reward Turns Reward Reward Turns Results on the test seen and unseen environments for our models.

Table content: <table><tr><td colspan="2">Inverse model game act</td><td>0.223</td><td>0.414</td><td>2.42</td><td>0.193</td><td>0.410</td><td>2.48</td></tr><tr><td>Topic RL</td><td>game act</td><td>0.314</td><td>0.502</td><td>2.25</td><td>0.268</td><td>0.467</td><td>2.33</td></tr><tr><td>Top-K RL</td><td>game act</td><td>0.319</td><td>0.460</td><td>2.37</td><td>0.270</td><td>0.450</td><td>2.39</td></tr><tr><td colspan="2">Inverse model emote</td><td>0.089</td><td>0.262</td><td>2.72</td><td>0.088</td><td>0.266</td><td>2.74</td></tr><tr><td>Topic RL</td><td>emote</td><td>0.170</td><td>0.342</td><td>2.63</td><td>0.146</td><td>0.326</td><td>2.64</td></tr></table>

Just keep taking good care of your beautiful littleThe king keeps the best weapons here.turtle family! Your species is quite unique and I love Take a look -to see you about when I go for a swim. We are the people who live in this town. Persona: I fly high and bring beautiful music to the people. We are common, and there are many... I soar high and low going where the ...

Table content: <table><tr><td>U RL 0</td><td colspan="2">This is the armory!</td><td>U RL 0</td></tr><tr><td>U env 0</td><td colspan="2">Hello, I need to get into the palace to see the</td><td>U env 0</td><td>Well, thank you for that. Do you happen to know</td></tr><tr><td/><td colspan="2">king. I think he might like to see these weapons.</td><td/><td>where my other turtle friend is? You haven't captured</td></tr><tr><td/><td/><td/><td/><td>any turtles have you?</td></tr><tr><td>A env 0</td><td>get weapon</td><td/><td>A env 0</td><td>hug swimmer</td></tr><tr><td colspan="2">Self: townsperson</td><td>Partner: villager</td><td colspan="2">Self: songbird</td><td>Partner: wasp</td></tr><tr><td colspan="3">Persona: Setting: The Lagoon, Lake</td><td colspan="2">Setting: Meadow, Countryside</td></tr><tr><td/><td colspan="2">The Lagoon is a dark and mysterious place</td><td/><td>Large clear outdoor meadow. Flowers of blue and</td></tr><tr><td/><td colspan="2">during the night hours. A lot of moss and lily...</td><td/><td>white appearing in bunches here and there. The ...</td></tr><tr><td>U RL 0</td><td colspan="2">It is cold up here. Would you like my coat</td><td>U RL 0</td><td>Get out of here, wasp!</td></tr><tr><td>U env 0</td><td colspan="3">Oh yes please if I may. My shoe has become sodden U env 0</td><td>You? Fly away from me? You're in my forest, bird.</td></tr><tr><td/><td colspan="2">from running to the market I should love to dry it a bit.</td><td/><td>I control this land earlier.</td></tr><tr><td>A env 0</td><td colspan="2">remove Cloak</td><td>A env 0</td><td>hit a songbird</td></tr></table>

Example 1-step episodes where after the Latent Discrete Variable (Topic) RL agent's utterance U RL 0 the environment agent's response action A env 0 was equal to the RL agent's goal g. Our RL agent both makes natural utterances given the situation, and that elicit the desired goal.

Table content: <table/>

Actor and object2 in same room actor is carrying object1 object1 is gettable object2 is surface or container object2 is carrying object1 put object1 in/on object2 Actor and object2 in same room object2 is carrying object1 object2 is container or surface actor is carrying object1 give object to agent Actor and agent in same room agent is carrying object object is a member of actor steal object from agent actor and agent in same room actor is carrying object object is a member of agent

Table content: <table><tr><td>Action</td><td>Constraints</td><td>Outcome</td></tr><tr><td>get object</td><td>actor and object in same room</td><td>actor is carrying object</td></tr><tr><td/><td>object is gettable</td><td/></tr><tr><td>drop object</td><td>actor is carrying object</td><td>object is in room</td></tr><tr><td/><td>object is gettable</td><td/></tr><tr><td>get object1 from object2</td><td/><td/></tr><tr><td>hit agent</td><td>Actor and agent in same room</td><td>inform agent of attack</td></tr><tr><td>hug agent</td><td>Actor and agent in same room</td><td>inform agent of hug</td></tr><tr><td>drink object</td><td>actor is carrying object</td><td>inform actor of drinking successfully</td></tr><tr><td/><td>object is a drink</td><td/></tr><tr><td>eat object</td><td>actor is carrying object</td><td>inform actor of eating successfully</td></tr><tr><td/><td>object is a food</td><td/></tr><tr><td>wear object</td><td>actor is carrying object</td><td>actor is wearing object</td></tr><tr><td/><td>object is wearable</td><td/></tr><tr><td>wield object</td><td>actor is carrying object</td><td>actor is wielding object</td></tr><tr><td/><td>object is a weapon</td><td/></tr><tr><td>remove object</td><td colspan="2">actor is wearing/wielding object actor is carrying object</td></tr><tr><td/><td>object is wearable or a weapon</td><td/></tr></table>

LIGHT actions and constraints fromUrbanek et al. (2019) B GAME EMOTES WITHIN LIGHT applaud, blush, cry, dance, frown, gasp, grin, groan, growl, laugh, nod, nudge, ponder, pout, scream, shrug, sigh, smile, stare, wave, wink, yawn

Table content: <table/>

Peer review (iclr2020):
This paper explores the goal-oriented dialogue setting with reinforcement learning in a Fantasy Text Adventure Game. To do so, the authors cast the original chit-chat task into a POMDP. In this setting, the observations are the agent*s previous utterances, the other (static) agent*s utterances and actions, and the game description. The actions either consist in selecting a topic (sec 4.2) or to select one utterance over a filtered set of human utterances. Finally, they report their results and observe that the RL approaches outperform the supervised learning models. I have several significant concerns about this paper on several facets, i.e., on the novelty and contribution part, on the algorithm part, and the initial claim. Contribution part: - It is an empirical paper as it assesses how scalable is an approach in a challenging setting. Such papers can be beneficial when they provide insights, extensive experiments, and share the authors* feedback. Unfortunately, those sections are not present in this paper. As a hint, the paper contains 7 pages to explain the method and 1/2 page to list the results and 1/2 page to analyze them, and there are no experimental appendices. Despite unarguable work (the authors perform heavy experiments on a big dataset), there are no simple ablation studies (10, 50, 100 topics? inverse model vs. non-inverse model), no comparison between the two models (e.g. how often the 50 utterances overlap? What are the error made by the models? Are there similar, complementary? Quid of mixture of expert), no qualitative analysis ( errors and/or limit analysis, why actions are harder than emotes?). Besides, I assume that the authors spend some time fine-tuning the network, but there is no hint or take-away about the training difficulties: what can be generalized to other domains? I agree that it is impossible to perform *all* those analysis, but I would have like *some* of them - Again, I acknowledge the considerable engineering work in this paper. However, it seems that most of this codebase contribution was already published in (Urbanek et al 2019). - The paper examines two methods: they either learn to predict the topic or perform topic retrieval. Even if those approaches make sense, they are not novels. For instance, the topic prediction is close to state tracking models where the state would be defined by the topic. There also exists an extensive literature of sentence retrieval, and some relevant works in RL such as (Dulac-Arnold, 2015) Algorithm part: - This paper focuses on RL, but I have a few concern about the soundness of the experiments. It is also surprising that RL training is summarized in two sentences when it is supposed to be the core contribution of the paper. I am particularly concerned with the RL experiments run with n=1. In this scenario, the agent only takes one action before the end of the episode. Thus, there is no planning involved, and applying RL has little interest here. At least, I would have appreciated a note on the impact of exploration. More importantly, the core issue is the following: The reward is defined by 1 in case of success, 0 otherwise. If we apply policy gradient for with n=1 (e.g. A2C), and if we ignore the baseline, for now, the setting is strictly equivalent to applying a cross-entropy loss for every new success while discarding failure cases. In other words, the *RL* training procedure is equivalent to performing data augmentation with a supervised model. In the case of A2C, the loss includes the td-error, an entropy regularizer, and a baseline. Yet, the predicted value function only learns to predict the success ratio as there is no bootstrapping n=1 while computing the td-error. It is thus equivalent to an auxiliary loss. While the baseline makes do penalize negative samples, the underlying reward definition issue remains, and the current setting is more or less equivalent as performing hidden data augmentation with a supervised training loss. Note that this observation does not apply for n=3, and I have no concern regarding those experiments. I am only missing the training hyperparameters. Claim: The paper states that the paper somehow bridges the gap between the chit-chat setting and the goal-oriented setting. However, they cast the problem into a goal-oriented setting by having a reward for specific emote or action (which terminate the episode.) Even if I acknowledge the paper*s motivations, I disagree that the paper provides a new perspective on this issue. Other remarks: Finally, I would have appreciated a sketch of the models, and a table listing the hyperparameters. As the model work as the utterance level, there is no risk of language drift. In the end, I recommend paper rejection for the following reason: - I feel that the paper is lacking sufficient analysis - The RL experiments with n=1 concern me - The novelty is extremely limited over the literature and (Urbanek et al 2019). - there is no training hyperparameters or experimental take-way which limit the reproducibility Dulac-Arnold, Gabriel, et al. *Deep reinforcement learning in large discrete action spaces.* arXiv preprint arXiv:1512.07679 (2015).