Year: 2021

Abstract:
Playing text-based games requires language understanding and sequential decision making. The objective of a reinforcement learning agent is to behave so as to maximise the sum of a suitable scalar reward function. In contrast to current RL methods, humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation. We propose a goal randomization method that leverages human intelligence to use random basic goals to train a policy in the absence of the reward of environments. Specifically, through simple but effective goal generation, our method learns to continuously propose challenging – yet temporal and achievable – goals that allow the agent to learn general skills for acting in a new environment, independent of the task to be solved. In a variety of text-based games, we show that this simple method results in competitive performance for agents. We also show that our method can learn policies that generalize across different text-based games. In further, we demonstrate an interesting result that our method works better than one of state-of-the-art agents GATA, which uses environment rewards for some text-based games.

heading: 1 INTRODUCTION
text: Text-based games are complex, interactive simulations in which the game state is described with text and players act using simple text commands (e.g., take sandwich from table, eat sandwich, open door, etc.) (Côté et al., 2018). They serve as a proxy for studying how agents can exploit language to comprehend and interact with the environment. Text-based games are a useful challenge in the pursuit of intelligent agents that communicate with humans (e.g., in customer service systems). Inspired by this, one of the long term goals in AI is to build agents that can learn to accomplish tasks with language. In the domain of text-based games, the key challenge is to decipher the long textual observations, extract reward cues from them, and generate semantically rich representations such that the policy learned on top of it is well informed. Most of the existing works learn to model text representations during the RL training (Hausknecht et al., 2020; Ammanabrolu & Hausknecht, 2020). Some works also study generalizability of games with different difficulty levels or layouts (Ammanabrolu & Riedl, 2019a; Adolphs & Hofmann, 2020). Deep reinforcement learning (RL) has been demonstrated to effectively learn to solve reward-driven problems in various tasks (Mnih et al., 2013; Silver et al., 2016; Schulman et al., 2017). In contrast, humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation.
Playing text-based games without a reward function is an exceedingly challenging problem. We consider the setting where reward functions are unknown, so we want to learn an agent that can drive itself without environmental rewards. Learning agents without reward has several practical applications (Eysenbach et al., 2018). Environments with sparse rewards effectively have no reward until the agent randomly reaches a goal state. Learning intelligent agents without supervision may help address challenges in exploration in these environments (Gupta et al., 2018; Sharma et al., 2019). In many practical settings, interacting with the environment is essentially free, but evaluating the reward requires human feedback (Christiano et al., 2017). However, there is no work on playing text-based games without reward functions. Solving such kinds of non-reward tasks can encourage agents to explore, experiment, and invent. Sometimes, as in many games and fantasies, without any direct link to reality or to any source of extrinsic reward, it is crucial to enable learning in real-world environments, even for humans (Schulz, 2012).
In this paper, we take a step towards agents that can learn from text-based games without reward functions. The environments are designed to mimic some real-world scenarios where there are no reward cues for guiding agents. The goal is for an agent to be able to learn one policy that is able to solve both tasks it was trained on as well as a variety of unseen tasks which contain similar tasks as the training tasks. To do this, we propose a new method for learning an agent with deep RL in the absence of any rewards. We use a set of available goals characterized by natural language via common-sense rules, and select one of them according to the current knowledge graph based observation. Then, we learn policies for goal-conditioned reinforcement learning. Specifically, our method works as follows. Whenever an agent builds a knowledge graph of the textural world, our method gives a goal in natural language to the agent based on the knowledge graph. The agent takes the goal to form a new experience with a corresponding intrinsic reward, alleviating the no reward problem. For example, our method can describe what the agent has achieved in the episode, and the agent can use goals as advice to obtain intrinsic rewards. In addition, our method also provides a time limit for a goal. If the agent can not accomplish the goal in the time limit, it can use a new goal replacing the old one. The agent can have the opportunity to get out of the difficult goals. We show many benefits brought by language goal representation when combined with goal advice. The agent can efficiently solve reinforcement learning problems in challenging text-based environments; it can generalize to unseen instructions, and even generalize to instruction with unseen lexicons.
Our method can be viewed as the intrinsic motivation of any agent trained with policy gradientbased methods (Oudeyer & Kaplan, 2009; Barto, 2013). Most intrinsic methods design intrinsic motivations to assist environmental rewards to make agents learn efficiently. However, our method is to use the intrinsic motivation to play text-based games without environmental rewards. Under this view, we also need to encode the intrinsic motivation into the policy. That is, the original policy network becomes a goal-conditional policy; the goal advice can then be seen as a “bolt-on” to the original policy network. Because we use natural language for self-advice. It is flexible and can be used on a variety of RL training model architectures and training settings by encoding the intrinsic motivation.
In summary, we make the following contributions: (i) we first study the problem of playing textbased games without any reward functions and propose a new goal randomization method for solving the problem;1 (ii) we show, through common-sense knowledge, that agents trained with goal randomization gradually learn to interact with the environment and solve tasks which are difficult for state-of-the-art methods; (iii) we perform an extensive qualitative analysis and ablation study, and we also find an interesting result that our method works better than one of state-of-the-art agents GATA (Adhikari et al., 2020), which uses environment rewards, for some text-based games.

heading: 2 RELATED WORK
text: Reinforcement learning for text-based games. Existing agents either perform based on predefined rules or learn to make responses by interacting with the environment. Rule-based agents (Atkinson et al., 2019; Fulda et al., 2017; Hausknecht et al., 2019; Kostka et al., 2017) attempt to solve text-based games by injecting heuristics. They are thus not flexible since a huge amount of prior knowledge is required to design rules (Hausknecht et al., 2020). Learning-based agents (Adolphs & Hofmann, 2020; Hausknecht et al., 2020; He et al., 2016; Jain et al., 2020; Narasimhan et al., 2015; Yin & May, 2019; Yuan et al., 2018; Zahavy et al., 2018) usually employ deep reinforcement learning algorithms to deliver adaptive game solving strategies. KG-based agents have been developed to enhance the performance of learning-based agents with the assistance of KGs. KGs can be constructed by simple rules so that it substantially reduces the amount of prior knowledge required by rule-based agents. While KGs have been leveraged to handle partial observability (Ammanabrolu & Hausknecht, 2020; Ammanabrolu & Riedl, 2019a; Zelinka et al., 2019), reduce action space (Ammanabrolu & Hausknecht, 2020; Ammanabrolu & Riedl, 2019a), and improve generalizability (Adhikari et al., 2020; Ammanabrolu & Riedl, 2019b). Recently, Murugesan et al. (2020) tried to introduce commonsense reasoning for playing synthetic games. While these works all follow the standard setting with reward functions, our work is the first work that trains an agent without reward functions.
1Code can be found here: https://anonymous.4open.science/r/goalrand-E167/
Intrinsic motivation for reinforcement learning. Intrinsic motivation has been widely used in reinforcement learning (Oudeyer et al., 2007; Oudeyer & Kaplan, 2009; Barto, 2013). It has been proven effective for solving various hard-exploration tasks (Bellemare et al., 2016; Pathak et al., 2017; Burda et al., 2018b). One prominent formulation is the use of novelty, which in its simplest form can be estimated with state visitation counts (Strehl & Littman, 2008) and has been extended to high-dimensional state spaces (Bellemare et al., 2016; Burda et al., 2018b; Ostrovski et al., 2017). Other sophisticated versions of curiosity (Schmidhuber, 1991) guide the agent to learn about environment dynamics by encouraging it to take actions that reduce the agent’s uncertainty (Stadie et al., 2015; Burda et al., 2018b), have unpredictable consequences (Pathak et al., 2017; Burda et al., 2018a), or a large impact on the environment (Raileanu & Rocktäschel, 2020). Other forms of intrinsic motivation include empowerment (Klyubin et al., 2005) which encourages control of the environment by the agent, and goal diversity (Pong et al., 2019) which encourages maximizing the entropy of the goal distribution. Intrinsic goals are discovered from language supervision (Lair et al., 2019). Except for exploration, intrinsic motivation is also used for other problems, such as evolutionary (Singh et al., 2009; Sorg et al., 2010; Zheng et al., 2018). Most works use intrinsic motivation as additive rewards for environmental rewards. Our work differs from those works by formulating intrinsic motivation to train an agent directly for text-based games.
Generalization in text-based games. Generalization is a challenging problem for reinforcement learning (Tobin et al., 2017; Agarwal et al., 2019). In text-based games, it is difficult to study generalization in games initially designed for human players (Hausknecht et al., 2020), as they are so challenging that existing RL agents are still far from being able to solve a large proportion of them even under the single game setting (Yao et al., 2020). Furthermore, these games usually have different themes, vocabularies and logics, making it hard to determine the domain gap (Ammanabrolu & Riedl, 2019b). Compared with these man-made games, the synthetic games (Côté et al., 2018; Urbanek et al., 2019) provide a more natural way to study generalization by generating multiple similar games with customizable domain gaps (e.g., by varying game layouts). Generally, the training and testing game sets in previous works have either the same difficulty level (Ammanabrolu & Riedl, 2019a; Murugesan et al., 2021), or a mixture of multiple levels (Adolphs & Hofmann, 2020; Yin et al., 2020), or both (Adhikari et al., 2020). Our works can be used for generalization in games. Moreover, our work does not use the reward functions of environments to train an agent.

heading: 3 PRELIMINARIES
text: Text-based games as POMDPs. Text-based games can be formally described as Partially Observable Markov Decision Processes (POMDPs). POMDPs can be defined as a tuple 〈S,A,T ,R,Ω,O,γ〉: the state set S, the action set A, the state transition probabilities T , the reward functionR, the observation set Ω, the conditional observation probabilitiesO and the discount factor γ ∈ (0, 1]. At each time step, the agent will receive a textual observation ot ∈ Ω, depending on the current state and previous action via the conditional observation probability O(ot|st,at−1). By executing an action at ∈ A, the environment will transit into a new state based on the state transition probability T (st+1|st,at), and the agent will receive the reward rt+1 = R(st,at). Same as Markov Decision Process (MDPs), the goal of the agent is to learn an optimal policy π∗ to maximize the expected future discounted sum of rewards from each time step: Rt = E[ ∑∞ k=0 γ krt+k+1].
Knowledge graph for text-based games. Graph-based representations are effective for text-based games because the state in these games adheres to a graph-like structure. The content in most observations of the environment corresponds either to entity attributes or to relational information about entities in the environment. Knowledge graph (KG) for a text-based game can be built from a set of triplets 〈Subject , Relation , Object〉, denoting that the Subject has Relation with the Object . For example, 〈Kitchen , Has , Food〉. The KG is denoted as G = (V,E), where V and E are the node set and the edge set, respectively. Both Subject and Object belong to the node set V . Relation , which corresponds to the edge connecting them, belongs to E.

heading: 4 METHODOLOGY

heading: 4.1 PROBLEM STATEMENT
text: We aim to design an RL-based agent that is able to solve text-based games without reward functions. We focus on man-made text-based games, which are initially designed for human play-
ers (Hausknecht et al., 2020). These games are devised with more complex logic and much larger action space than synthetic games (Côté et al., 2018). Most of them share similar themes and vocabularies. There are no obvious goals for accomplishing games. There are also no rewards provided by the text-based games or environments. At every step we construct an input st as the combination of three components: a textual observation ot,text and a KG ot,KG (note that here st should not be regarded as a true game state as the games are not fully observable). ot,text further includes the current state ot,desc (describing the environment), inventory ot,inv (describing items collected by a player), game feedback ot,feed, and previous action taken at−1. While ot,text mainly reflects the current observation, ot,KG records the game history. Therefore, the KG can help the agent to handle partial observability. At each time step, the triples extracted from the current textual observation ot,text are used to update the KG as ot,KG = GraphUpdate(ot−1,KG,ot,text).
We consider two scenarios for text-based games: seen levels, where the training and testing games have the same levels, but different layouts, and unseen levels, where the training and testing games are different in levels and layouts. Many games share similar themes and vocabularies, but vary in their layouts and / or difficulty levels. For example, games of the cooking theme (Côté et al., 2018) share the same overall objective: prepare the meal. The layout of a game contains the room connectivity and the preparing steps (e.g., the type / location of ingredients). The difficulty of a game depends on the complexity of the map (e.g., the number of rooms) and the recipe (e.g., the number of ingredients), such that two games with different levels are naturally different in their layouts. More examples are provided in Appendix A.

heading: 4.2 GOAL RANDOMIZATION
text: Text-based games use texts for describing what the player or agent needs to do to win the game. The goal of the agent in the game is to complete some objective. Such textual information hints about the objective/purpose of the game, what dangers are present, and provides clues that might become handy later in the game for solving puzzles. However, without rewards, it becomes difficult for the agent to learn to win a game and generalize to other environments. The agent must create intrinsic motivation by itself to learn new skills in the textual world.
The purpose of goal randomization is to provide enough motivation variability at training time such that at test time the agent is able to generalize to other environments. Goal randomization provides a way to guide the agent to learn essential skills. With goal randomization, the agent can obtain experiences and skills by accomplishing tasks with random goals. While a whole game may be difficult to accomplish due to long-term temporal dependency, decomposing it into a sketch of essential skills or subtasks will make the game easier to be solved (Sohn et al., 2018; Shiarlis et al., 2018). If we consider the solving strategy for a subtask as a skill, the generalizability for an unseen game will also be improved by recomposing the learnt skills. We can characterize a subtask by a very basic goal. In text-based games, we make the goal to be instruction-like textual descriptions (e.g., “find purple potato”), yielding better flexibility and interpretability than using a state as the goal (Schaul et al., 2015; Andrychowicz et al., 2017; Plappert et al., 2018).
Figure 1 shows the overview of our method, which consists of a goal set generator and a goalconditional policy. We denote the set containing all required goals for solving a game as G. Inspired by (Jiang et al., 2019), the goal set generator can be implemented by different approaches, including supervised language models and non-learning-based methods such as human supervisors and functional programs. In our work, simply we use common-sense rules to obtain Gt, defined as follows:
• (I) if an ingredient i is not collected in the knowledge graph, then the goal is “find i”;
• (II) if an ingredient i is collected and has a requirement q in the knowledge graph, then the goal is “q i”.
In the real world, the goal of some task can be complex and depend on other goals. However, these generated goals are simple and basic for text-based games. On one hand, these goals are common and shareable for different tasks and environments. On the other hand, these goals can be easily described using a simple phrase or sentence. For example, “find an object”, “prepare something” or “eat something”. The agent can learn some essential skills for finishing these tasks with basic and simple goals. Common-sense rules or human instruction can provide a lot of such basic and simple
fundamental goals and make goals human-interpretable and consistent with existing games: e.g., food items can be combined, cooked, and eaten. More details are in Appendix B.
We train the agent by generating random goals during games. We randomize the following aspects of the goals for the agent used during training: different types of goals and different objects. Language can provide an abstract representation for a goal. This motivates our proposal to use natural language for representing the goal space. Since we use language to represent goals, we can use natural language to represent different goals easily. Concretely, we can define a lot of goals according to common-sense rules and knowledge. They are natural language descriptions of the goals. Then we have a goal set G. With the goal set, we introduce a new objective for the agents:
θ∗ = arg min θ L(πθ; g), g ∼ G. (1)
We assume that the goal is also time-awareness. A goal is bounded by a time limit [0, ξ]. Thus there is a time limit to reach a goal. If a goal is not finished in its time limit, we sample a new goal to replace the old goal. It means during an episode, there may be more than one goal. One desirable property is goal diversity (Pong et al., 2019; Raileanu & Rocktäschel, 2020; Campero et al., 2020). In our implementation, we use uniform sampling that encourages such diversity. This sampling strategy, along with the time limits of the goals, helps the agent avoid getting stuck in local minima.
Without any reward functions of environments, we design the following pseudo-reward:
r(st, at; gi) = fgi(st+1),where fgi : S → {0, 1}. (2)
The pseudo-reward is general for different tasks and environments because it just involves essential goals. During goal randomization, we sequentially sample goals for each episode. Each goal is delivered as a subtask for the agent to act throughout the episode. In each subtask that is timelimited, the learning objective is to optimize the expectation of the return Gt:t+∆t.

heading: 4.3 MODEL ARCHITECTURE AND TRAINING
text: We use the goal-oriented reinforcement learning framework following (Schaul et al., 2015). We augment the previously defined infinite-horizon, discounted POMDPs with a goal space G. A goal g is chosen from G. The difference of our model is that g varies for each episode. The goal induces a reward function rg : S × A → R, that assigns reward to a state conditioned on a given goal. The policy π follows the goal-conditioned RL setting (Kaelbling, 1993), where at is selected by considering both oKGt and g. Figure 1 shows the architecture of π, which is constructed based on both oKGt and g. π has the graph encoder and text encoder to process game information.
Algorithm 1 Goal Randomization (GoalRand) Require: RL algorithm π, goal set G, replay buffer B
1: Initialize π 2: for episode = 1, 2, · · · ,M do 3: Sample an initial goal g from G 4: Set the time limit of g: ∆t = 0 5: for t = 0, · · · , T − 1 do 6: if ∆t = ξ or g is done then 7: Sample a new goal g from G 8: Set the time limit of g: ∆t = 0 9: end if
10: Sample action at ∼ π 11: Execute the action at and observe a new state st+1 12: Set the pseudo-reward reward rt according to (2) 13: Store the translation to B 14: ∆t = ∆t+ 1 15: end for 16: for t = 1, · · · , N do 17: Sample a minibatch B from the replay buffer B 18: Optimize π using the minibatch B 19: end for 20: end for
To integrate language representation of goals into the model, we design the architecture as follows. The architecture consists of three modules. Similar to previous works (Adhikari et al., 2020), we use a graph encoder to encode oKGt as state representation st, and a text encoder to encode Gt as a stack of goal representations. Arbitrary graph encoders and text encoders can be applied. We use the graph encoder based on the Relational Graph Convolutional Networks (R-GCNs) (Schlichtkrull et al., 2018) to take both nodes and edges into consideration. For the text encoder, a simple singleblock transformer (Vaswani et al., 2017) is sufficient as the goal candidates are short texts. As this work does not aim at handling the combinatorial action space, we consider the admissible action set At ⊆ A for each time step (He et al., 2016). We denote an action as “admissible” if it does not lead to meaningless feedback (e.g., “Nothing happens”). The action scorer will pair st with each candidate ai ∈ At, followed by linear layers to compute the action scores. We name our method as GoalRand and detailed procedures are given in Algorithm 1. The goal set updates during an episode. Experiences (or transitions) from the agent are collected at every time step. In an episode, each goal g is randomly drawn and updated if the time limit of the goal terminates. For the agent, at every time step, an action is drawn based on π with a goal. The model parameters θ are periodically updated by drawing experiences from replay memories.

heading: 5 EXPERIMENTS

heading: 5.1 EXPERIMENTAL SETUP
text: We conduct experiments on multiple levels of cooking games (Côté et al., 2018). We consider two scenarios for the experiments: (1) seen levels, where the training and testing games have the same levels, but different layouts. (2) unseen levels, where the training and testing games are different in levels and layouts. While previous work (Adhikari et al., 2020) considered either a single level, or a mixture of 4 levels, we extend their setting to 8 levels. Based on the rl.0.1 game set2, we build a training game set Dtrain with 4 levels, including 100 games per level. We build a validating game set Dval with the same 4 levels ofDtrain, where each level contains 20 games. We build two testing game sets: Dseentest , and Dunseentest , both of which contain 4 levels and 20 games per level. The levels within Dseentest have been seen inDtrain andDval, while there is no overlapping game. The levels withinDunseentest are unseen during training. Table 1 shows the game statistics averaged over each level, where “S#” denotes a seen level and “US#” denotes an unseen level. We set the step limit of an episode as 50 for
2https://aka.ms/twkg/rl.0.1.zip
training and 100 for validation / testing. We train the models for 60,000 episodes. For every 1,000 episodes, we validate the model on Dval, and report the testing performance on Dseentest and Dunseentest .

heading: 5.2 BASELINES
text: As far as we know, there is no prior work on playing text-based games without reward functions. To evaluate the performance of our algorithm, we consider constructing two baselines. One baseline is a random agent and takes actions randomly. The other baseline is a reward-free version of BeBold (Zhang et al., 2020). The intrinsic rewards are often used as additive rewards for environmental rewards. For a fair comparison, all methods do not use environmental rewards. We evaluated the following methods:
• Random: this is an agent that takes actions randomly. At each time step, the action to be executed is uniformly sampled from available.
• BeBold (Zhang et al., 2020): it is one of the state-of-the-art methods using intrinsic rewards. It provides count-based reward rcountt as intrinsic reward. It counts the visitation of observations within an episode, and the accumulated visitation throughout the training process, defined as:
rcountt+1 = max( 1 Nacc(o KG t ) − 1 Nacc(o KG t+1) , 0) · I{Nepi(oKGt+1) = 1}, (3)
where Nacc and Nepi denote the accumulated and episodic visitation count, respectively. The I operation returns 1 if oKGt+1 is visited for the first time in the current episode, otherwise 0.
• GoalRand (ours): this is our method that performs training with random basic goals. We use ξ = 5 as the goal time limit. Without receiving true reward signals from environments, we generate pseudo-reward for training agents to solve different tasks.

heading: 5.3 MAIN RESULTS
text: Table 2 shows the testing performance at the end of training, and Figure 2 shows testing performance with respect to the training episodes. We use the normalized score, which is defined as the collected score normalized by the maximum available score for this game (“MaxScore” in Table 1), to measure the performance. The proposed GoalRand outperforms the baselines in both seen and unseen levels.
It solves most of the simple levels (“S1”, “S2”) and completes 50% of the hard levels (“S3”, “S4”). In the unseen levels, where both the layouts and complexities are different from those training games, our method also achieves good performance. We also observed that only encouraging exploration is not sufficient when the environment reward is not available: the BeBold baseline learns very slow, and performs even worse than the non-learnable random agent.

heading: 5.4 ABLATION STUDY
text: We then conduct ablation studies to further investigate the contributions of GoalRand’s components. We consider following variants:
• GoalRand w/o Threshold: our agent without the goal time limit. That is, the agent will keep being conditioned a same goal until this goal is not available.
• GATA (Adhikari et al., 2020): we adopt the GATA agent, which utilizes the environment reward for learning.
Table 3 shows the testing performance at the end of training, and Figure 3 shows testing performance with respect to the training episodes. The results show that the goal time limit can contribute to the performance, especially in more difficult levels such as “S4” and “US4” (“GoalRand” v.s., “GoalRand w/o Threshold”). In these levels, there are more available goals at each time step, the agent will end up failing a game if it selects a too difficult goal and gets stuck in solving it. Setting a time limit for the goal helps the agent to prevent such condition, as it can choose another goal if the current goal is too hard to solve. In the simpler levels such as “S1” and “S2”, there may be fewer available goals, therefore re-sampling a new goal might be less effective. Although the GATA agent is provided with the environment reward, it shows worse performance in comparison to our agent. One possible reason is that the agent may have difficulties in solving a whole game, which is with long term dependency. Our goal randomization method helps the agent to effectively explore the environment, and simplify a complex task as a set of easier-to-solve sub-goals.

heading: 6 CONCLUSION
text: In this paper, we present goal randomization, a method for playing text-based games without a reward function. We show that the proposed goal randomization method via common-sense rules can learn skills for complex tasks, often solving benchmark tasks with the learned skills without actually receiving any environment rewards. The experiments demonstrate that our proposed method can improve the generalization of the agent and works well for the games of the seen and unseen levels. Interestingly, for some text-based games, the experiments show that the proposed method is
better than one of state-of-the-art method GATA, which uses environment rewards, demonstrating the superiority of our method.

heading: A EXAMPLES OF TEXT-BASED GAMES
text: Figure 4 visualizes the initial observation oKG0 for four games, where “S1 Game1” and “S1 Game2” belong to level “S1”, “S2 Game1” and “S2 Game2” belong to level “S2”. Figure 5 visualizes the initial observation of two games belonging to level “S3”. Figure 6 visualizes the initial observation of one game belonging to level “S4”. Games within the same level have the same complexity, but are different in their layouts. For example, “S2 Game1” and “S2 Game2” have the same number of rooms, number of ingredients and number of requirements, but the ingredient and requirement are different. Similarly, “S3 Game1” and “S3 Game2” have the same number of rooms, but are with different room connectivity. Games within different levels have different complexities, and their layouts are naturally different (e.g., “S1 Game1” v.s., “S2 Game1” v.s., “S3 Game1” v.s., “S4 Game1”). The TextWorld also provides other game themes, such as the default House theme, the Coin Collector theme and the Treasure Hunter theme. In these themes, the agent needs to go through the rooms to find either the coin, or an object specified at the beginning of an episode. We believe that the Cooking theme we use in this work is able to cover these themes, as it requires the agent to navigate through different numbers of rooms, collect different numbers of ingredients, and prepare them in different ways.

heading: S2 Game1 S2 Game2

heading: S1 Game1 S1 Game2

heading: S3 Game1

heading: S3 Game2

heading: B GOAL SET GENERATOR
text: We design a simple non-learning-based goal set generator by exploiting the KG-based observation in the cooking theme. Algorithm 2 shows the pipeline for obtaining the goal set Gt. We first obtain the ingredient set I. For each ingredient i ∈ I, we first check whether it has been collected, then obtain its status set Si and requirement set Ri. We consider three types of goals: 1) “find” requires the agent to find and collect an uncollected ingredient, 2) “prepare” requires the agent to prepare an ingredient to satisfy a requirement, and 3) “eat”, that the agent is required to prepare and eat the final meal. Algorithm 3 shows the pipeline for assigning the goal-conditioned reward rgoalt . We first obtain the type of a goal g, then check whether this goal has been accomplished given at and oKGt+1. Some functions in Algorithm 2 can be reused here. r goal t is a binary reward that we will assign

heading: S4 Game1
text: rgoalt = rmax if g is accomplished successfully, otherwise rmin (still not finished, or failed). Algorithm 2 and Algorithm 3 can also be implemented via learning-based methods. For example, the functions can be achieved by a QA model by answering specific questions.
Algorithm 2 Goal set generation Input: Knowledge graph oKGt Initialize: Goal set Gt ← ∅ 1: Get ingredient set I ← GetIng(oKGt ) 2: for each ingredient i ∈ I do 3: if CheckCollection(i, oKGt ) = False then 4: Add goal “find i” to Gt 5: else 6: Get status set Si ← GetStatus(i, oKGt ) 7: Get requirement setRi ← GetReq(i, oKGt ) 8: for each requirement ri ∈ Ri do 9: if ri /∈ Si then 10: Add goal “ri i” to Gt 11: end if 12: end for 13: end if 14: end for 15: if Gt = ∅ then 16: Add goal “prepare and eat meal” to Gt 17: end if 18: return Gt.
Algorithm 3 Goal-conditioned reward acquisition Input: Knowledge graph oKGt+1, goal g, rewards {rmin, rmax} Initialize: FLAG← False 1: Obtain goal type gtype ← GetGoalType(g) 2: if gtype = “find” then 3: Get ingredient i from g 4: if CheckCollection(i, oKGt+1) = True then 5: FLAG← True 6: end if 7: else if gtype = “prepare” then 8: Get ingredient i, requirement r from g 9: Get status set Si ← GetStatus(i, oKGt+1) 10: if r ∈ Si then 11: FLAG← True 12: end if 13: else 14: if CheckExistance(“meal”, oKGt+1) = True then 15: FLAG← True 16: end if 17: end if 18: if FLAG = True then 19: return rmax. 20: else 21: return rmin. 22: end if

Review summaries:
This paper introduces GoalRand, an algorithm for playing text-based games in the absence of extrinsic reward. In particular, it uses generated goals to train a goal-conditioned reinforcement learning agent. These goals are generated by extracting factual information from the agent's knowledge graph. Given a goal sampled from the set of possible goals, as defined by the current knowledge graph, the agent is given a intrinsic reward for completing this goal within a specified time limit of 5 steps. Goals are randomly sampled from the set of possible goals.   The paper contributes this goal generation methodology which allows the agent to be trained without access to extrinsic rewards. Additionally, experiments are conducted on Textworld's Cooking Games, comparing against an intrinsic motivation baseline (BeBold) as well as a previously high-performing agent on these games, GATA. The scores show higher performance from GoalRand compared to the baselines across a set of heldout layouts and levels.

The authors propose a simple intrinsic reward for text-based games, in the absence of environment rewards. The method, GoalRand, is based on uniformly sampling random goals from a set of natural language goals generated using common-sense rules. The authors evaluate their method on both seen and unseen text-based games and find that their method outperforms a random agent and a count-based intrinsically motivated agent.

This paper studies text-based games using a technique called goal randomization. This method uses random basic goals to train a policy in the absence of the reward of environment. Authors show that agents can learn policies that generalize well across different text-based games. It also seems working better than a state-of-the-art algorithm that uses environment reward.