Goal Randomization for Playing Text-based Games without a Reward Function

1 INTRODUCTION . Text-based games are complex , interactive simulations in which the game state is described with text and players act using simple text commands ( e.g. , take sandwich from table , eat sandwich , open door , etc . ) ( Côté et al. , 2018 ) . They serve as a proxy for studying how agents can exploit language to comprehend and interact with the environment . Text-based games are a useful challenge in the pursuit of intelligent agents that communicate with humans ( e.g. , in customer service systems ) . Inspired by this , one of the long term goals in AI is to build agents that can learn to accomplish tasks with language . In the domain of text-based games , the key challenge is to decipher the long textual observations , extract reward cues from them , and generate semantically rich representations such that the policy learned on top of it is well informed . Most of the existing works learn to model text representations during the RL training ( Hausknecht et al. , 2020 ; Ammanabrolu & Hausknecht , 2020 ) . Some works also study generalizability of games with different difficulty levels or layouts ( Ammanabrolu & Riedl , 2019a ; Adolphs & Hofmann , 2020 ) . Deep reinforcement learning ( RL ) has been demonstrated to effectively learn to solve reward-driven problems in various tasks ( Mnih et al. , 2013 ; Silver et al. , 2016 ; Schulman et al. , 2017 ) . In contrast , humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation . Playing text-based games without a reward function is an exceedingly challenging problem . We consider the setting where reward functions are unknown , so we want to learn an agent that can drive itself without environmental rewards . Learning agents without reward has several practical applications ( Eysenbach et al. , 2018 ) . Environments with sparse rewards effectively have no reward until the agent randomly reaches a goal state . Learning intelligent agents without supervision may help address challenges in exploration in these environments ( Gupta et al. , 2018 ; Sharma et al. , 2019 ) . In many practical settings , interacting with the environment is essentially free , but evaluating the reward requires human feedback ( Christiano et al. , 2017 ) . However , there is no work on playing text-based games without reward functions . Solving such kinds of non-reward tasks can encourage agents to explore , experiment , and invent . Sometimes , as in many games and fantasies , without any direct link to reality or to any source of extrinsic reward , it is crucial to enable learning in real-world environments , even for humans ( Schulz , 2012 ) . In this paper , we take a step towards agents that can learn from text-based games without reward functions . The environments are designed to mimic some real-world scenarios where there are no reward cues for guiding agents . The goal is for an agent to be able to learn one policy that is able to solve both tasks it was trained on as well as a variety of unseen tasks which contain similar tasks as the training tasks . To do this , we propose a new method for learning an agent with deep RL in the absence of any rewards . We use a set of available goals characterized by natural language via common-sense rules , and select one of them according to the current knowledge graph based observation . Then , we learn policies for goal-conditioned reinforcement learning . Specifically , our method works as follows . Whenever an agent builds a knowledge graph of the textural world , our method gives a goal in natural language to the agent based on the knowledge graph . The agent takes the goal to form a new experience with a corresponding intrinsic reward , alleviating the no reward problem . For example , our method can describe what the agent has achieved in the episode , and the agent can use goals as advice to obtain intrinsic rewards . In addition , our method also provides a time limit for a goal . If the agent can not accomplish the goal in the time limit , it can use a new goal replacing the old one . The agent can have the opportunity to get out of the difficult goals . We show many benefits brought by language goal representation when combined with goal advice . The agent can efficiently solve reinforcement learning problems in challenging text-based environments ; it can generalize to unseen instructions , and even generalize to instruction with unseen lexicons . Our method can be viewed as the intrinsic motivation of any agent trained with policy gradientbased methods ( Oudeyer & Kaplan , 2009 ; Barto , 2013 ) . Most intrinsic methods design intrinsic motivations to assist environmental rewards to make agents learn efficiently . However , our method is to use the intrinsic motivation to play text-based games without environmental rewards . Under this view , we also need to encode the intrinsic motivation into the policy . That is , the original policy network becomes a goal-conditional policy ; the goal advice can then be seen as a “ bolt-on ” to the original policy network . Because we use natural language for self-advice . It is flexible and can be used on a variety of RL training model architectures and training settings by encoding the intrinsic motivation . In summary , we make the following contributions : ( i ) we first study the problem of playing textbased games without any reward functions and propose a new goal randomization method for solving the problem ; 1 ( ii ) we show , through common-sense knowledge , that agents trained with goal randomization gradually learn to interact with the environment and solve tasks which are difficult for state-of-the-art methods ; ( iii ) we perform an extensive qualitative analysis and ablation study , and we also find an interesting result that our method works better than one of state-of-the-art agents GATA ( Adhikari et al. , 2020 ) , which uses environment rewards , for some text-based games . 2 RELATED WORK . Reinforcement learning for text-based games . Existing agents either perform based on predefined rules or learn to make responses by interacting with the environment . Rule-based agents ( Atkinson et al. , 2019 ; Fulda et al. , 2017 ; Hausknecht et al. , 2019 ; Kostka et al. , 2017 ) attempt to solve text-based games by injecting heuristics . They are thus not flexible since a huge amount of prior knowledge is required to design rules ( Hausknecht et al. , 2020 ) . Learning-based agents ( Adolphs & Hofmann , 2020 ; Hausknecht et al. , 2020 ; He et al. , 2016 ; Jain et al. , 2020 ; Narasimhan et al. , 2015 ; Yin & May , 2019 ; Yuan et al. , 2018 ; Zahavy et al. , 2018 ) usually employ deep reinforcement learning algorithms to deliver adaptive game solving strategies . KG-based agents have been developed to enhance the performance of learning-based agents with the assistance of KGs . KGs can be constructed by simple rules so that it substantially reduces the amount of prior knowledge required by rule-based agents . While KGs have been leveraged to handle partial observability ( Ammanabrolu & Hausknecht , 2020 ; Ammanabrolu & Riedl , 2019a ; Zelinka et al. , 2019 ) , reduce action space ( Ammanabrolu & Hausknecht , 2020 ; Ammanabrolu & Riedl , 2019a ) , and improve generalizability ( Adhikari et al. , 2020 ; Ammanabrolu & Riedl , 2019b ) . Recently , Murugesan et al . ( 2020 ) tried to introduce commonsense reasoning for playing synthetic games . While these works all follow the standard setting with reward functions , our work is the first work that trains an agent without reward functions . 1Code can be found here : https : //anonymous.4open.science/r/goalrand-E167/ Intrinsic motivation for reinforcement learning . Intrinsic motivation has been widely used in reinforcement learning ( Oudeyer et al. , 2007 ; Oudeyer & Kaplan , 2009 ; Barto , 2013 ) . It has been proven effective for solving various hard-exploration tasks ( Bellemare et al. , 2016 ; Pathak et al. , 2017 ; Burda et al. , 2018b ) . One prominent formulation is the use of novelty , which in its simplest form can be estimated with state visitation counts ( Strehl & Littman , 2008 ) and has been extended to high-dimensional state spaces ( Bellemare et al. , 2016 ; Burda et al. , 2018b ; Ostrovski et al. , 2017 ) . Other sophisticated versions of curiosity ( Schmidhuber , 1991 ) guide the agent to learn about environment dynamics by encouraging it to take actions that reduce the agent ’ s uncertainty ( Stadie et al. , 2015 ; Burda et al. , 2018b ) , have unpredictable consequences ( Pathak et al. , 2017 ; Burda et al. , 2018a ) , or a large impact on the environment ( Raileanu & Rocktäschel , 2020 ) . Other forms of intrinsic motivation include empowerment ( Klyubin et al. , 2005 ) which encourages control of the environment by the agent , and goal diversity ( Pong et al. , 2019 ) which encourages maximizing the entropy of the goal distribution . Intrinsic goals are discovered from language supervision ( Lair et al. , 2019 ) . Except for exploration , intrinsic motivation is also used for other problems , such as evolutionary ( Singh et al. , 2009 ; Sorg et al. , 2010 ; Zheng et al. , 2018 ) . Most works use intrinsic motivation as additive rewards for environmental rewards . Our work differs from those works by formulating intrinsic motivation to train an agent directly for text-based games . Generalization in text-based games . Generalization is a challenging problem for reinforcement learning ( Tobin et al. , 2017 ; Agarwal et al. , 2019 ) . In text-based games , it is difficult to study generalization in games initially designed for human players ( Hausknecht et al. , 2020 ) , as they are so challenging that existing RL agents are still far from being able to solve a large proportion of them even under the single game setting ( Yao et al. , 2020 ) . Furthermore , these games usually have different themes , vocabularies and logics , making it hard to determine the domain gap ( Ammanabrolu & Riedl , 2019b ) . Compared with these man-made games , the synthetic games ( Côté et al. , 2018 ; Urbanek et al. , 2019 ) provide a more natural way to study generalization by generating multiple similar games with customizable domain gaps ( e.g. , by varying game layouts ) . Generally , the training and testing game sets in previous works have either the same difficulty level ( Ammanabrolu & Riedl , 2019a ; Murugesan et al. , 2021 ) , or a mixture of multiple levels ( Adolphs & Hofmann , 2020 ; Yin et al. , 2020 ) , or both ( Adhikari et al. , 2020 ) . Our works can be used for generalization in games . Moreover , our work does not use the reward functions of environments to train an agent . 3 PRELIMINARIES . Text-based games as POMDPs . Text-based games can be formally described as Partially Observable Markov Decision Processes ( POMDPs ) . POMDPs can be defined as a tuple 〈S , A , T , R , Ω , O , γ〉 : the state set S , the action set A , the state transition probabilities T , the reward functionR , the observation set Ω , the conditional observation probabilitiesO and the discount factor γ ∈ ( 0 , 1 ] . At each time step , the agent will receive a textual observation ot ∈ Ω , depending on the current state and previous action via the conditional observation probability O ( ot|st , at−1 ) . By executing an action at ∈ A , the environment will transit into a new state based on the state transition probability T ( st+1|st , at ) , and the agent will receive the reward rt+1 = R ( st , at ) . Same as Markov Decision Process ( MDPs ) , the goal of the agent is to learn an optimal policy π∗ to maximize the expected future discounted sum of rewards from each time step : Rt = E [ ∑∞ k=0 γ krt+k+1 ] . Knowledge graph for text-based games . Graph-based representations are effective for text-based games because the state in these games adheres to a graph-like structure . The content in most observations of the environment corresponds either to entity attributes or to relational information about entities in the environment . Knowledge graph ( KG ) for a text-based game can be built from a set of triplets 〈Subject , Relation , Object〉 , denoting that the Subject has Relation with the Object . For example , 〈Kitchen , Has , Food〉 . The KG is denoted as G = ( V , E ) , where V and E are the node set and the edge set , respectively . Both Subject and Object belong to the node set V . Relation , which corresponds to the edge connecting them , belongs to E .

Summary: This paper introduces GoalRand, an algorithm for playing text-based games in the absence of extrinsic reward. In particular, it uses generated goals to train a goal-conditioned reinforcement learning agent. These goals are generated by extracting factual information from the agent's knowledge graph. Given a goal sampled from the set of possible goals, as defined by the current knowledge graph, the agent is given a intrinsic reward for completing this goal within a specified time limit of 5 steps. Goals are randomly sampled from the set of possible goals.   The paper contributes this goal generation methodology which allows the agent to be trained without access to extrinsic rewards. Additionally, experiments are conducted on Textworld's Cooking Games, comparing against an intrinsic motivation baseline (BeBold) as well as a previously high-performing agent on these games, GATA. The scores show higher performance from GoalRand compared to the baselines across a set of heldout layouts and levels.