Goal Randomization for Playing Text-based Games without a Reward Function

1 INTRODUCTION . Text-based games are complex , interactive simulations in which the game state is described with text and players act using simple text commands ( e.g. , take sandwich from table , eat sandwich , open door , etc . ) ( Côté et al. , 2018 ) . They serve as a proxy for studying how agents can exploit language to comprehend and interact with the environment . Text-based games are a useful challenge in the pursuit of intelligent agents that communicate with humans ( e.g. , in customer service systems ) . Inspired by this , one of the long term goals in AI is to build agents that can learn to accomplish tasks with language . In the domain of text-based games , the key challenge is to decipher the long textual observations , extract reward cues from them , and generate semantically rich representations such that the policy learned on top of it is well informed . Most of the existing works learn to model text representations during the RL training ( Hausknecht et al. , 2020 ; Ammanabrolu & Hausknecht , 2020 ) . Some works also study generalizability of games with different difficulty levels or layouts ( Ammanabrolu & Riedl , 2019a ; Adolphs & Hofmann , 2020 ) . Deep reinforcement learning ( RL ) has been demonstrated to effectively learn to solve reward-driven problems in various tasks ( Mnih et al. , 2013 ; Silver et al. , 2016 ; Schulman et al. , 2017 ) . In contrast , humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation . Playing text-based games without a reward function is an exceedingly challenging problem . We consider the setting where reward functions are unknown , so we want to learn an agent that can drive itself without environmental rewards . Learning agents without reward has several practical applications ( Eysenbach et al. , 2018 ) . Environments with sparse rewards effectively have no reward until the agent randomly reaches a goal state . Learning intelligent agents without supervision may help address challenges in exploration in these environments ( Gupta et al. , 2018 ; Sharma et al. , 2019 ) . In many practical settings , interacting with the environment is essentially free , but evaluating the reward requires human feedback ( Christiano et al. , 2017 ) . However , there is no work on playing text-based games without reward functions . Solving such kinds of non-reward tasks can encourage agents to explore , experiment , and invent . Sometimes , as in many games and fantasies , without any direct link to reality or to any source of extrinsic reward , it is crucial to enable learning in real-world environments , even for humans ( Schulz , 2012 ) . In this paper , we take a step towards agents that can learn from text-based games without reward functions . The environments are designed to mimic some real-world scenarios where there are no reward cues for guiding agents . The goal is for an agent to be able to learn one policy that is able to solve both tasks it was trained on as well as a variety of unseen tasks which contain similar tasks as the training tasks . To do this , we propose a new method for learning an agent with deep RL in the absence of any rewards . We use a set of available goals characterized by natural language via common-sense rules , and select one of them according to the current knowledge graph based observation . Then , we learn policies for goal-conditioned reinforcement learning . Specifically , our method works as follows . Whenever an agent builds a knowledge graph of the textural world , our method gives a goal in natural language to the agent based on the knowledge graph . The agent takes the goal to form a new experience with a corresponding intrinsic reward , alleviating the no reward problem . For example , our method can describe what the agent has achieved in the episode , and the agent can use goals as advice to obtain intrinsic rewards . In addition , our method also provides a time limit for a goal . If the agent can not accomplish the goal in the time limit , it can use a new goal replacing the old one . The agent can have the opportunity to get out of the difficult goals . We show many benefits brought by language goal representation when combined with goal advice . The agent can efficiently solve reinforcement learning problems in challenging text-based environments ; it can generalize to unseen instructions , and even generalize to instruction with unseen lexicons . Our method can be viewed as the intrinsic motivation of any agent trained with policy gradientbased methods ( Oudeyer & Kaplan , 2009 ; Barto , 2013 ) . Most intrinsic methods design intrinsic motivations to assist environmental rewards to make agents learn efficiently . However , our method is to use the intrinsic motivation to play text-based games without environmental rewards . Under this view , we also need to encode the intrinsic motivation into the policy . That is , the original policy network becomes a goal-conditional policy ; the goal advice can then be seen as a “ bolt-on ” to the original policy network . Because we use natural language for self-advice . It is flexible and can be used on a variety of RL training model architectures and training settings by encoding the intrinsic motivation . In summary , we make the following contributions : ( i ) we first study the problem of playing textbased games without any reward functions and propose a new goal randomization method for solving the problem ; 1 ( ii ) we show , through common-sense knowledge , that agents trained with goal randomization gradually learn to interact with the environment and solve tasks which are difficult for state-of-the-art methods ; ( iii ) we perform an extensive qualitative analysis and ablation study , and we also find an interesting result that our method works better than one of state-of-the-art agents GATA ( Adhikari et al. , 2020 ) , which uses environment rewards , for some text-based games . 2 RELATED WORK . Reinforcement learning for text-based games . Existing agents either perform based on predefined rules or learn to make responses by interacting with the environment . Rule-based agents ( Atkinson et al. , 2019 ; Fulda et al. , 2017 ; Hausknecht et al. , 2019 ; Kostka et al. , 2017 ) attempt to solve text-based games by injecting heuristics . They are thus not flexible since a huge amount of prior knowledge is required to design rules ( Hausknecht et al. , 2020 ) . Learning-based agents ( Adolphs & Hofmann , 2020 ; Hausknecht et al. , 2020 ; He et al. , 2016 ; Jain et al. , 2020 ; Narasimhan et al. , 2015 ; Yin & May , 2019 ; Yuan et al. , 2018 ; Zahavy et al. , 2018 ) usually employ deep reinforcement learning algorithms to deliver adaptive game solving strategies . KG-based agents have been developed to enhance the performance of learning-based agents with the assistance of KGs . KGs can be constructed by simple rules so that it substantially reduces the amount of prior knowledge required by rule-based agents . While KGs have been leveraged to handle partial observability ( Ammanabrolu & Hausknecht , 2020 ; Ammanabrolu & Riedl , 2019a ; Zelinka et al. , 2019 ) , reduce action space ( Ammanabrolu & Hausknecht , 2020 ; Ammanabrolu & Riedl , 2019a ) , and improve generalizability ( Adhikari et al. , 2020 ; Ammanabrolu & Riedl , 2019b ) . Recently , Murugesan et al . ( 2020 ) tried to introduce commonsense reasoning for playing synthetic games . While these works all follow the standard setting with reward functions , our work is the first work that trains an agent without reward functions . 1Code can be found here : https : //anonymous.4open.science/r/goalrand-E167/ Intrinsic motivation for reinforcement learning . Intrinsic motivation has been widely used in reinforcement learning ( Oudeyer et al. , 2007 ; Oudeyer & Kaplan , 2009 ; Barto , 2013 ) . It has been proven effective for solving various hard-exploration tasks ( Bellemare et al. , 2016 ; Pathak et al. , 2017 ; Burda et al. , 2018b ) . One prominent formulation is the use of novelty , which in its simplest form can be estimated with state visitation counts ( Strehl & Littman , 2008 ) and has been extended to high-dimensional state spaces ( Bellemare et al. , 2016 ; Burda et al. , 2018b ; Ostrovski et al. , 2017 ) . Other sophisticated versions of curiosity ( Schmidhuber , 1991 ) guide the agent to learn about environment dynamics by encouraging it to take actions that reduce the agent ’ s uncertainty ( Stadie et al. , 2015 ; Burda et al. , 2018b ) , have unpredictable consequences ( Pathak et al. , 2017 ; Burda et al. , 2018a ) , or a large impact on the environment ( Raileanu & Rocktäschel , 2020 ) . Other forms of intrinsic motivation include empowerment ( Klyubin et al. , 2005 ) which encourages control of the environment by the agent , and goal diversity ( Pong et al. , 2019 ) which encourages maximizing the entropy of the goal distribution . Intrinsic goals are discovered from language supervision ( Lair et al. , 2019 ) . Except for exploration , intrinsic motivation is also used for other problems , such as evolutionary ( Singh et al. , 2009 ; Sorg et al. , 2010 ; Zheng et al. , 2018 ) . Most works use intrinsic motivation as additive rewards for environmental rewards . Our work differs from those works by formulating intrinsic motivation to train an agent directly for text-based games . Generalization in text-based games . Generalization is a challenging problem for reinforcement learning ( Tobin et al. , 2017 ; Agarwal et al. , 2019 ) . In text-based games , it is difficult to study generalization in games initially designed for human players ( Hausknecht et al. , 2020 ) , as they are so challenging that existing RL agents are still far from being able to solve a large proportion of them even under the single game setting ( Yao et al. , 2020 ) . Furthermore , these games usually have different themes , vocabularies and logics , making it hard to determine the domain gap ( Ammanabrolu & Riedl , 2019b ) . Compared with these man-made games , the synthetic games ( Côté et al. , 2018 ; Urbanek et al. , 2019 ) provide a more natural way to study generalization by generating multiple similar games with customizable domain gaps ( e.g. , by varying game layouts ) . Generally , the training and testing game sets in previous works have either the same difficulty level ( Ammanabrolu & Riedl , 2019a ; Murugesan et al. , 2021 ) , or a mixture of multiple levels ( Adolphs & Hofmann , 2020 ; Yin et al. , 2020 ) , or both ( Adhikari et al. , 2020 ) . Our works can be used for generalization in games . Moreover , our work does not use the reward functions of environments to train an agent . 3 PRELIMINARIES . Text-based games as POMDPs . Text-based games can be formally described as Partially Observable Markov Decision Processes ( POMDPs ) . POMDPs can be defined as a tuple 〈S , A , T , R , Ω , O , γ〉 : the state set S , the action set A , the state transition probabilities T , the reward functionR , the observation set Ω , the conditional observation probabilitiesO and the discount factor γ ∈ ( 0 , 1 ] . At each time step , the agent will receive a textual observation ot ∈ Ω , depending on the current state and previous action via the conditional observation probability O ( ot|st , at−1 ) . By executing an action at ∈ A , the environment will transit into a new state based on the state transition probability T ( st+1|st , at ) , and the agent will receive the reward rt+1 = R ( st , at ) . Same as Markov Decision Process ( MDPs ) , the goal of the agent is to learn an optimal policy π∗ to maximize the expected future discounted sum of rewards from each time step : Rt = E [ ∑∞ k=0 γ krt+k+1 ] . Knowledge graph for text-based games . Graph-based representations are effective for text-based games because the state in these games adheres to a graph-like structure . The content in most observations of the environment corresponds either to entity attributes or to relational information about entities in the environment . Knowledge graph ( KG ) for a text-based game can be built from a set of triplets 〈Subject , Relation , Object〉 , denoting that the Subject has Relation with the Object . For example , 〈Kitchen , Has , Food〉 . The KG is denoted as G = ( V , E ) , where V and E are the node set and the edge set , respectively . Both Subject and Object belong to the node set V . Relation , which corresponds to the edge connecting them , belongs to E .

Summary: The authors propose a simple intrinsic reward for text-based games, in the absence of environment rewards. The method, GoalRand, is based on uniformly sampling random goals from a set of natural language goals generated using common-sense rules. The authors evaluate their method on both seen and unseen text-based games and find that their method outperforms a random agent and a count-based intrinsically motivated agent.