Goal Randomization for Playing Text-based Games without a Reward Function

Abstract: Playing text-based games requires language understanding and sequential decision making. The objective of a reinforcement learning agent is to behave so as to maximise the sum of a suitable scalar reward function. In contrast to current RL methods, humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation. We propose a goal randomization method that uses random basic goals to train a policy in the absence of the reward of environments. Specifically, through simple but effective goal generation, our method learns to continuously propose challenging -- yet temporal and achievable -- goals that allow the agent to learn general skills for acting in a new environment, independent of the task to be solved. In a variety of text-based games, we show that this simple method results in competitive performance for agents. We also show that our method can learn policies that generalize across different text-based games. In further, we demonstrate an interesting result that our method works better than one of state-of-the-art agents GATA, which uses environment rewards for some text-based games.

Reviews and author responses:

This paper introduces GoalRand, an algorithm for playing text-based games in the absence of extrinsic reward. In particular, it uses generated goals to train a goal-conditioned reinforcement learning agent. These goals are generated by extracting factual information from the agent's knowledge graph. Given a goal sampled from the set of possible goals, as defined by the current knowledge graph, the agent is given a intrinsic reward for completing this goal within a specified time limit of 5 steps. Goals are randomly sampled from the set of possible goals. 

The paper contributes this goal generation methodology which allows the agent to be trained without access to extrinsic rewards. Additionally, experiments are conducted on Textworld's Cooking Games, comparing against an intrinsic motivation baseline (BeBold) as well as a previously high-performing agent on these games, GATA. The scores show higher performance from GoalRand compared to the baselines across a set of heldout layouts and levels. This paper presents a compelling narrative about how long-horizon text-based games can be decomposed into short-horizon goals, which can greatly help an agent solve the larger task. I really like the overall idea, and think that it is particularly applicable to text-based games which large tasks can be neatly decomposed into series of smaller tasks.

The main weaknesses of this paper are the lack of detail regarding GoalRand. 

The following points would benefit from clarification:

0) What RL algorithm was used to train GoalRand?
1) How is the knowledge graph populated/constructed from step to step? Is it trained to parse the observation into a <subject,relation,object> tuple or does it use hand-crafted rules?
2) How are goals extracted from the knowledge graph? In Section 4.2 we learn that goals are generated by common-sense rules relating to collecting ingredients that are not currently in inventory and preparing ingredients according to their requirements in the KG. How does the agent know what objects are ingredients versus other items in the environment like the stove? How are the requirements for preparing ingredients understood by the agent?
3) How is the pseudo-reward constructed for each goal - in particular the agent must know when the goal has terminated and whether the termination was successful or a failure. How does the agent determine successful criterion for each goal?

In short, while the authors describe a main contribution of the paper as an agent learning without rewards, however it's unclear how much of the learning ability of this agent comes from built-in heuristics surrounding the creation of the knowledge graph, the extraction of goals, and the construction of pseudo rewards. At the minimum I think the authors need to be much more clear on exactly how much heuristic knowledge is involved in steps 1-3 above.

Update after author response: I appreciate the author response and updated manuscript to include necessary detail. I think the amount of prior knowledge that is incorporated in the agent is a significant drawback of this approach - and one that would prevent it from being applied beyond the TextWorld Cooking games. Additionally, it makes comparisons to other baselines difficult. As the authors suggest it may be possible to replace some of this domain knowledge with learned components, which might yield a more general agent. Score updated accordingly. In the current form, I believe the paper is missing details that are crucial to understanding the agent and evaluating how much prior domain-specific prior knowledge was used to build and utilize the knowledge graph and goal sets. I'm hopeful the authors can provide these details in the review process and I'm open to adjusting the scoring of the paper accordingly.

Updated

 Q1:  It seems that significant amounts of domain-specific knowledge is used in both of these components. This makes it difficult to evaluate comparisons against baseline algorithms that may not benefit from the same amount of domain-specific knowledge.

A1: Note that we assume there is no environment rewards. This setting is not studied before and created by our work. In Table 2 and Figure 2, our main results test methods that can be used in the no reward setting. We introduce to use BeBold and Random as baselines and there is no GATA. Because both of them can be unsupervised and GATA needs environment rewards. 

Our method is to leverage human intelligence for solving text-based games. Using common-sense knowledge or rules is part of our method. Why other methods do not use such kind of knowledge? On one hand, there is no other methods for the no reward setting. On the other hand, previous methods like GATA (needs rewards and is designed for the standard setting) can not use common-sense knowledge directly. 

Q2: In particular, the main comparisons with BeBold and GATA might favor GoalRand simply because of this additional knowledge. 

A2: First, this is one of our conclusions. Second, our method is about not only using such human knowledge but also how to use it. 

We revised our paper to make how to use human intelligence and its impact more clear. 


 I appreciate the updates that were added to the paper to clarify the goal set generation and goal-conditioned reward acquisition. It seems that significant amounts of domain-specific knowledge is used in both of these components. This makes it difficult to evaluate comparisons against baseline algorithms that may not benefit from the same amount of domain-specific knowledge. In particular, the main comparisons with BeBold and GATA might favor GoalRand simply because of this additional knowledge. In the future I would suggest re-examining the experiments to ensure that all methods have access to the same amount of heuristic knowledge.

 Q1: What RL algorithm was used to train GoalRand?

A1: Following our building backbone GATA, we train GoalRand using Double DQN with Prioritized Experience Replay. 

Q2: How is the knowledge graph populated/constructed from step to step? Is it trained to parse the observation into a <subject,relation,object> tuple or does it use hand-crafted rules?

A2: We use the experiment setting of ``GATA-GTF’’, where the KG-based observation is obtained directly from the environment. Such setting helps us to avoid the influence of information extraction error. 

Q3: How are goals extracted from the knowledge graph? In Section 4.2 we learn that goals are generated by common-sense rules relating to collecting ingredients that are not currently in inventory and preparing ingredients according to their requirements in the KG. How does the agent know what objects are ingredients versus other items in the environment like the stove? How are the requirements for preparing ingredients understood by the agent?

A3: We use common sense rules to generate goals.  We provide more details in in Appendix B in our paper revision and the pipeline is as follows: For obtaining the goal set Gt, we first obtain the ingredient set I. For each ingredient i \in I, we first check whether it has been collected, then obtain its status set Si and requirement set Ri. We consider three types of goals: 1) “find” requires the agent to find and collect an uncollected ingredient, 2) “prepare” requires the agent to prepare an ingredient to satisfy a requirement,
and 3) “eat”, that the agent is required to prepare and eat the final meal.

Q4: How is the pseudo-reward constructed for each goal - in particular the agent must know when the goal has terminated and whether the termination was successful or a failure. How does the agent determine a successful criterion for each goal?

A4: We provide more details in in Appendix B in our paper revision and the pipeline is as follows: For assigning the goal-conditioned reward r^goal at t, we first obtain the type of a goal g, then check whether this goal has been accomplished given at and o^KG at t+1. Some functions in Algorithm 2 can be reused here. r^goal at t is a binary reward that we will assign r^goal_t = r_max if g is accomplished successfully, otherwise r_min (failed).

Q5:  At the minimum I think the authors need to be much more clear on exactly how much heuristic knowledge is involved in steps 1-3 above.

A5: In our revised paper, we make it more clear and add more details.


 Thank you for your review, regarding your comments and questions:

Q1: The novelty of this paper seems quite low, not enough to meet the ICLR bar. Goal randomization is a very simple idea and similar to intrinsic motivation.

A1: The idea is simple. However, it works without reward functions. We also find that it even works better than one of state-of-the-art agents GATA, which uses environment rewards for some text-based games.

Q2: I am a bit skeptical on whether the experimental results are solid. For example, the authors suggest that only encouraging exploration is not sufficient when the environmental reward is not available, because the BeBold baseline performs even worse than the non-learnable random agent. From Figure 2 it seems BeBold's normalized scores are still increasing and are very likely to surpass "random".

A2: In our paper revision, we update the experiments and extend the number of  training episodes to 100,000, which is the default setting in the original GATA paper. We found that BeBold baseline stops making progress and encounters performance drop with the progress of training, which might be caused by forgetting [1] or detachment [2]. Besides, from the perspective of sample efficiency, it’s impractical to assume the availability of unlimited training episodes. Under the condition of limited training episodes (100k for our work), our method is with much higher sample efficiency, while the BeBold baseline performs badly. 

[1] Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., ... & Silver, D. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782), 350-354.
[2] Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2021). First return, then explore. Nature, 590(7847), 580-586.

Q3: Same for the comparison in Figure 3, I kept wondering whether GATA is still undertrained. It seems in many settings GATA is still improving even at 60k. I also feel the authors need more explanation on how goal randomization is able to outperform an agent that has access to environmental reward. This is still a very surprising finding this is indeed the case.

A3: In our revised paper, we update the experiments and extend the number of training episodes to 100,000, which is the default setting in the original GATA paper. The baseline GATA is with the access to environment reward. Even though our method is without environment reward, it outperforms in most of the levels (S1, S2, S4, US2, US3 and US4). 

Q4:  Other than pure random goal, perhaps explore some other types of goal generation, e.g. something that is similar to curriculum learning?

A4: It’s interesting to investigate such a goal-based curriculum learning, we will leave it as a future direction. Thanks for providing the insights!


 Thank you for your review, regarding your comments and questions:

Q1: Goal generation sets are based on common sense rules specific to the game — it seems like relying on such curated rules may be hard to scale and/or extrapolate to more complex text games (e.g. where the action and item spaces are large). It would be interesting to see if the goal sets could be automatically generated or learned, rather than hand engineered?

A1: First, inspired by humans, common sense is to target general knowledge of the world and tasks. Thus it is not specific to a game and can be used different games. Our experiments demonstrate that we can use common sense rules to other unseen games. 
Second, we use a non-learnable goal set generation method for simplicity, however it can also be implemented via learning-based methods. For example, common sense knowledge graphs and other pretrained models. Concurrent to our work, [1] has proven the effectiveness of pre-trained QA modules in graph construction for text-based games. We are willing to investigate a similar module for automatic goal generation through answering specific questions.
[1] Ammanabrolu, P., Tien, E., Hausknecht, M., & Riedl, M. O. (2020). How to avoid being eaten by a grue: Structured exploration strategies for textual worlds. arXiv preprint arXiv:2006.07409.
 

Q2: As the goal space grows with game complexity, uniform sampling might be less and less effective. Is there a way to prioritize a curriculum of goals to sample instead?

A2: It’s interesting to investigate such a goal-based curriculum learning, we will leave it as a future direction. Thanks for providing the insights!


Q3: Would it be possible to combine GoalRand with a method using environmental reward for further gains?

A3: Yes, we will include such a variant in the final version. 


Q4: More clarity on the specifics of game scoring would help.

A4: We define the score as the sum of rewards of an episode (a game instance will be sampled per episode). In a cooking game, collecting an ingredient (an ingredient is a food object that is essential for making the meal), pre-preparing an ingredient correctly, preparing and making the meal will lead to +1 reward, otherwise 0. 
Depending on the difficulties of games, the games from different levels will have different maximum available scores. For example, an S4 game has 3 ingredients (3), where each ingredient has 2 requirements (6), so its maximum available score is 11 (prepare meal +1, make meal +1).
In order to make it clearer for comparison, we use the normalized score \in [0,1], which is defined as the collected score normalized by the maximum available score for this game, to measure the performance.


Q5: Bottom left plot on Figure 2 seems to be missing the Random baseline.

A5: We test the random agent directly on the testing game set. Therefore there’s no training performance for the Random baseline


The authors propose a simple intrinsic reward for text-based games, in the absence of environment rewards. The method, GoalRand, is based on uniformly sampling random goals from a set of natural language goals generated using common-sense rules. The authors evaluate their method on both seen and unseen text-based games and find that their method outperforms a random agent and a count-based intrinsically motivated agent. ### Strengths
- The proposed method seems straightforward and intuitive. Natural language is a promising modality for goal specification, and this work can potentially be insightful for goal conditioned RL research beyond text games.
- Empirical results  seem convincing. The authors suggest the improved performance of GoalRand is due to goal randomization improving exploration and decomposing complex tasks, which is interesting. 

### Weaknesses
- Goal generation sets are based on common sense rules specific to the game — it seems like relying on such curated rules may be hard to scale and/or extrapolate to more complex text games (e.g. where the action and item spaces are large). It would be interesting to see if the goal sets could be automatically generated or learned, rather than hand engineered?   
- As the goal space grows with game complexity, uniform sampling might be less and less effective. Is there a way to prioritize a curriculum of goals to sample instead? 

### Questions / Additional Feedback
- Would it be possible to combine GoalRand with a method using environment reward for further gains?  
- More clarity on the specifics of game scoring would help.  
- Bottom left plot on Figure 2 seems to be missing the Random baseline.  The empirical results seem compelling and the method is straightforward, and overall this paper provides insights into a more self-supervised method for text-based games. However, it is hard to say how well the method will extend to games where common sense rules are harder to come by or the goal space is much larger, where uniform sampling may work less well or be less sample efficient. It seems like the strengths come from crafting a goal space and generator tailored the given text games, and it would be interesting to see if the method could be improved by reducing reliance on hand picked rules. 

This paper studies text-based games using a technique called goal randomization. This method uses random basic goals to train a policy in the absence of the reward of environment. Authors show that agents can learn policies that generalize well across different text-based games. It also seems working better than a state-of-the-art algorithm that uses environment reward. Strength

The method proposed seems quite effective. It not only beats a random agent but also a SOTA algorithm that uses environment reward.

Weakness
1. The novelty of this paper seems quite low, not enough to meet ICLR bar. Goal randomization is a very simple idea and similar to intrinsic motivation.

2. I am a bit skeptical on whether the experimental results are solid. For example, the authors suggest that only encouraging exploration is not sufficient when the environment reward is not available, because the BeBold baseline performs even worse than the non-learnable random agent. From Figure 2 it seems BeBold's normalized scores are still increasing and is very likely to surpass "random".

3. Same for the comparison in Figure 3, I kept wondering whether GATA is still undertrained. It seems in many settings GATA is still improving even at 60k. I also feel the authors need more explanation on how goal randomization is able to outperform an agent that has access to environment reward. This is still a very surprising finding this is indeed the case.

4. Other than pure random goal, perhaps explore some other types of goal generation, e.g. something that is similar to curriculum learning? This paper studies an interesting problem, but lacks novelty and results are not so convincing. Authors need more results / discussions to show why this proposed method works so well.

Meta-review: This paper proposes intrinsic rewards to train agents without environment rewards in text-based games. The key contribution is a goal generation method that samples random goals from a set of valid goals in natural language, which are obtained based on commonsense rules. Reviewers generally agree that the proposed method is intuitive and simple to implement, and appreciates the new results added during discussion. However, there are two main concerns: 1) the goal creation process is largely rule-based and task-specific, therefor it's unclear how well this method would generalize to other tasks; 2) related to 1), the generate goals carry a significant amount of domain knowledge about the task that is not available to the baselines, making the comparison a bit unfair. A future submission would benefit from demonstrating the generalizability of the proposed approach, e.g., by using more generic resources such as game meta data, generic knowledge/commonsense bases.

Decision: Reject