Microsoft Research Open Sourced TextWorld to Train Reinforcement Learning by Playing Text Games
By: Jesus Rodriguez
Tags: Machine Learning, Deep Learning, Data Science, Artificial Intelligence, Thesequence

Microsoft Research Open Sourced TextWorld to Train Reinforcement Learning by Playing Text Games

Language games are fun. How can they be used to train reinforcement learning agents?

I recently started a new newsletter focus on AI education. TheSequence is a no-BS( meaning no hype, no news etc) AI-focused newsletter that takes 5 minutes to read. The goal is to keep you up to date with machine learning projects, research papers and concepts. Please give it a try by subscribing below:

Conversational interfaces and natural language processing(NLP) are, arguably, the most widely adopted segment of modern artificial intelligence(AI). Despite the continuous progress in NLP research, most conversational interfaces today still feel rather primitive compare to a human equivalent. Most popular conversational AI agents such as Alexa or the Google Assistant are solid on very short dialogs but lack cognitive aspects of human dialogs such as memory, planning and even common sense. How can we establish a repeatable and quantifiable mechanism for training AI agents in sophisticated conversational capabilities? A few months ago, researchers from the Microsoft Research Montreal Lab, released an open source project called TextWorld, which attempts to train reinforcement learning agents using Text-Based games. The ideas behind TextWorld were captured in a recent research paper published by Microsoft.

Text-Based Games

It might seem unusual to talk about Text-Based games in a time in which AI agents are mastering complex multi-player games such as Dota2 or Quake III. Whereas multi-player graphic environments are great to train agents on spatial and time-based planning, Text-Based games can play a similar role developing advanced conversational skills such as affordance extraction, memory and planning, exploration and several others.

Conceptually, Text-based games are complex, interactive simulations in which text describes the game state and players make progress by entering text commands. After each command, the game usually provides some feedback to inform players how that command altered the game environment. A typical text-based game poses a series of puzzles to solve, treasures to collect, and locations to reach. Take, for instance, the legendary Zork game which is shown in the figure below. The game uses natural language to describe the state of the world, to accept actions from the player, and to report subsequent changes in the environment.

The richness and complexities of Text-Based games makes them an ideal environment to train reinforcement learning agents. If we extrapolate Text-Based games to the context of reinforcement learning agents, language plays both the role of the action and observation space. In both cases, the complexity of the space is combinatorial and compositional which creates many challenges for reinforcement learning agents.

Reinforcement Learning Challenges in Text-Based Games

The only simple thing about Text-Based games is the input/output mechanism via a terminal. Other than that, the nature of Text-Based games poses numerous challenges for reinforcement learning agents:

· Partial Observability: In the context of reinforcement learning, Text-Based games can be considered partially observable environments. At any given time, only a snapshot of the current game environment is presented to the players and even many of the local details might not be apparent in the observation.

· Exploration vs. Exploitation: The infinite friction of reinforcement learning models is amplified in Text-Based games. Throughout the game, players need to balance the ability of exploring the environment further vs. capitalizing on immediate rewards.

· Long-Term Credit Assignment: Sparse rewards are inherent to Text-Based games in which the agent must generate a sequence of actions before observing a change in the environment state or getting a reward signal. Knowing which actions produced a specific reward becomes incredibly challenging for reinforcement learning agents.

Entering TextWorld

The idea of TextWorld is not to directly create reinforcement learning agents that can beat a specific Text-Based games. That has been done before. TextWorld leverages Text-Based games differently by creating simplified representations of them that can be used to train reinforcement learning agents. Having simpler versions of Text-Based games improves the evaluation and interpretability of reinforcement learning algorithms in a highly controlled game space.

From the functional standpoint, TextWorld is a Python framework for creating Text-Based games environments that can be used to train reinforcement learning agents. The framework has two main components: a game generator and a game engine. The game generator converts high-level game specifications, such as number of rooms, number of objects, game length, and winning conditions, into an executable game source code in the Inform 7 language. The game engine is a simple inference machine that ensures that each step of the generated game is valid by using simple algorithms such as one-step forward and backward chaining.

Using TextWorld, it is possible to generate combinatorial sets of Text-Based games based on a specific set of parameters. TextWorld’s game generator takes as input a high-level specification of a game and outputs the corresponding executable game with specific parameters such as the number of rooms, the number of objects, the length of the quest, the winning conditions, and options for the text generation.

Reinforcement learning models can interact with TextWorld using a simple API that can be encapsulated in a few lines of Python code:

import textworld env = textworld.start("zork1.z5") game_state = env.reset() # Reset/initialize the game. reward, done = 0, False while not done: # Ask the agent for a command. command = agent.act(game_state, reward, done) # Send the command to the game and get the new state. game_state, reward, done = env.step(command)

Using that model, developers can create rich natural language training environments for reinforcement learning agents that will help with the development of skills such as memory, contextual analysis or long-term planning. TextWorld is available as an open source release in GitHub and is interoperable with many of the popular deep learning frameworks such as TensorFlow or PyTorch.