Deep Zork — Part 2. Now that I have given some background…
Tags: Artificial Intelligence, Reinforcement Learning, Machine Learning, Interactive Fiction, Data Science

Game emulation

Since I am using Python for this project, the first step was to be able to play Zork through Python. The game file I found to use was what’s known as a Z-machine file. Emulators have been made to play this format of file but none of them are written in Python. I thus had to use an executable version I found called Frotz for Windows (https://github.com/DavidGriffith/frotz) which runs through the command line.

To interface the program with Python I had to use the Subprocess module, piping the command line input and output so that I could read the lines of the game and write new commands to the game. To be able to read the game line by line without blocking, I had to make use of a separate thread to run the game on.

With this interface set up, I was able to both pass commands from Python to the game and read the responses and narratives given by the game.

Step one down.

Command generation

The next challenge is that of command generation. From my initial research and play-throughs of the game, I knew that Zork only accepts certain commands.

These commands fall into one of three categories:

Basic commands: commands to go a certain direction, commands to get info

Go north

Look around

Check inventory

Verb-Object commands: commands that consist of a verb and a noun or noun phrase

Take key

Open the door

Verb-Object-Prep-Object commands: commands that consist of a verb and noun phrase followed by a preposition and second noun phrase

Attack the monster with the sword

Unlock the chest with the key

By scrubbing the game file using ZTools (https://www.inform-fiction.org/zmachine/ztools.html) and using the Spacy library to detect verb and noun phrases, I compiled a list of possible commands for each type.

Basic commands

'go north',

'go south',

'go west',

'go east',

'go northeast',

'go northwest',

'go southeast',

'go southwest',

'd',

'u'

Verb-Object commands

'open OBJ',

'get OBJ',

'set OBJ',

'hit OBJ',

'eat OBJ',

'put OBJ',

'cut OBJ',

'dig OBJ',

'ask OBJ',

'fix OBJ',

'make OBJ',

'wear OBJ',

'move OBJ',

'kick OBJ',

'kill OBJ',

'find OBJ',

'play OBJ',

'feel OBJ',

'hide OBJ',

'read OBJ',

'fill OBJ',

'flip OBJ',

'burn OBJ',

'pick OBJ',

'pour OBJ',

'pull OBJ',

'apply OBJ',

'leave OBJ',

'ask OBJ',

'break OBJ',

'enter OBJ',

'curse OBJ',

'shake OBJ',

'burn OBJ',

'inflate OBJ',

'brandish OBJ',

'donate OBJ',

'squeeze OBJ',

'attach OBJ',

'find OBJ',

'banish OBJ',

'read OBJ',

'enchant OBJ',

'feel OBJ',

'pour OBJ'

Verb-Object-Prep-Object commands

'pour OBJ on DCT',

'hide OBJ in DCT',

'pour OBJ in DCT',

'move OBJ in DCT',

'hide OBJ on DCT',

'flip OBJ for DCT',

'fix OBJ with DCT',

'spray OBJ on DCT',

'dig OBJ with DCT',

'cut OBJ with DCT',

'pick OBJ with DCT',

'squeeze OBJ on DCT',

'pour OBJ from DCT',

'fill OBJ with DCT',

'brandish OBJ at DCT',

'burn OBJ with DCT',

'flip OBJ with DCT',

'read OBJ with DCT',

'hide OBJ under DCT',

'carry OBJ from DCT',

'inflate OBJ with DCT',

'unlock OBJ with DCT',

'give OBJ to DCT',

'carry OBJ to DCT',

'spray OBJ with DCT'

You will notice that the verb-object commands have ‘OBJ’ in them as a placeholder for the noun phrase and the verb-object-prep-object commands have ‘OBJ’ and ‘DCT’ for the first and second noun phrases. Using these placeholders allowed me to easily replace them with the different nouns found in the game’s descriptions.

At this point I knew that for any given state of the game, I would simply grab all the nouns in the description and generate a list of all combinations of those nouns with the commands.

To better assist the AI in choosing relevant commands, I compiled a Word2Vec model using walkthroughs and tutorials for other text-based games. I used BeautifulSoup to scrape two different sites — https://www.ifarchive.org/indexes/if-archive/solutions/ and http://www.textfiles.com/adventure/ — and compiled all the instructions into a single file. I then used the GenSim library to train a Word2Vec model on the corpus.

Using this Word2Vec model I then used the similarity score on the commands to give a score to each command. I then used an exponential weight to make the difference in these scores more significant. Using these scores, when the AI selects a random action, actions that that are more likely to be valid will have a higher weight.

Reinforcement learning

The basic setup for reinforcement learning is to have an agent which selects actions to perform and sends them to the game environment. The new state of the game and the reward for the chosen action is then fed back to the agent to influence the agent’s next move.

Flow chart for RL

The state in this case is the surroundings given to the player as well as the player’s inventory. The two of these together provides the agent with all the information needed to select appropriate commands and interact with the environment.

The reward should influence “good” behaviour while penalizing “bad” behaviour by the agent. There is already an in-game score given to the player when reaching certain areas or performing certain actions. However additional rewards can help to direct the agent more successfully.

One of the most important parts of playing a text-based game is exploration. Any human player playing Zork would need to take part in exploration of the various areas and rooms to learn more about the game world and to find essential items.

Another important part is finding, taking, and using items in the game. For Zork especially, the main goal is to find and collect the various treasures so interacting with items should be rewarded.

As far as negative rewards go, taking too many turns in a game usually is a bad thing as the goal should be to get to the end as quick as possible. Thus, a small negative reward can be given per each turn to help dissuade the agent from taking more turns than neccesary.

Below are the different rewards.

negative_per_turn_reward : -1 points

- Each turn taken this reward will be added to the total reward for the round. This way, on turns where no other points were scored, the turn will be remembered as being non-productive.

new_area_reward : 2 points

- Discovering a new area or room in the game should be rewarded in order to encourage exploration.

moving_around_reward: 0.5 points

- In order to prevent the agent from staying in one room too long, a small positive reward is given each time the agent moves from one area to another that has already been visited. This also helps to encourage the agent to return to an area it previously left when it had not yet completed all objectives in that area. Providing this reward instead of the new_area_reward when the agent revisits an area will prevent it from exploiting the reward system and hopping back and forth between rooms.

inventory_reward : 3 points

- If the agent picks up an item or uses an item, the in-game inventory will change. This will be rewarded with a large number of points as finding the treasures of the game ultimately is how the agent can win.

inventory_not_new_reward : 0.5 points

- In order to prevent the agent taking advantage of the large inventory_reward, a much smaller reward will be given when an inventory change occurs that has already taken place. This will prevent exploitable situations in which the agent repeatedly picks up and then drops an item.

in_game_score_reward : in-game score

- Scoring points in the game is the best type of action the agent could take so it should be rewarded the greatest. Each time an in-game score is granted, a weight (initially set at 10) will be applied to it to show its significance to the agent.

The agent consists of a neural network — more specifically a Deep-Q Network (referred to as DQN). The architecture of the model can be seen below.

Input