Title: Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals
Venue: NeurIPS 2023
Accepted: yes
Keywords: Games

Instruction Manual

Atari Games

Large Language Models

Language Models

Zero-shot

In-context prompting
Abstract: High sample complexity has long been a challenge for RL. On the other hand, humans learn to perform tasks not only from interaction or demonstrations, but also by reading unstructured text documents, e.g., instruction manuals. Instruction manuals and wiki pages are among the most abundant data that could inform agents of valuable features and policies or task-specific environmental dynamics and reward structures. Therefore, we hypothesize that the ability to utilize human-written instruction manuals to assist learning policies for specific tasks should lead to a more efficient and better-performing agent. We propose the Read and Reward framework. Read and Reward speeds up RL algorithms on Atari games by reading manuals released by the Atari game developers. Our framework consists of a QA Extraction module that extracts and summarizes relevant information from the manual and a Reasoning module that evaluates object-agent interactions based on information from the manual. An auxiliary reward is then provided to a standard A2C RL agent, when interaction is detected. Experimentally, various RL algorithms obtain significant improvement in performance and training speed when assisted by our design. Code at github.com/Holmeswww/RnR

Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals

Abstract

High sample complexity has long been a challenge for RL. On the other hand, humans learn to perform tasks not only from interaction or demonstrations, but also by reading unstructured text documents, e.g., instruction manuals. Instruction manuals and wiki pages are among the most abundant data that could inform agents of valuable features and policies or task-specific environmental dynamics and reward structures. Therefore, we hypothesize that the ability to utilize humanwritten instruction manuals to assist learning policies for specific tasks should lead to a more efficient and better-performing agent. We propose the Read and Reward framework. Read and Reward speeds up RL algorithms on Atari games by reading manuals released by the Atari game developers. Our framework consists of a QA Extraction module that extracts and summarizes relevant information from the manual and a Reasoning module that evaluates object-agent interactions based on information from the manual. An auxiliary reward is then provided to a standard A2C RL agent, when interaction is detected. Experimentally, various RL algorithms obtain significant improvement in performance and training speed when assisted by our design. Code at github.com/Holmeswww/RnR.

1 Introduction

Reinforcement Learning (RL) has achieved impressive performance in a variety of tasks, such as Atari (Mnih et al., 2015; Badia et al., 2020), Go (Silver et al., 2017), or autonomous driving (Fuchs et al., 2021; Kiran et al., 2022), and is hypothesized to be an important step toward artificial intelligence (Silver et al., 2021). However, RL still faces a great challenge when applied to complicated, realworld-like scenarios (Shridhar et al., 2020b; Szot et al., 2021) due to its low sample efficiency. The Skiing game in Atari, for example, requires the skier (agent) to ski down a snowy hill and hit the objective gates in the shortest time, while avoiding obstacles. Such an intuitive game with simple controls (left, right) still required 80 billion frames to solve with existing RL algorithms (Badia et al., 2020), roughly equivalent to 100 years of non-stop playing.

Observing the gap between RL and human performance, we identify an important cause: the lack of knowledge and understanding about the game. One of the most available sources of information, that is often used by humans to solve real-world problems, is unstructured text, e.g., books, instruction manuals, and wiki pages. In addition, unstructured language has been experimentally verified as a powerful and flexible source of knowledge for complex tasks (Tessler et al., 2021). Therefore, we hypothesize that RL agents could benefit from reading and understanding publicly available instruction manuals and wiki pages.

26

0:10.73

Object:

To run through all gates (between the poles) in the fastes time.

Detection

scoring:

Elapsed time on timer.You are penalized five seconds for each gate

you miss.At the end of each run，the number of gates you missed

and your total time_(including penalties，if any）are shown on the

screen.

Grounding

Ifyouhitagate oratree

do not lose heart:Youskierwill jump back up

andkeep going.But you do lose time，so be careful!

tree

The red button on your Joystick is not used in slalom racing,but can be used

instead of the game reset switch to reset your skier at the top of the course.

skier

Don't push it unless you want to start over.

To make a run tougher, push your right difficulty switch up to a.Now trees

ACTVSION

pole

p       1. o'(s

will appear on the slalom course in front of the gates (not just off to the

gates.

Read

QAExtractionModule

Question:What happens when player hits pole?

Answer:You are penalized five seconds foreachgate you miss.

Question:Howto succeed in the game?

Question:Whatistheobjectiveof thegame?

Answer:To reach the bottom of the hillin the fastes timeHave fun andgod bless!".

Question:What happens when playerhits tree?

Answer:Learning to control the tips of your skis and anticipating and avoiding trouble.

Answer:You'lllosetime.

Question:Howtoscoreat thegame?

Answer:Elapsed time at the end of therun.

ReasoningModule

Bounding box intersect?

Question:Shouldyou hit agateif you want towin?

Answer:Yes

Auxiliary Reward:

(player, gate) -> 5

Question: Should you hit a tree if you want to win?

(player,tree)->-5

Answer:No

Reward

In this work, we demonstrate the possibility for RL agents to benefit from human-written instruction manuals. We download per-game instruction manuals released by the original game developers 1 or Wikipedia and experiment with 4 Atari games where the object location ground-truths are available.

Two challenges arise with the use of human-written instruction manuals: 1) Length : Manuals contain a lot of redundant information and are too long for current language models. However, only a few paragraphs out of the 2-3 page manuals contain relevant information for any specific aspect of concern, i.e., the game's objective or a specific interaction. 2) Reasoning : References may be implicit and attribution may require reasoning across paragraphs. For example, in Skiing, the agent needs to understand that hitting an obstacle is bad using both 'you lose time if the skier hits an obstacle' and 'the objective is to finish in shortest time', which are from different parts of the manual. Furthermore, the grounding of in-game objects to references in the text remains an open problem and manuals may contain significant syntactic variation . For example, the phrase 'don't hit an obstacle' is also referred to as 'don't get wrapped around a tree' or 'avoid obstacles'. Finally, the grounding of in-game features to references in the text remains an open problem (Luketina et al., 2019).

Our proposed Read and Reward framework (Figure 1) addresses the challenges with a two-step process. First, a zero-shot extractive QA module is used to extract and summarize relevant information from the manual, and thus tackles the challenge of Length . Second, a zero-shot reasoning module, powered by a large language model, reasons about contexts provided by the extractive QA module and assigns auxiliary rewards to in-game interactions. After object detection and localization, our system registers an interaction based on the distance between objects. The auxiliary reward from the interaction can be consumed by any RL algorithm.

Even though we only consider 'hit' interactions (detected by tracking distance between objects) in our experiments, Read and Reward still achieves 60% performance improvement on a baseline using 1000x fewer frames than the SOTA for the game of Skiing. In addition, since Read and Reward does not require explicit training on synthetic text (Hermann et al., 2017; Chaplot et al., 2018; Janner et al., 2018; Narasimhan et al., 2018; Zhong et al., 2019, 2021; Wang & Narasimhan, 2021), we observe consistent performance on manuals from two different sources: Wikipedia and Official manuals.

To our knowledge, our work is the first to demonstrate the capability to improve RL performance in an end-to-end setting using real-world manuals designed for human readers.

2 Background

2.1 Reducing sample complexity in RL

Prior works have attempted to improve sample efficiency with new exploration techniques (Schulman et al., 2017; Haarnoja et al., 2018; Badia et al., 2020), self-supervised objectives (Pathak et al., 2017; Schrittwieser et al., 2020), and static demonstration data (Kumar et al., 2019, 2020). Fan & Xiao (2022) cast RL training into a training data distribution optimization problem, and propose a policy mathematically controlled to traverse high-value and non-trivial states. However, such a solution requires manual reward shaping. Currently, the SOTA RL agent (Badia et al., 2020) still spends billions of frames to learn the Skiing game that can be mastered by humans within minutes.

2.2 Grounding objects for control problems

Most prior works study grounding - associating perceived objects with appropriate text - in the setting of step-by-step instruction following for object manipulation tasks (Wang et al., 2016; Bahdanau et al., 2018) or indoor navigation tasks (Chaplot et al., 2018; Janner et al., 2018; Chen et al., 2019; Shridhar et al., 2020a). The common approach has been to condition the agent's policy on the embedding of both the instruction and observation (Mei et al., 2016; Hermann et al., 2017; Janner et al., 2018; Misra et al., 2017; Chen et al., 2019; Pashevich et al., 2021). Recently, modular solutions have been shown to be promising (Min et al., 2021). However, the majority of these works use synthetic language. These often take the form of templates, e.g., 'what colour is < object > in < room > ' (Chevalier-Boisvert et al., 2018). On the other hand, recent attempts on large-scale image-text backbones, e.g., CLIP (Radford et al., 2021) have shown robustness at different styles of visual inputs, even on sketches. Therefore, we pick CLIP as our model for zero-shot grounding.

2.3 Reinforcement learning informed by natural language

In the language-assisted setting, step-by-step instructions have been used to generate auxiliary rewards, when environment rewards are sparse. Goyal et al. (2019); Wang et al. (2019) use auxiliary reward-learning modules trained offline to predict whether trajectory segments correspond to natural language annotations of expert trajectories.

In a more general setting, text may contain both information about optimal policy and environment dynamics. Branavan et al. (2012) improve Monte-Carlo tree search planning by accessing a natural language manual. They apply their approach to the first few moves of the game Civilization II, a turn-based strategy game. However, many of the features they use are handcrafted to fit the game of Civilization II, which limits the generalization of their approach.

Narasimhan et al. (2018); Wang & Narasimhan (2021) investigate planning in a 2D game environment with a fixed number of entities that are annotated by natural language (e.g. the 'spider' and 'scorpion' entities might be annotated with the descriptions 'randomly moving enemy' and 'an enemy who chases you', respectively). The agent learns to generalize to new environment settings by learning the correlation between the annotations and the environmental goals and mechanics. The proposed agent achieves better performance than the baselines, which do not use natural language. However the design of the 2D environment uses 6-line descriptions created from templates. Such design oversimplifies the observations into a 2D integer matrix, and removes the need for visual understanding and generalization to new objects and new formats of instruction manual, e.g., human written manuals.

2.4 RL Models that Read Natural Language Instructions

Zhong et al. (2019, 2021) make use of special architectures with Bidirectional Feature-wise Linear Modulation Layers that support multi-hop reasoning on 6 × 6 grid worlds with template-generated instruction manuals. However, the model requires 200 million training samples from templates identical to the test environments. Such a training requirement results in performance loss even on 10 × 10 grid worlds with identical mechanics, thus limiting the generalization of the model.

Wang & Narasimhan (2021) mixes embedding of entities with multi-modal attention to get entity representations, which are then fed to a CNN actor. The attention model and CNN actor are trained jointly using RL. However, the whole framework is designed primarily for a grid world with a template generated instruction manual consisting of one line per entity. In our experiments, we find

Question:What is the objective of the game?

The cbject of the gane La to score as nany points as you cas vhile

2. GME PLAY

Answer:The player earns points by eating pellets and avoiding ghosts.

Extractive

Question:How to succeed inthe game?

QA

Answer:The player earns points by eating pellets and avoiding ghosts.

belov the ghoeta' relscarnation chaber in the conter of the maze

(see Figure 1)-

Question:How toscore at thegame?

Dse your Joyatick controller to guide KS. rAC=MAs arousd the naze, Mben

Flgure 1 (pietere of the gane sereen)

abe passes over dota, abe gobbles thenup,and you score pointa.If she

eats all the dots on the sereen, the nase flashes nonestarily. and you

reu og e 0o fur.6erd s28

Answer:The player earns points by eating pellets and avoiding ghosts.

Question:What happens when player hits point?

Context

out1 Tearful ghosts scurry about trying to gobble up Ns. 2ac-sA. Ser oly

recoerse is to eat an ebergy pill and then go after the ghost8.

point

Answer: N/A.

400forththirdghot,adawopping 1600 gotnta forthe fourthghot,

4l1 os the same energy pill. (See Section 6, SCoRI3G, for details about

as soon as she gelps dovs the energy pil1, the ghosts turs blse with fright.

Tou ears 200 points for the first ghoet, 400 points for the seoond ghost,

pacman

point values-)

pellet

Extractive

gobble up the ghosts, they start to blink just before they turn back to

thelir original colors.

3. USISG TEE COBTROLLER

ghost

with

QA

Answer:She gets gobbled up.

Question:What happens when playerhits ghost?

Dse one Joystiek Costroller vith the AtAkI Gane Progran cartridge. Be sure

pill

Question

Vith theredbuttontoyourepperleft,toard the televiain scren.

(*eTeep eg g re s,30  )

eat

Template

naze, Use it also to slide Ms. FAC-KAH is and out of tunnels on both sides

Dse the Joystiek to saneuver Ms. FACKAs up, dovn, right and left areund the

of tbe saze.

round

Figure 2 (poiating out all four tusnels)

If HS. PAC-MM exits on the right, she re-enters on the leftj if she exits

on the left, sbe ze=enters on the zight (oee Figure 2).

Answer:Shegobbles them up

Question:What happens when player hitsfruit?

BJECrs/scORING:Eachtine as Alies catches yo, ne Hnana 1ost.Yo

to reters.

that extracting entity embeddings from human-written Atari instruction manuals following Wang & Narasimhan (2021) does not improve the performance of our vision-based agent due to the complexity and diversity of the language.

3 Read and Reward

We identify two key challenges to understanding and using the instruction manual with an RL agent.

The first challenge is Length . Due to the sheer length of the raw text ( 2 ∼ 3 pages), current pre-trained language models cannot handle the raw manual due to input length constraints. Additionally, most works in the area of long-document reading and comprehension (Beltagy et al., 2020; Ainslie et al., 2020; Zemlyanskiy et al., 2021) require a fine-tuning signal from the task, which is impractical for RL problems since the RL loss already suffers from high variance. Furthermore, an instruction manual may contain a lot of task-irrelevant information. For example, the official manual for MsPacman begins with: 'MS. PAC-MAN and characters are trademark of Bally Midway Mfg. Co. sublicensed to Atari, Inc. by Namco-America, Inc.'

It is therefore intuitive in our first step to have a QA Extraction Module that produces a summary directed toward game features and objectives, to remove distractions and simplify the problem of reasoning.

The second challenge is Reasoning . Information in the manual may be implicit. For example, the Skiing manual states that the goal is to arrive at the end in the fastest time, and that the player will lose time when they hit a tree, but it never states directly that hitting a tree reduces the final reward. Most prior works (Eisenstein et al., 2009; Narasimhan et al., 2018; Wang & Narasimhan, 2021; Zhong et al., 2019, 2021) either lack the capability of multi-hop reasoning or require extensive training to form reasoning for specific manual formats, limiting the generalization to real-world manuals with a lot of variations like the Atari manuals.

On the other hand, large language models have achieved success without training in a lot of fields including reading comprehension, reasoning, and planning (Brown et al., 2020; Kojima et al., 2022; Ahn et al., 2022). Therefore, in the Reasoning Module, we compose natural language queries about specific object interactions in the games, to borrow the in-context reasoning power of large language models.

The rest of the section describes how the two modules within our Read and Reward framework (a QA extraction module and a reasoning module) are implemented.

QA Extraction Module (Read) In order to produce a summary of the manual directed towards game features and objectives, we follow the extractive QA framework first proposed by Devlin et al. (2018), which extracts a text sub-sequence as the answer to a question about a passage of text. The extractive QA framework takes raw text sequence S manual = { w 0 , ..., w L } and a question string S question as input. Then for each token (word) w i in S manual, the model predicts the probability that the current token is the start token p i start and end token p i end with a linear layer on top of word piece embeddings. Finally, the output S answer = { w start , ..., w end } is selected as a sub-sequence of S manual that maximizes the overall probability of the start and end: w start , w end = arg max w start ,w end p start p end .

Implementation-wise, we use a RoBERTa-large model (Liu et al., 2019) for Extractive QA fine-tuned on the SQUAD dataset (Rajpurkar et al., 2016). Model weights are available from the AllenNLP API 2 . To handle instruction manual inputs longer than the maximum length accepted by RoBERTa, we split the manual into chunks according to the max-token size of the model, and concatenate extractive QA outputs on each chunk.

As shown in Figure 2, we compose a set of generic prompts about the objective of the game, for example, 'What is the objective of the game?'. In addition, to capture information on agent-object interactions, we first identify the top 10 important objects using TFIDF. Then for each object, we obtain an answer to the generated query:'What happens when the player hit a < object > ?'.

Finally, we concatenate all non-empty question-answer pairs per interaction into a < context > string. Note that the format of context strings closely resembles that of a QA dialogue.

Zero-shot reasoning with pre-trained QA model (Reward) The summary < context > string from the QA Extraction module could directly be used as prompts for reasoning through large language models (Brown et al., 2020; Kojima et al., 2022; Ahn et al., 2022).

Motivated by the dialogue structure of the summary, and the limit in computational resources, we choose Macaw (Tafjord & Clark, 2021), a general-purpose zero-shot QA model with performance comparable to GPT-3 (Brown et al., 2020). Compared to the RoBERTa extractive QA model for the QA Extraction module, Macaw is significantly larger and is better suited for reasoning. Although we find that GPT-3 generally provides more flexibility and a better explanation for its answers for our task, Macaw is faster and open-source.

As shown in Figure 3, for each interaction we compose the query prompt as ' < context > Question: Should you hit a < object of interaction > if you want to win? Answer: ', and calculate the LLM score on choices { Yes, No } . The Macaw

Concatenated

Question:What istheobjectiveof thegame?

Answer:The player earns pointsbyeatingpellets and avoidingghosts.

Question:Howtosucceed inthegame?

Question:Howtoscoreatthegame?

context+question

Answer:Theplayerearnspointsbyeatingpelletsandavoidingghosts.

Question:Whathappens when playerhitsghost?

Answer:Shegetsgobbledup.

Answer:Yes

Answer:No

Reward:5

Reward:-5

model directly supports scoring the options in a multiple-choice setting. We note that for other generative LLMs, we can use the probability of 'Yes' vs. 'No' as scores, following Ahn et al. (2022).

Finally, during game-play, a rule-based algorithm detects 'hit' events by checking the distance between bounding boxes. For detected interactions, an auxiliary reward is provided to the RL algorithm according to { Yes, No } rating from the reasoning module (see Section 4.4 for details).

4 Experiments

4.1 Atari Environment and Baselines

The Openai gym Atari environment contains diverse Atari games designed to pose a challenge for human players. The observation consists of a single game screen (frame): a 2D array of 7-bit pixels, 160 pixels wide by 210 pixels high. The action space consists of the 18 discrete actions defined by the joystick controller. Instruction manuals released by the original game developers have been scanned and parsed into publicly available HTML format 3 . For completeness, we also attach an explanation of the objectives for each game in Appendix A.3.

4.2 Delayed Reward Schedule

Most current Atari environments include a dense reward structure, i.e., the reward is obtained quite often from the environment. Indeed, most RL algorithms for the Atari environments perform well due to their dense reward structure. However, in many real-world scenarios (Kolve et al., 2017; Wang et al., 2022a) or open-world environments (Fan et al., 2022), it is expensive to provide a dense reward. The reward is usually limited and 'delayed' to a single positive or negative reward obtained at the very end of an episode (e.g., once the robot succeeds or fails to pour the coffee).

Notably, Atari games with the above 'delayed' reward structure, such as Skiing, pose great challenges for current RL algorithms, and are among some of the hardest games to solve in Atari (Badia et al., 2020). Therefore, to better align with real-world environments, we use a delayed reward version of the tennis, breakout, and Pac-Man games by providing the reward (the 'final game score') only at the end of the game. This delayed reward structure does not change the optimization problem, but is more realistic and imposes additional challenges to RL algorithms.

4.3 Grounding Objects in Atari

We find grounding (detecting and relating visual objects to keywords from TFIDF) a significant challenge to applying our framework. In some extensively studied planning/control problems, such as indoor navigation, pre-trained models such as masked RCNN reasonably solve the task of grounding (Shridhar et al., 2020a). However, the objects in the Atari environments are too different from the training distribution for visual backbones like mask RCNN. Therefore, the problem becomes very challenging for the domain of Atari. Since it is not the main focus of our proposed work, we attempt to provide an unsupervised end-to-end solution only for the game of Skiing.

Full end-to-end pipeline on Skiing with A2C

To demonstrate the full potential of our work, we offer a proof-of-concept agent directly operating on raw visual game observations and the instruction manual for the game of Skiing. For unsupervised object localization, we use SPACE, a method that uses spatial attention and decomposition to detect visual objects (Lin et al., 2020). SPACE is trained on 50000 random observations from the environment, and generates object bounding boxes. These bounding boxes are fed to CLIP, a zero-shot model that can pair images with natural language descriptions (Radford et al., 2021), to classify the bounding box

into categories defined by key-words from TF-IDF. The SPACE/CLIP models produce mostly reasonable results; however, in practice we find that CLIP lacks reliability over time, and SPACE cannot distinguish objects that cover each other. Therefore, to improve classification reliability, we use a Multiple Object Tracker (Bewley et al., 2016) in conjunction with CLIP and set the class of the bounding box to be the most dominant class over time.

Ground-truth object localization/grounding experiments For other games (Tennis, Ms. PacMan, and Breakout), we follow Anand et al. (2019) and directly calculate the coordinates of all game objects from the simulator RAM state. Specifically, we first manually label around 15 frames with object name and X,Y coordinates and then train a linear regression model mapping from simulator RAM state to labeled object coordinates. Note that this labeling strategy fails for games in which multiple objects of the same kind appears, i.e., multiple trees appear in one frame of Skiing.

4.4 Simplifying Interaction Detection to Distance Tracking

We make the assumption that all object interactions in Atari can be characterized by the distance between objects, and we ignore all interactions that do not contain the agent. We note that since the Atari environment is in 2D, it suffices to consider only interactions where objects get close to the agent so that we can query the manual for 'hit' interaction.

We, therefore, monitor the environment frames and register interaction whenever an object's bounding boxes intersect with the agent. For each detected interaction, we assign rewards from {-r n , r p } accordingly as detailed in Section 3. Experimentally, we find that it suffices to set r n = r p = 5 .

4.5 RL Baselines

Since Read and Reward only provides an auxiliary reward, it is compatible with most RL frameworks such as A2C, PPO, SAC, TD3, and Agent 57 (Mnih et al., 2016; Schulman et al., 2017; Haarnoja et al., 2018; Fujimoto et al., 2018; Badia et al., 2020). We use popular open-source implementations A2C,PPO,R2D1 (Stooke & Abbeel, 2019), and Agent57 (Hu, 2022).

4.6 Full Pipeline Results on Skiing with A2C

We compare the performance of our Read and Reward agent using the full pipeline to baselines in Figure 5, our agent is best on the efficiency axis and close to best on the performance axis. A2C equipped with Read and Reward outperforms A2C baseline and more complex methods like MuZero (Schrittwieser et al., 2020). We note a 50% performance gap between our agent and the SOTA Agent 57, but our solution does not require policy mixing and requires more than 10 times fewer frames to converge to performance 60% better than random.

30000

25000

20000

15000

10000

5000

10 7

10 8

10 9

10 10

10 11

Frames Seen (Efficiency)

Average Human

Random Agent

Read and Reward (Ours)

A2C (Baseline)

R2D2

MuZero

Agent 57

GDI-H3

Game Score (Performance)

When experimenting with a full end-to-end pipeline as described in Section 4.3, we notice instability with the detection and grounding models. As shown in Figure 4, missing bounding boxes cause lots of mislabeling when objects are close to each other. Such errors may lead to the auxiliary reward being incorrectly assigned.

Table 3: Table showing the outputs of the QA Extraction module on Wikipedia instructions vs the official Atari manual. The Wikipedia manual is significantly shorter, and contains less information, causing the extractive QA model to use the same answer for all questions. Full table for all 4 games is shown in Table 5 in the Appendix.

Although Skiing is arguably one of the easiest games in Atari for object detection and grounding, with easily distinguishable objects and white backgrounds, we still observe that current unsupervised object detection and grounding techniques lack reliability.

4.7 Results on Games with Delayed Reward Structure

Since the problem of object detection and grounding is not the main focus of this paper, we obtain ground-truth labels for object locations and classes as mentioned in Section 4.3. In addition, for a setting more generalizable to other environments (Kolve et al., 2017; Fan et al., 2022), we implement delayed reward schedule (Section 4.2) for Tennis, MsPacman, and Breakout. Various RL frameworks benefit significantly from Read and Reward, as shown in Table 1, 2.

We plot the Game Score and the auxiliary rewards of A2C with Read and Reward v.s. the A2C baseline in Figure 6. The A2C baseline fails to learn under the sparse rewards, while the performance of the Read and Reward agent continues to increase with more frames seen. In addition, we observe that the auxiliary reward from the Read and Reward framework has a strong positive correlation with the game score (game reward) of the agent, suggesting that the A2C agent benefits from optimizing the auxiliary rewards at training time.

4.8 Results without Delayed Reward

Without delayed reward, Read and Reward does not offer performance improvement in terms of game score, but still improves training speed. As shown in Table 4, Agent57 with Read and Reward requires up to 112X less training steps to reach the same performance as the Agent57 baseline without the instruction manuals.

0.0

0.2

0.4

0.6

0.8

1.0

1e8

2

4

6

8

10

12

Breakout

Game Score

Read and Reward (Ours)

A2C (Baseline)

0.0

0.2

0.4

0.6

0.8

1.0

1e8

20

40

60

80

Auxiliary Reward

0.0

0.2

0.4

0.6

0.8

1.0

1e8

-20

-15

-10

-5

Tennis

Game Score

Read and Reward (Ours)

A2C (Baseline)

0.0

0.2

0.4

0.6

0.8

1.0

1e8

100

200

300

400

500

Auxiliary Reward

0.0

0.2

0.4

0.6

0.8

1.0

1e8

100

200

300

400

MsPacman

Game Score

Read and Reward (Ours)

A2C (Baseline)

0.0

0.2

0.4

0.6

0.8

1.0

1e8

-17.0

-16.5

-16.0

-15.5

-15.0

Auxiliary Reward

4.9 R&RBehavior on Manuals from Different Source

To illustrate the generalization of our proposed framework to a different source of information, we download the 'Gameplay' section from Wikipedia 4 for each game and feed the section as manual to our model. In Table 3, we show a comparison of the answers on the set of 4 generic questions on the Pacman by our QA Extraction Module. Note that the QA Extraction module successfully extracts all important information about avoiding ghosts and eating pellets to get points. In addition, since the official manual contains more information than the Gameplay section of Wikipedia (2 pages v.s. less than 4 lines), we were able to extract much more information from the official manual.

Due to the high similarity in the output from the QA Extraction Module, the Reasoning module demonstrates good agreement for both the Wiki manual and the official manual. Therefore, we observe consistent agent performance between the official manual and the Wikipedia manual in terms of game score (Table 1).

5 Limitations and Future Works

Table 4: Table showing the number of training steps required by Agent57 with Read and Reward (Official) to reach the same performance as the Agent57 baseline at 1e6 training steps and the speedup ratio, under delayed-reward.

One potential challenge and limitations is the requirement of object localization and detection as mentioned in Section 4.3. However, since Read and Reward does not need to detect all objects to assign reward, i.e., energy pills are not grounded in the game Pacman but Read and Reward is still achieves improvement. In addition, reliable backbone object detection models (He et al., 2017) have already shown success in tasks such as indoor navigation (Shridhar et al., 2020b) and autonomous driving. With recent progress on visual-language models (Bubeck et al., 2023; Li et al., 2022; Driess et al., 2023; Liu et al., 2023; Zou et al., 2023), we believe that our proposed framework can benefit from reliable and generalizable grounding models in the foreseeable future.

Another simplification we made in Section 4.4 is to consider only interactions where objects get sufficiently close to each other. While such simplification appears to be sufficient in Atari, where observations are 2D, one can imagine this to fail in a 3D environment like Minecraft (Fan et al., 2022) or an environment where multiple types of interaction could happen between same objects (Kolve et al., 2017). Future works would benefit by grounding and tracking more types of events and interactions following recent works on video scene understanding Wang et al. (2022b) or game progress tracking Wu et al. (2023).

Finally, the performance of Read and Reward may be limited by information within the natural language data it reads.

6 Conclusions

In this work, we propose Read and Reward, a method that assists and speeds up RL algorithms on the Atari challenges by reading downloaded game manuals released by the Atari game developers. Our method consists of a QA extraction module that extracts and summarizes relevant information from the manual and a reasoning module that assigns auxiliary rewards to detected in-game events by reasoning with information from the manual. The auxiliary reward is then provided to standard RL agents. To our knowledge, this work is the first successful attempt for using instruction manuals in a fully automated and generalizable framework for solving the widely used Atari RL benchmark (Badia et al., 2020).

Our results suggest that even small open-source QA language models could extract meaningful gameplay information from human-written documentations for RL agents. With recent break-trough in zero-shot language understanding and reasoning with LLMs (Bubeck et al., 2023), we hope that our work could incentivize future works on integrating human prior knowledge into RL training.

Broader Impacts

Our research on using instruction manuals for RL could lead to both positive and negative impacts. The benefits include more efficient RL agents that could solve more real-world problems. The risks may involve more game cheating and exploitation, over reliance on the instruction manuals, and potential job loss.

References

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022.

Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers. arXiv preprint arXiv:2004.08483 , 2020.

Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Cˆ ot´ e, and R Devon Hjelm. Unsupervised state representation learning in atari. arXiv preprint arXiv:1906.08226 , 2019.

Adri` a Puigdom` enech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning , pp. 507-517. PMLR, 2020.

Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprint arXiv:1806.01946 , 2018.

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020.

Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP) , pp. 3464-3468. IEEE, 2016.

SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in a monte-carlo framework. Journal of Artificial Intelligence Research , 43:661-704, 2012.

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020.

S´ ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 12538-12547, 2019.

Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272 , 2018.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

Jacob Eisenstein, James Clarke, Dan Goldwasser, and Dan Roth. Reading to learn: Constructing features from semantic abstracts. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing , pp. 958-967, 2009.

Jiajun Fan and Changnan Xiao. Generalized data distribution iteration. arXiv preprint arXiv:2206.03192 , 2022.

Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv:2206.08853 , 2022.

Florian Fuchs, Yunlong Song, Elia Kaufmann, Davide Scaramuzza, and Peter D¨ urr. Super-human performance in gran turismo sport using deep reinforcement learning. IEEE Robotics and Automation Letters , 6(3):4257-4264, 2021. doi: 10.1109/LRA.2021.3064284.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actorcritic methods. In International conference on machine learning , pp. 1587-1596. PMLR, 2018.

Prasoon Goyal, Scott Niekum, and Raymond J Mooney. Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv:1903.02020 , 2019.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Kaiming He, Georgia Gkioxari, Piotr Doll´ ar, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pp. 2961-2969, 2017.

Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551 , 2017.

Michael Hu. Deep RL Zoo: A collections of deep rl algorithms implemented with pytorch, 2022. URL https://github.com/michaelnny/deep_rl_zoo .

Michael Janner, Karthik Narasimhan, and Regina Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics , 6:49-61, 2018.

Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations , 2018.

B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick P´ erez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 23(6):4909-4926, 2022. doi: 10.1109/ TITS.2021.3054625.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 , 2022.

Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv , 2017.

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pretraining for unified vision-language understanding and generation. In International Conference on Machine Learning , pp. 12888-12900. PMLR, 2022.

Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint arXiv:2001.02407 , 2020.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rockt¨ aschel. A survey of reinforcement learning informed by natural language. arXiv preprint arXiv:1906.03926 , 2019.

Hongyuan Mei, Mohit Bansal, and Matthew R Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Thirtieth AAAI Conference on Artificial Intelligence , 2016.

So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. Film: Following instructions in language with modular methods. arXiv preprint arXiv:2110.07342 , 2021.

Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint arXiv:1704.08795 , 2017.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pp. 1928-1937. PMLR, 2016.

Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning. Journal of Artificial Intelligence Research , 63:849-874, 2018.

Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and-language navigation. arXiv preprint arXiv:2105.06453 , 2021.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pp. 2778-2787. PMLR, 2017.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 , 2021.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 , 2016.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 10740-10749, 2020a.

Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cˆ ot´ e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768 , 2020b.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. Reward is enough. Artificial Intelligence , 299:103535, 2021. ISSN 0004-3702. doi: https://doi.org/10.1016/j.artint.2021.103535. URL https://www.sciencedirect.com/science/article/pii/S0004370221000862 .

Adam Stooke and Pieter Abbeel. rlpyt: A research code base for deep reinforcement learning in pytorch. arXiv preprint arXiv:1909.01500 , 2019.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems , 34:251-266, 2021.

Oyvind Tafjord and Peter Clark. General-purpose question-answering with macaw. arXiv preprint arXiv:2109.02593 , 2021.

Michael Henry Tessler, Jason Madeano, Pedro A Tsividis, Brin Harper, Noah D Goodman, and Joshua B Tenenbaum. Learning to solve complex tasks by growing knowledge culturally across generations. arXiv preprint arXiv:2107.13377 , 2021.

HJ Wang and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. arXiv preprint arXiv:2101.07393 , 2021.

Sida I Wang, Percy Liang, and Christopher D Manning. Learning language games through interaction. arXiv preprint arXiv:1606.02447 , 2016.

Tsun-Hsuan Wang, Alexander Amini, Wilko Schwarting, Igor Gilitschenski, Sertac Karaman, and Daniela Rus. Learning interactive driving policies via data-driven simulation. In 2022 International Conference on Robotics and Automation (ICRA) . IEEE, 2022a.

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 6629-6638, 2019.

Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, et al. Language models with image descriptors are strong few-shot video-language learners. arXiv preprint arXiv:2205.10747 , 2022b.

Yue Wu, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Yuanzhi Li, Tom Mitchell, and Shrimai Prabhumoye. Plan, eliminate, and track-language models are good teachers for embodied agents. arXiv preprint arXiv:2305.02412 , 2023.

Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, and Fei Sha. Readtwice: Reading very large documents with memories. arXiv preprint arXiv:2105.04241 , 2021.

Victor Zhong, Tim Rockt¨ aschel, and Edward Grefenstette. Rtfm: Generalising to novel environment dynamics via reading. arXiv preprint arXiv:1910.08210 , 2019.

Victor Zhong, Austin W Hanjie, Sida I Wang, Karthik Narasimhan, and Luke Zettlemoyer. Silg: The multi-environment symbolic interactive language grounding benchmark. arXiv preprint arXiv:2110.10661 , 2021.

Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718 , 2023.

A Additional Details

A.1 Scale of Auxiliary Rewards

In practice, the R&R framework is quite robust to the scale of auxiliary reward since reward clipping has been implemented by most RL algorithms (Hu, 2022). Any reward in the range of (2,50) should result in the same behavior.

A.2 Atari Baselines

A2C/PPO. A2C (Mnih et al., 2016) and PPO (Schulman et al., 2017) are among some of the earliest Actor-Critic algorithms that brought success for deep RL on Atari games. A2C learns a policy π and a value function V to reduce the variance of the REINFORCE (Sutton et al., 1999) algorithm. The algorithm optimizes the advantage function A ( s, a ) = Q ( s, a ) -V ( s ) instead of the action value function Q ( s, a ) to further stabilize training.

MuZero. MuZero algorithm (Schrittwieser et al., 2020) combines a tree-based search with a learned model of the environment. The model of MuZero iteratively predicts the reward, the action-selection policy, and the value function. When applied to the Atari benchmarks, MuZero achieves super-human performance in a lot of games.

R2D2/R2D1. R2D2 (Kapturowski et al., 2018) proposes an improved training strategy for RNNbased RL agents with distributed prioritized experience replay. The algorithm is the first to exceed human-level performance in 52 of the 57 Atari games. R2D1 is a single-machine variant implemented by (Stooke & Abbeel, 2019).

Agent57. Badia et al. (2020) train a neural network that parameterizes a family of policies with different exploration, and proposes a meta-controller to choose which policy to prioritize during training. Agent57 exceeds human-level performance on all 57 Atari games.

A.3 Explanation of Game Objectives

Skiing. The player controls the direction and speed of a skier and must avoid obstacles, such as trees and moguls. The goal is to reach the end of the course as rapidly as possible, but the skier must pass through a series of gates (indicated by a pair of closely spaced flags). Missed gates count as a penalty against the player's time.

Tennis. The player plays the game of tennis. When serving and returning shots, the tennis players automatically swing forehand or backhand as the situation demands, and all shots automatically clear the net and land in bounds.

Pacman (Ms. Pac-Man). The player earns points by eating pellets and avoiding ghosts, which contacting them results in losing a life. Eating a 'power pellet' causes the ghosts to turn blue, allowing them to be eaten for extra points. Bonus fruits can be eaten for increasing point values, twice per round.

Breakout Using a single ball, the player must use the paddle to knock down as many bricks as possible. If the player's paddle misses the ball's rebound, the player will lose a life.

A.4 Outputs of QA Extraction Module

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRL has a notorious sample efficiency problem, and the authors propose to tackle this problem by having RL agents read instruction manuals. The authors propose a novel framework called Read and Reward (R&R), which is composed of two modules. The first module is QA which extracts and summarizes relevant information in the manual, and the second module is reasoning which evaluates events in the game and provides appropriate rewards for the RL agents to learn with algorithms such as A2C and Agent 57. The authors found R&R to significantly improve the agent's sample efficiency in several challenging games, such as Skiing.\n\nSTRENGTHS:\nThe paper is well-written and has the following strengths:\n\n* **Interetsing approch**: as the authors mentioned, this is the first work that demonstrates improving RL performance by reading instruction manuals. Many pure RL methods suffer significantly in Skiing, so seeing other approaches that improve training is encouraging.\n* **Further contribution to NLP + RL**: the authors further explore the space of using NLP to improve RL agents. This line of research will help reduce the need to train agents from scratch and provide useful priors to the agent.\n* **No labeling required**: the amount of human labor required in the proposed framework is minimal — they only provide some generic questions, such as \"what is the objective of the game?\"\n\nWEAKNESSES:\nDespite its strengths, the paper has the following weaknesses:\n\n* **Adhoc setting**: the setting seems ad hoc, stitching together various modules such as QA and object detection. A multi-modality model such as GPT-4 can probably serve as a general replacement for these modules, unlocking new capabilities, such as learning from image/video tutorials, in addition to instruction manuals. That said, there has not been an excellent open-source multi-modality model available. \n* **Slightly insufficient evaluation**: the authors only tested their approach on four games, but maybe it's worthwhile to evaluate some other games, such as Montezuma Revenge, which RL algorithms usually struggle with.\n\nQUESTIONS:\nI'm curious about the delayed reward schedule. Would it not artificially create a sparse reward problem? \n\n\nMisc:\n* The citation format seems slightly problematic. `However, in many real-world scenarios (Kolve et al., 2017)` feels more common and readable compared to `However, in many real-world scenarios Kolve et al. (2017);`\n* Lines 44 and 48 seem repetitive.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Read and Reward framework, where RL agents accelerate their learning of a new environment by interpreting user manuals. Specifically, the framework consists of a QA Extraction module that extracts and summarizes relevant information and a Reasoning module that evaluates object-agent interactions based on the information. The algorithm accelerates learning by using auxiliary rewards, whenever the interactions are detected by the reasoning module. The method empirically boosts the efficiency and performance of many Atari games.\n\nSTRENGTHS:\nThe topic in focus is very relevant because of the recent surge in the abilities of LLMs. The solution of utilizing LLMs to map auxiliary rewards in RL is novel. The performance of speeding up RL training for the four games investigated is apparent. The ablations performed are thorough and complete.\n\nWEAKNESSES:\n- document length and unstructuredness are overstressed in the paper as a method/contribution. However, a) many NLP papers have already addressed them, and the method employed in the paper is simple summarization; b) long context models are already available to ingest more than the text needed to reason\n- the overall idea is quite nice; however, due to the difficulty of grounding knowledge in the environment, the paper only considered \"hit\" interactions and required unsupervised object detectors (which might be unreliable) OR game states, and the interaction is quite primitive in many scenarios, therefore cannot describe most tasks/useful interactions.\n- Table 1 needs error bars/std\n\nQUESTIONS:\n- have you tried using VLMs to do the reasoning? instead of object detectors. This way the set of interactions can also be expanded\n- does the scale of the aux reward matter in the games investigated?\n- is there an intuition on the speedup ratio (Table 3) Vs. game? (Breakout appears to really benefit from R&R)\n\nLIMITATIONS:\nBoth limitations and broader impact sections are included in the draft", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Read and Reward (RR), a method to incorporate prior human knowledge about the environment to achieve performance and efficiency gains in RL environments. The paper instantiates RR in several Atari environments by reading the information from the instruction manual. A full end-to-end pipeline is demonstrated on Skiing and a pipeline without object detection is demonstrated on Tennis, Pacman, and Breakout.\n\nSTRENGTHS:\n*Originality*: While incorporating prior knowledge into RL training is a previously investigated topic, this paper provides an end-to-end pipeline and demonstrates its usefulness on Atari Skiing. The method introduced is more general than methods previously investigated.\n\n*Quality*: There are thorough experiments with ablation studies, such as training the QA module on a different set of instructions.\n\n*Clarity*: the paper is well-written.\n\n*Significance*: as RL agents are eventually deployed in real-world scenarios, it will be increasingly important to have them learn efficiently and incorporate prior knowledge, which is often in the form of text. This paper provides a complete demonstration of a method leveraging natural language data to improve performance. More generally, it sketches out a framework for exploiting external data that future algorithms may build upon. Such agents may be rapidly deployed to learn in an unsupervised fashion.\n\nWEAKNESSES:\nAlthough RR is an interesting proof-of-concept of automated reward shaping, it is unclear whether it scales to more complex environments or works on noisier sources of textual knowledge. In particular:\n1. Does RR generalize to noisier sources of textual data? While the Wikipedia ablation is interesting, the data itself is rather clean and contains similar information. One interesting experiment to try might be with text-based RL environments, such as the Jericho [1] or Machiavelli [2] environments. Oftentimes these environments have publicly available [walkthroughs online](https://forum.choiceofgames.com/t/guides-for-all-games/15569/4), although they are often incomplete and noisy, as the instructions are intended for humans. Is RR able to extract the rewards and guide the agents in this setting? (I realize that this would be a significant undertaking so am not expecting it in the rebuttal)\n2. The reward shaping only surrounds object detection. As a result, the QA prompt, even though it is hand-designed, still generalizes. However, there are several RL environments where the rewards are more heuristics rather than strict rules. For example, dialogue in the Diplomacy environment [3] does not have clear positive or negative reward but instead depends on the context. One could imagine gaining better understanding of dialogue in Diplomacy by leveraging knowledge on the internet, but it is unclear how RR might be applied in that scenario, as the reward shaping must center on text and the QA prompt likely could not be hand-designed.\n3. It seems like it was quite difficult to train a bounding-box detector for most games. However, this would imply that RR's capabilities are bottlenecked by limitations of such bounding-box detectors. This might make RR difficult to scale to more complex environments with more ambiguous objects.\n\n[1] Hausknecht, M., Ammanabrolu, P., Coté Marc-Alexandre, & Yuan Xingdi (2019). Interactive Fiction Games: A Colossal Adventure. CoRR, abs/1909.05398.\n[2] Pan, A., Chan, J., Zou, A., Li, N., Basart, S., Woodside, T., Ng, J., Zhang, H., Emmons, S., & Hendrycks, D. (2023). Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark. ICML.\n[3] Meta Fundamental AI Research Diplomacy Team (FAIR) et al. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science 378,1067-1074 (2022). DOI:10.1126/science.ade9097\n\nQUESTIONS:\n1. Would it be possible to have more results on other Atari games? I understand that the bounding-box detection was challenging, but are Tennis, Pacman, and Breakout the only other Atari games that allow for ground truth calculation of the objects from the RAM state?\n2. How robust is the automated reward shaping to the quality of the bounding box detector in Skiing?\n\nLIMITATIONS:\nWhile some limitations are addressed, it would be interesting if there were additional discussion about the limitations of extracting information from natural language data. For example, humans understanding of the physical world may not be adequately encoded in natural language. Does this limit the application scope of agents using the RR method? Additionally, if the LM's reasoning abilities are erroneous, will this irreparably break RR?\n\nFinally, it would be helpful to directly note that as RR is fundamentally a reward shaping method, it is difficult to apply it to situations where the reward itself is unclear. This happens often with delayed reward. In particular, RR would likely not improve performance on Go or Chess.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel method for Single Agent Reinforcement Learning which utilises computer game instruction manuals to enhance learning efficiency and performance. The Atari game manuals are used to accelerate RL algorithms in learning to play four different games. The framework comprises a Question-Answer Extraction module and a Reasoning module.\n\nThe QA Extraction module summarizes salient information from the manual. The Reasoning module evaluates in-game events using the extracted information from the manual and then assigns auxiliary rewards when such interactions are detected. These auxiliary rewards are provided to standard RL agents.\n\nThe paper's results show improvements in both performance and training speed for various RL algorithms when aided by this novel framework. It is asserted that this is the first successful attempt to use instruction manuals in a fully automated framework for solving the Atari RL benchmarks.\n\nIt is noted that even small open-source QA language models could effectively extract useful gameplay information from human-written documentation for RL agents.\n\nSTRENGTHS:\nThe results in this paper are certainly extremely promising, and the ability to use large language models to create additional rewards from the instruction manual is very promising. That such methods are so much more sample efficient is clearly going to be important in realms beyond simple game play.\n\nWEAKNESSES:\nThe main, and very problematic issue with this paper is the lack of robust evaluation. There are no statistics given, no averaging over random seeds provided and no code made available. It is thus entirely unclear how much the results themselves can be trusted under scrutiny. Thorough and transparent evaluation is absolutely vital in all RL scenarios, and because this has been left out completely, this paper is, I believe, not ready for publication.\n\nQUESTIONS:\nWithout a full analysis of the variation in results over a large number of random seeds, how can the results be trusted?\n\nLIMITATIONS:\nI believe that the limitations provided are fair.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-03"}