Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals
Authors: Yue Wu
Yewen Fan
Paul Pu Liang
Amos Azaria
Yuanzhi Li
Tom Mitchell
Venue: NeurIPS 2023 poster (2023)
Keywords: Games
Instruction Manual
Atari Games
Large Language Models
Language Models
Zero-shot
In-context prompting
TL;DR: Read and Reward speeds up RL algorithms on Atari games by reading manuals released by the Atari game developers

Abstract: High sample complexity has long been a challenge for RL. On the other hand, humans learn to perform tasks not only from interaction or demonstrations, but also by reading unstructured text documents, e.g., instruction manuals. Instruction manuals and wiki pages are among the most abundant data that could inform agents of valuable features and policies or task-specific environmental dynamics and reward structures. Therefore, we hypothesize that the ability to utilize human-written instruction manuals to assist learning policies for specific tasks should lead to a more efficient and better-performing agent. We propose the Read and Reward framework. Read and Reward speeds up RL algorithms on Atari games by reading manuals released by the Atari game developers. Our framework consists of a QA Extraction module that extracts and summarizes relevant information from the manual and a Reasoning module that evaluates object-agent interactions based on information from the manual. An auxiliary reward is then provided to a standard A2C RL agent, when interaction is detected. Experimentally, various RL algorithms obtain significant improvement in performance and training speed when assisted by our design. Code at github.com/Holmeswww/RnR

Read and Reap the Rewards:
Learning to Play Atari with the Help of Instruction Manuals
Yue Wu'r, Yewen Fan', Paul Pu Liang', Amos Azaria?, Yuanzhi Li Tom Mitchell'
CCarnegie Mellon University, 2 AAiiel University, Microsoft Research
"ywu5 @ andrew.cmu.edu
Abstract
High sample complexity has long been a challenge for RL. On the other hand, humans learn to perform tasks not only from interaction demonstrations, but also by reading unstructured text documents, e.g., instruction manuals. Instruction manuals and wiki pages among the most abundant data that could inform agents of valuable features and policies or task-specific environmental dynamics and reward structures. Therefore, hypothesize that the ability to utilize human- written instruction manuals to assist learning policies for specific tasks should lead to more efficient and better-performing agent. We propose the Read and Reward framework. Read and Reward speeds up RL algorithms on Atari games by reading manuals released by the Atari game developers. Our framework consists of QA Extraction module that extracts and summarizes relevant information from the manual and Reasoning module that evaluates object-agent interactions based on information from the manual. An auxiliary reward is then provided to standard A2C RL agent, when interaction is detected. Experimentally, various RL algorithms obtain significant improvement in performance and training speed when assisted by our design. Code github:com/Hommmwww//ktt
Introduction
Reinforcement Learning (RL) has achieved impressive performance variety of tasks, such as Atari (Mnih et al. 22055) Badia et al. 2020), Go (Silver et al. (017), or autonomous driving (Fuchs et al.
2021 et al. 2022), and is hypothesized to be an important step toward artificial intelligence
(Silver et al. 2021). However, RL still faces great challenge when applied to complicated, real- world-like scenarios (Shridhar al. (020b)] [Szot et al. (02T)) due to its low sample efficiency. The Skiing game in Atari, for example, requires the skier (agent) to ski down snowy hill and hit the objective gates in the shortest time, while avoiding obstacles. Such an intuitive game with simple controls (left, right) still required 80 billion frames to solve with existing RL algorithms (Badia et al.) 2020). roughly equivalent to 100 years of non-stop playing.
Observing the gap between RL and human performance, we identify an important cause: the lack of knowledge and understanding about the game. One of the most available sources of information, that is often used by humans to solve real-world problems, is unstructured text, e.g., books, instruction manuals, and wiki pages. addition, unstructured language has been experimentally verified as powerful and flexible source of knowledge for complex tasks (Tessler et al.) (021). Therefore, we hypothesize that RL agents could benefit from reading and understanding publicly available instruction manuals and wiki pages.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

GEECIAL FRATIRES CLAUAR 30CING fastes Lime.
010075
dbjecss
through
Detection
Sobdangt-.. Klspset time on timer. Toe penslived five seconds euch gate and your total time lleclading peralsine, if any are shown on the screen.
Grounding
if you hit sate or a firett de not 1000 hearts you (ccior] will yeup baok up and teep going. sot you Losn time, to carefell
The red luctoon on your Joystiok is not used in slaion coving, lut can be usel inaseed of the gane reses exiich to reest your skter as the sop of the ocaree.
skier
no wake von sougher, pook your risht diffiouiny switch up so now sreee have
pole
aide), potos.
appear
then and ssill all the
Read
QA Extraction Module
Arnower You panaliced seconch wach
Qutetin:: what happens when player poll??
Quention: lios sacowed the
the
god
Answer: You/ lose time.
Queetion: What happens when player hits tree?
Arsiaan Learning control the of your skis and antidputing and aviddng troubie.
Queertion: kio score ot game?
Arsawar: Eapadd time at the on men
Reasoning Module
Bounding box internec?
Queet on Should you hits gate yoou want win?
Arswer: Yes
Audillary Roward (player, geta] [playor;
Questio:: Should you troe if you want win?
Reward
Figure 1: An overview of our Read and Reward framework. Our system receives the current frame in the environment, and the instruction manual input. After object detection and grounding. the QA Extraction Module extracts and summarizes relevant information from the manual, and the Reasoning Module assigns auxiliary rewards to detected in-game events by reasoning with outputs from the QA Extraction Module. The "Yes/No" answers are then mapped to +5/- auxiliary rewards.
In this work, we demonstrate the possibility for RL agents to benefit from human- human-written written instruction manuals. We download per-game instruction manuals released by the original game developers or Wikipedia and experiment with 4 Atari games where the object location ground-trutee truttss are available.
Two challenges arise with the use of human- written instruction manuals: 1) Length: Manuals contain a lot of redundant information and are too long for current language models. However, only a few paragraphs out of the 2-3 page manuals contain relevant information for any specific aspect of concern, i.e., the game's 's objective or a specific interaction. 2) Reasoning: References may be implicit and attribution may require reasoning across paragraphs. For example, in Skiing, the agent needs to understand that hitting an obstacle is bad using both "you lose time if the skier hits an obstacle" and "the objective is to finish in shortest time", which are from different parts of the manual. Furthermore, the grounding of in-game objects to references in the text remains an open problem and manuals may contain significant syntactic variation. For example, the phrase "don't 't hit an obstacle" is also referred to as "don't get wrapped around a tree" or "avoid obstacles". Finally, the grounding of in-game features to references in the text remains an open problem (Luketina al. 2019
Our proposed Read and Reward framework (Figure I addresses the challenges with a two-step process. First, zero-shot extractive QA module is used to extract and summarize relevant information from the manual, and thus tackles the challenge of Length. Second, a zero-shot reasoning module, powered by a large language model, reasons about contexts provided by the extractive QA module and assigns auxiliary rewards to in- game interactions. After object detection and localization, our system registers an interaction based the distance between objects. The auxiliary reward from the interaction can be consumed by any RL algorithm.
Even though we only consider "hit" interactions (detected by tracking distance between objects) in our experiments, Read and Reward still achieves 60% performance improvement on a baseline using 1000x fewer frames than the SOTA for the game of Skiing. In addition, since Read and Reward does not require explicit training on synthetic text (Hermann et al. [0117] Chaplot et al. 2018 Janner et al.)
2018 Narasimhan al., 2018 Zhong et al. 2019 2021 Wang & Narasimhan, 2021 we observe consistent performance on manuals from two different sources: Wikipedia and Official manuals.
To our knowledge, our work is the first to demonstrate the capability to improve RL performance in an end-to-end setting using real-world manuals designed for human readers.
atariage.com

---

2 Background
2.1 Reducing sample complexity in RL
Prior works have attempted to improve sample efficiency with new exploration techniques (Schulman man
et arl.7,2217] |Haarojj et al. 2018 [Bdia et al. 2020), self-supervised objectives (Pathak et al. 2017 Schrittwieser et al., 2020p. and static demonstration data (Kumar et al., 2019 2020). Fan & Xiao (2022) cast RL training into a training data distribution optimization problem, and propose a policy mathematically controlled to traverse high-value and non-trivial states. However, such a solution requires manual reward shaping. Currently, the SOTA RL agent (Badia al.) 2020) still spends billions of frames to learn the Skingg game that can be mastered by humans within minutes.
2.2 Grounding objects for control problems
Most prior works study grounding associating perceived objects with appropriate text in the setting of step-by-step instruction following for object manipulation tasks (Wang et al. 2016 Bahdanau et al.
2018) or indoor navigation tasks (Chaplot et al., 2018, Janner et al. [018 Chen et al. 2019 SSridhar ct al. 2020a). The common approach has been to condition the agent's policy on the embedding of both the instruction and observation (Mei et al. 2016 Hermann et al., 2017 Janner et al. 2018 Misra iet al. [017] Chen et al., 2019; Pashevich et al. 2021). Recently, modular solutions have been shown to be promising (Min et al. (022). However, the majority of these works synthetic language. These often take the form of templates, e.g.. "what colour is <object> in <room>" "Chevalier-Boiveer jet al. (018). On the other hand, recent attempts large-scale image-text backbones, e.g., CLIP (Radford et al. 2021) have shown robustness at different styles visual inputs, even on sketches. Therefore, we pick CLIP our model for zero-shot grounding.
2.3 Reinforcement learning informed by natural language
In the aagguage-assister setting, step-by-step instructions have been used to generate auxiliary rewards, when environment rewards are sparse. Goyal et al. (2019); Wang et (2019) use auxiliary reward-learning modules trained offline to predict whether trajectory segments correspond natural language annotations of expert trajectories.
In more general setting, text may contain both information about optimal policy and environment dynamics. Branavan et al. (2012) improve Monte-Cart tree search planning by accessing a natural language manual. They apply their approach to the first few moves of the game Civilization turn-based strategy game. However, many of the features they use are handcrafted to fit the game of Civilization II, which limits the generalization of their approach.
Narasimhan et al. (2018); Wang & Narasimhan (20TT) investigate planning in a 2D game environment with fixed number of entities that annotated by natural language (e.g. the 'spider' and 'scorpion' entities might be annotated with the descriptions "randomly moving enemy" and "an enemy who chases you", respectively). The agent learns to generalize new environment settings by learning the correlation between the annotations and the environmental goals and mechanics. The proposed agent achieves better performance than the baselines, which do not use natural language. However the design of the 2D environment uses 6-line descriptions created from templates. Such design over- simplifies the observations into 2D integer matrix, and removes the need for visual understanding and generalization to new objects and new formats of instruction manual, e.g., human written manuals.
2.4 RL Models that Read Natural Language Instructions
Zhong et al. (2019 I make use of special architectures with Bidirectional Feature- Linear Modulation Layers that support multi-hop reasoning on 6 x 6 grid worlds with instruction manuals. However, the model requires 200 million training samples from templates identical to the test environments. Such a training requirement results in performance loss even on 10 10 grid worlds with identical mechanics, thus limiting the generalization of the model.
Wang & Narasimhan 2021 mixes embedding of entities with multi-modal attention to get entity representations, which are then fed to CNN actor. The attention model and CNN actor are trained jointly using RI. However, the whole framework is designed primarily for a grid world with template generated instruction manual consisting of one line per entity. In experiments, we find
3

---

Dueetion: What the objective of the game?
Arswer The points esting pellets avolding ghosts.
Extractive
Duestion: How
Arswer
exting pollets avolding ghosts.
Question: How
pollets avoidina shoets
Context
Ducstion: What hapeere when player his point?
point
Arswer N/A.
pacman
pellet
Extractive QA with
Dueition: happara player ghas?? gets gobbled up
ghost
plll
Question
est
Templete
Dueetion: gobiles them up
his
lessen Figure 2: Illustration of the QA Extraction Module on the game PacMan. We obtain generic information about the game by running extractive QA on generic questions (3 shown since one question did not have an answer). We then obtain object-specific information using question template. We concatenate generic and object-specific information to obtain the <context> string.
that extracting entity embeddings from human- Atari instruction manuals following Wang & Narasimhan (202T) does not improve the performance of our vision-based agent due to the complexity and diversity of the language.
3 Read and Reward
We identify two key challenges to understanding and using the instruction manual with an RL agent.
The first challenge is Length. Due to the sheer length of the raw text (2 ~ 3 pages), current pre-trained language models cannot handle the raw manual due to input length constraints. Additionally, most works in the area of long- docummntt reading and comprehension (Beltagy et al. 20200 Ainslie et al., 2020, Zemlyanskiy et al., 202T) require fine-tuning signal from the task, which impractical for RL problems since the RL loss already suffers from high variance. Furthermore, an instruction manual may contain a lot of task-irrelevant information. For example, the official manual for MsPacman begins with: "MS. PAC-MAN and characters are trademark of Bally Midway Mfg. Co. sublicensed to Atari, Inc. by Namco-America, Inc."
It is therefore intuitive in our first step to have a QA Extraction Module that produces a summary directed toward game features and objectives, to remove distractions and simplify the problem of reasoning.
The second challenge is Reasoning. Information in the manual may be implicit. For example, the Skiing manual states that the goal is to arrive at the end in the fastest time, and that the player will lose time when they hit tree, but never states directly that hitting tree reduces the final reward. Most prior works (Eisenstein et al., 2009: Narasimhan et al. 2018 Wang & Narasimhan 2021 Zhong al.) 2019 2021) either lack the capability of multi-hop reasoning or require extensive training to form reasoning for specific manual formats, limiting the generalization to real-world manuals with a lot of variations like the Atari manuals.
On the other hand, large language models have achieved success without training in a lot of fields including reading comprehension, reasoning, and planning (Brown et al. 2020 Kojima al. 2022 Ahn et al., (022). Therefore, in the Reasoning Module, we compose natural language queries about specific object interactions in the games, to borrow the in-context reasoning power of large language models.
The rest of the section describes how the two modules within our Read and Reward framework (a QA extraction module and a reasoning module) are implemented.
QA Extraction Module (Read) order to produce a summary of the manual directed towards game features and objectives, we follow the extractive QA framework first proposed by Devlin et al. (2018), which extracts a text sub-sequence as the answer to a question about a passage of text. The extractive QA framework takes raw text sequence Smanual Squestion as input. Then for each token (word) w² in Smamul-- the model predicts the probability that the current token is the start token Plun and end token Pena with a linear layer on top of word piece embeddings. Finally, the output Sanswer = (w.tart :..., woeud) selected as a sub-sequence of Smanual that maximizes the overall probability of the start and end: w start aad arg max wnnd start p
{w") ..., w²} and a question string

---

Implementation-win wise, we use a RoBERTa-lare model (Liu et al. 2019) for Extractive QA fine-tuned on the SQUAD dataset (Rajpurkar et al. 2016). Model weights are available from the AllenNLP A To handle instruction manual inputs longer than the maximum length accepted by RoBERTa, we split the manual into chunks according the max-token size of the model, and concatenate extractive QA outputs on each chunk.
As shown in Figure 2 we compose a of generic prompts about the objective of the game, for example, "What is the objective of the game?". In addition, to capture information on agent-object interactions, we first identify the top 10 important objects using TFIDF. Then for each object, we obtain an answer to the generated query:"Whal happens when the player hit <object>>? ?"
Finally, concatenate all non-empty question-answer -nnserr pairs per interaction into <context>> string. Note that the format of context strings closely resembles that of a QA dialogue.
Zero-shot reasoning with pre-trained QA model (Reward) The summary <context>> string from the QA Extraction module could di-
Question: What the objective the game?
Answer: The player points by eating pellets and awodding ghosts.
rectly be used as prompts for reasoning through
Question:
large language models (Brown et al., 2020; Ko-
Answer: player earns points by eating peliets avaiding ghosts.
Quastion:
game?
jima et al. 2022 Ahn et al.] 2022).
Answer: The player carns points by eating pellets and avoiding ghosts.
Loggers seooan
Answer: She gets gobbled
Quesiion: What happens when player hiss ghost?
Motivated by the dialogue structure of the sum- mary, and the limit in computational resources, we choose Macaw (Tafjord & Clark, (021). general-purpose zero-shot QA model with per- formance comparable to GPT-3 (Brown et al. 2020). Compared to the RoBERTa extrac- tive QA model for the QA Extraction module, Macaw is significantly larger and is better suited for reasoning. Although we find that GPT-3 generally provides more flexibility and a better explanation for its answers for our task, Macaw faster and open-source.
you ghost you
Large Language Model
Answer:
Rewand: 5
Reward: :5
Figure 3: Illustraiion of the Reasoning Module. The <context>> (from F related to the ob- ject ghost from the QA Extraction module is con- catenated with template- aeabaategeerrtedd question to form the zero-shot in-context reasoning prompt for a Large Language Model. The Yes/No answer from the LLM is then turned into auxiliary re- ward for the agent.
As shown in Figure 3 for each interaction we compose the query prompt as "<context>> Ques- tion: Should you hit <object of interaction> if you want to win? Answer: " and calculate the LLM score choices {Yes, No). The Macaw model directly supports scoring the options in a multiple-choice setting. We note that for other generative LLMs,
the probability of "Yes" vs. "No" as scores, following Ahnn et al. (0022).
Finally, during play. a rule-based algorithm detects 'hit' events by checking the distance between bounding boxes. For detected interactions, auxiliary reward is provided to the RL algorithm according to (Yes, No} rating from the reasoning module (see Section 4.4 for details).
4 Experiments
4.1 Atari Environment and Baselines
The Openai gym Atari environment contains diverse Atari games designed to pose a challenge for human players. The observation consists of single game screen (frame): 2D array of 7-bit pixels, 160 pixels wide by 210 pixels high. The action space consists of the 18 discrete actions defined by the joystick controller. Instruction manuals released by the original game developers have been scanned and parsed into publicly available HTML forma For completeness, we also attach an explanation of the objectives for each game in Appendix A.3
AllenNI. Transformer- QA
atariage.com
5

---

4.2 Delayed Reward Schedule
Most current Atari environments include dense reward structure, i.e., the reward is obtained quite often from the environment. Indeed, most RL algorithms for the Atari environments perform well due to their dense reward structure. However, in many real-world scenarios (Kolve al., 2017 Wang er al.] (022a) or open-world environments (Fan et al. 2022), it is expensive to provide a dense reward. The reward is usually limited and "delayed" to a single positive or negative reward obtained the very end of an episode (e.g., once the robot succeeds or fails to pour the coffee).
Notably, Atari games with the above "delayed" reward structure, such as Sking, pose great challenges for current RL algorithms, and are among some of the hardest games to solve in Atari (Badia et al. 2020). Therefore, better align with real-world environments, the tennis, breakout, and Pac-Man games by providing the reward (the "final game score") only at the end of the game. This delayed reward structure does not change the optimization problem, but is more realistic and imposes additional challenges to RL algorithmss
use delayed reward version of
4.3 Grounding Objects in Atari
3.9
2
We find grounding (detecting and relating visual objects to keywords from TFIDF) a significant challenge to applying framework. In some extensively studied planning/control problems, such indoor navigation, pre- pre-traine trained models such as masked RCNN reasonably solve the task of grounding (Shridhar et al., 2020a). However, the objects in the Atari environments are too different from the training distribution for vi- sual backbones like mask RCNN. Therefore, the problem becomes very challenging for the do- main of Atari. Since it is not the main focus of our proposed work, we attempt to provide an unsupervised and-to-end solution only for the game of Skiing.
Full end-to-end pipeline on Skiing with A2C To demonstrate the full potential of our work, we offer a proof-of-concept agent directly op- erating raw visual game observations and the instruction manual for the game of Skiing use
Figure 4: Examples of SPACE (Lin et 2020 and CLIP (Radford et al., 2021 in the full end- to-end pipeline (Section 4.3). The top row shows bounding boxes for objects and the bottom row shows corresponding object masks as detected by SPACE. Most of the bounding boxes generated are correct. Left: SPACE confuses bounding boxes of agent and tree into and the box gets classified as "tree" (bluc), and the auxiliary penalty is not properly triggered. Right: The flag next to the agent (in red circle) is not detected, and therefore the auxiliary reward is not provided.
For unsupervised object localization, SPACE, a method that uses spatial attention and decomposition to detect visual objects (Lin et al.) 2020). SPACE is trained on 50000 random ob- servations from the environment, and generates object bounding boxes. These bounding boxes are fed CLIP, zero-shot model that can pair images with natural language descriptions (Rad- ford et al. 2021), to classify the bounding box into categories defined by key-words from TF-IDE. The SPACE/CLIP models produce mostly reason- able results; however, in practice we find that CLIP lacks reliability over time, and SPACE cannot distinguish objects that cover each other. Therefore, to improve classification reliability, use Multiple Object Tracker (Bewley et al. 2016) in conjunction with CLIP and set the class of the bounding box to be the most dominant class over time.
Ground-truth object localization/grounding experiments For other games (Tennis, Ms. Pac- Man, and Breakout), we follow Anand et al. (2019) and directly calculate the coordinates of all game objects from the simulator RAM state. Specifically, we first manually label around frames with object name and X,Y coordinates and then train a linear regression model mapping from simulator RAM state to labeled object coordinates. Note that this labeling strategy fails for games in which multiple objects of the same kind appears, i.c., multiple trees appear in frame of Skiing.

---

A2C+R&R
A2C.
PPO+R&R
PPO
R2D1+R&R
R2D1
Agent57+R&R
Agent57
Tennis
-5 (-5)
-23
(-6)
-23
(-I)
-23
(
-23
Pacman
580 (580)
452
253
(253)
204
3115 (3115)
2116
1000 (1000)
708
Breakout
14 (14)
2
9
2
10 (10)
2
38 (38)
2
Table Table of the game score of different RL. algorithms trained under delayed reward schedule,
RL with Read and Reward consistently outperforms their counterparts that are not using the manual. In addition, Read and Reward performance remains consistent across official Atari manual and Wikipedia in brackets "())"
A2C+R&R
A2C
PPO+R&
PPO
R2D1+R&R
R2D1
Tennis
- 5.2
23 + 1.3
-8.1 2.3
223 - 1.0
3.1
- 23.0 = 1.3
Pacman
455.2 2..2+66..4 63.2
387.1
+ 66.2
2884.
67.
200.1
65.4
65.4
3001.3
203.2
1999.2 2 + 109.2
Breakout
-
2.1 111:0. 0.3
0.1
1.9 0.5
10
2.1 +
Table 2: Table of the game score of different RI. algorithms trained under delayed reward schedule
on the official Atari manual, with statistics computed over random seeds. RL with Read and Reward consistently outperforms their counterparts that are not using the manual.
4.4 Simplifying Interaction Detection to Distance Tracking
We make the assumption that all object interactions in Atari can be characterized by the distance between objects, and we ignore all interactions that do not contain the agent. We note that since the Atari environment in 2D, suffices to consider only interactions where objects get close to the agent so that we can query the manual for "hit" interaction.
We, therefore, monitor the environment frames and register interaction whenever object's s bounding boxes intersect with the agent. For each detected interaction, we assign rewards from { {-Tn. accordingly as detailed in Section Experimentally, we find that it suffices to set Tn ryp 5.
4.5
RL Baselines
107
Since Read and Reward only provides an aux- iliary reward, it is compatible with most RL frameworks such A2C, PPO, SAC, TD3.
Frames Seen (Efficiency)
10a
and Agent 57 (Mnih et al.] 2016 Schul-
man al.; [201]] Haarnoja et al. 2018
10a
Fujimoto et al., 2018; Badia et al. 2020).
We use popular open-source implementations
A2C,PPO.R2D1 (Stooke & Abbeel, 2019), and
10w0
Agent57 Hu 2022).
kaedion Aone
M::2mm
Toac and Rawwrdd (Dur))
Agari.57
104h
(Buudd
4.6 Full
-30000
-25000
-20000 -15000
10000
-5000
Game Score (Performance)
Pipeline Results on Skiing with A2C
Figure 5: Performance (in Game Score) vs Effi- ciency (in Frames Scen) of fully end-to-end Read and Reward (green) compared to an A2C base- line, and MuZero (Schrittwieser al., 2020) Skiing. Benefiting from the auxiliary rewards from the instruction manual, Read and Reward out-performs the A2C Baseline and achieves 60% improvement on random performance while using much fewer training frames compared to the SOTA mixed- mixed-policy Agent 57 (Badia al. 2020 and GDI-H3 (Fan & Xiao 2022).
We compare the performance of our Read and Reward agent using the full pipeline to baselines in Figure [5] our agent is best on the efficiency axis and close to best on the performance axis. A2C equipped with Read and Reward outper- forms A2C baseline and more complex methods like MuZero (Schrittwieser et al. 2020). We note 50% performance gap between and the SOTA Agent 57, but our solution does not require policy mixing and requires more than 10 times fewer frames to converge to perfor- mance 60% better than random.
agent
When experimenting with a full end-to-end pipeline as described in Section 4.3 we notice instability with the detection and grounding models. As shown in Figure missing bounding boxes cause lots of mislabeling when objects are close to each other. Such errors may lead to the auxiliary reward being incorrectly assigned.
7

---

Game
Wikipedia
Official Manual
What is the objective of the game?
To score many points as you can prac- tice clearing the maze of dots before try- ing to gobble up the ghosts.
The player earns points by eating pellets and avoiding ghosts.
How to succeed in the game?
The player earns points by eating pellets and avoiding ghosts.
Pacman
Score as many points as you can.
How to score at the game?
The player earns points by eating pellets and avoiding ghosts.
N/A
Who are your enemies?
Ghosts. stay close to an energy pill be- fore eating it, and tease the ghosts.
N/A
Table 3: Table showing the outputs of the QA Extraction module on Wikipedia instructions vs the official Atari manual. The Wikipedia manual is significantly shorter, and contains less information, causing the extractive QA model to use the same answer for all questions. Full table for all games is shown in Table S in the Appendix.
Although Skiing is arguably one of the casiest games in Atari for object detection and grounding, with easily distinguishable objects and white backgrounds, we still observe that current unsupervised object detection and grounding techniques lack reliability.
4.7 Results on Games with Delayed Reward Structure
Since the problem of object detection and grounding is not the main focus of this paper, we obtain ground-truth labels for object locations and classes as mentioned in Section 3 In ad- dition, for a setting more generalizable to other environments (Kolve et al.) 2017 Fan et al.) 2022), we implement delayed reward sched- ule (Section (.2) for Tennis, MsPacman, and Breakout. Various RL frameworks benefit sig- nificantly from Read and Reward, as shown in
Read Fanard Dans AZC Suoe inal
40
Came
Deverd
400
Table 2
AZC
200
We plot the Game Score and the auxiliary re- wards of A2C with Read and Reward v.s. the A2C baseline in Figure 6 The A2C baseline fails to learn under the sparse rewards, while the performance of the Read and Reward agent continues to increase with more frames seen. In addition, we observe that the auxiliary reward from the Read and Reward framework has strong positive correlation with the game score (game reward) of the agent, suggesting that the A2C agent benefits from optimizing the auxil- iary rewards at training time.
100
Garra Score
Aud Revard
420
faed und noverr (Dar))
ACC the Insd
300
-150
-155
2.8
1.2
2.0 1.0
Planes Sean
Figure 6: Comparison of the training curves of Read and Reward + A2C v.s. A2C Baseline in terms of game score alongside Auxiliary Reward Frames Seen under our delayed reward setting for different Atari games. Read and Reward (red) consistently outperforms the A2C baseline (brown) in all games. In addition, the auxiliary reward from Read and Reward demonstrates strong positive correlation with the game score, suggesting that the model benefits from optimizing the auxiliary rewards at training time.
4.8 Results without Delayed Reward
Without delayed reward, Read and Reward does not offer performance improvement in terms of game score, but still improves training speed. As shown Table 4 Agent57 with Read and Reward requires up to 112X less training steps to reach the same performance as the Agent57 baseline without the instruction manuals.
8

---

4.9 R&R Behavior on Manuals from Different Source
To illustrate the generalization of proposed framework different source of information, download the "Gameplay" section from Wikipedia for each game and feed the section as manual to our model. In Table 3 we show comparison of the answers on the set of 4 generic questions on the Pacman by our QA Extraction Module. Note that the QA Extraction module successfully extracts all important information about avoiding ghosts and eating pellets to get points. In addition, since the official manual contains more information than the Gameplay section of Wikipedia (2 pages v.s. less than lines), we were able to extract much more information from the official manual.
Due to the high similarity in the output from the QA Extraction Module, the Reasoning module demonstrates good agreement for both the Wiki manual and the official manual. Therefore, we observe consistent agent performance between the official manual and the Wikipedia manual in terms of game score (Table I..
Tennis
Pacman
Breakout
steps reach base- line le6
6e5
5.1c5
8911
speed-up ratio
1.67
112
Table 4: Table showing the number of training steps required by Agent57 with Read and Reward (Official) to reach the same performance as the Agent57 baseline at 1e6 training steps and the speedup ratio, under delayed-reward.
5 Limitations and Future Works
One potential challenge and limitations is the requirement of object localization and detection as mentioned Section 4.3] However, since Read and Reward does not need to detect all objects to assign reward, i.e., energy pills not grounded in the game Paeman but Read and Reward is still
achieves improvement. In addition, reliable backbone object detection models (He et al.]
(017) have
already shown success in tasks such indoor navigation (Shridhar et al. (022b) and autonomous driving. With recent progress on visual- languag models (Bubeck et al.] 2003; Li et al., 2022] Driess
al. 2023 Liu et al. 2023) Zou al. 2023), we believe that our proposed framework can benefit from reliable and generalizable grounding models in the foreseeable future.
made in Section 4.4 is to consider only interactions where objects get sufficiently close to each other. While such simplification appears to be sufficient in Atari, where observations are 2D, one imagine this to fail in 3D environment like Minecraft (Fan al.) 2022) or an environment where multiple types of interaction could happen between same objects (Kolve al., 2017). Future works would benefit by grounding and tracking and interactions following recent works on video scene understanding Wang et al. (2022h) or game progress tracking Wu et al. (2023).
Another simplification
types of events
Finally, the performance of Read and Reward may be limited by information within the natural language data it reads.
Conclusions
In this work, we propose Read and Reward, method that assists and speeds up RL algorithms on the Atari challenges by reading downloaded game manuals released by the Atari game developers. Our method consists of QA extraction module that extracts and summarizes relevant information from the manual and a reasoning module that assigns auxiliary rewards to detected in-game events by reasoning with information from the manual. The auxiliary reward is then provided to standard RL agents. To our knowledge, this work is the first successful attempt for using instruction manuals in fully automated and generalizable framework for solving the widely used Atari RL benchmark (Badia et al. 2020).
Our results suggest that even small open-source QA language models could extract meaningful gameplay information from human- documentations for RL agents. With recent break-trough in zero-shot language understanding and reasoning with LLMs (Bubeck et al. (203), we hope that our work could incentivize future works on integrating human prior knowledge into RL training.
wikipedia.org

---

Broader Impacts
Our research using instruction manuals for RL could lead to both positive and negative impacts. The benefits include more efficient RL agents that could solve more real-world problems. The risks may involve more game cheating and exploitation, over reliance on the instruction manuals, and potential job loss.
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not say: 2022.
Grounding language in robotic affordances. arXiv preprint arXiv:2204.0160
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham. Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs transformers. arXiv preprint arXiv:2004. 084483, 2020.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. arXiv preprinl arXiv:1906.082: 2019.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507-517. PMLR, 2020.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprint 11t6::1806.19444 2018.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05 2020.
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP), pp. 3464-3468. IEEE, 2016.
SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in a monte-carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 2005v:2005.1440 2020.
Sébastien Bubeck, Varun Chandrasekaran. Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 23Xi::2303.1277 2023.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural
language navigation and spatial reasoning in visual street environments. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12538-12547, 2019.
Maxime Chevalier- Boisverr, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform study the sample efficiency of grounded language learning. arXiv preprint 1rXiv:1810.082 2018.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:/810.048800 2018.
Danny Driess, Fei Xia, Mehdi SM Sajadii Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et Palm-e: embodied multimodal language model. arXiv preprint MrXiv:3303.03374 2023.
10

---

Jacob Eisenstein, James Clarke, Dan Goldwasser, and Dan Roth. Reading to learn: Constructing features from semantic abstracts. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pp. 958-967, 2009.
Jiajun Fan and Changnan Xiao. Generalized data distribution iteration. arXiv preprint arXiv:2206.03192 1992.. 2022.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang. De-An Huang, Yuke Zhu, and Anima Anandkumar, Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint urXiv:2206.08855 2022.
Florian Fuchs, Yunlong Song, Elia Kaufmann, Davide Scaramuzza, and Peter Dürr. Super-human per- formance gran turismo sport using deep reinforcement learning. IEEE Robotics and Automation Letters, (33):4257-4264 2021. doi: 10.1009/RA.222) 10.11099LRA2222133666488 33664424...
Scott Fujimoto. Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Prasoon Goyal, Scott Nickum, and Raymond Mooney. Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv: 2019.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 2018.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in simulated 3d world. arXiv preprint (1t.:v:770600555500 2017.
Michael Hu. Deep RL Zoo: A collections of deep rl algorithms implemented with pytorch, 2022.
URL httpp:///gtt thub. http//icchaennnyydeeeeee ottpo//ooo/acccceeeee//eeeeeeoop _r00
Michael Janner. Karthik Narasimhan, and Regina Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics, 6:49-61, 2018.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018
Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: survey, IEEE Transactions on Intelligent Transportation Systems, 23(6):4909-4926 2022. doi: 10.1
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shor shot reasoners. arXiv preprint arXiv:2205. 2022.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual Al. arXiv, 2017.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 139-11991 2020.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888-1221 12900. PMLR, 2022.

---

Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint arXiv:2001.0000 102407, 2020.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0553 054499, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. survey of reinforcement learning informed by natural language. arXiv preprint MrXivv1906. 1906.03966, 2019.
Hongyuan Mei, Mohit Bansal, and Matthew R Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. Film: Following instructions in language with modular methods. arXiv preprint 10.00221 2021.
Dipendra Misra, John Langford, and Yoav Artzi Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint arXiv:1704.0879 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 51 ((755))):529- 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-1937. PMLR, 2016.
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning. Journal of Artificial Intelligence Research, 63:849-874, 2018.
Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for iiioo--andllagguage navigation. arXiv preprint 1rXiv:2105.06453, 2021.
Deepak Pathak, Pulkit Agrawal, Alexci A Efros, and Trevor Darrell. Curiosity-driven exploration by self-suuervised prediction. In International conference on machine learning, pp. 2778-2781 PMLR, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 00020, 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprchension of text. arXiv preprint arXiv:1606. 05250, 2016.
Julian Schritwwesser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-609, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (00.:007700637777 2017.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020a.
12

---

Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, arXiv preprint arXiv:2010.03768, 2020b.
David Silver, Julian Schrittwisser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(767))354-359) 2017.
David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. Reward is enough. Artificial Intelligence, 199:5, 2021, ISSN 0004-3702, doi:
10tt:///rr000000.........00000.0 URL https: // www. wwwesscceeeeeeseeeeeeeeeeeeeeeee com http:///seee.aaat..t////////////
Adam Stooke and Pieter Abbeel. rlpyt: A research code base for deep reinforcement learning in pytorch. arXiv preprint arXiv: 1900001500, 2019.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34:251-266 2021.
Oyvind Tafjord and Peter Clark. General-purpor question-answern with macaw. arXiv preprint arXiv:2/09.02550, 2021.
Michael Henry Tessler, Jason Madeano, Pedro A Tsividis, Brin Harper, Noah D Goodman, and Joshua B Tenenbaum. Learning to solve complex tasks by growing knowledge culturally across generations. arXiv preprint arXiv:2107.13377 2021.
HJ Wang and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. arXiv preprint arXiv:2101.07393 2021.
Sida Wang, Percy Liang, and Christopher D) Manning. Learning language games through interaction.
arXiv preprint arXiv: wte.:/6666.4444 2016.
Tsun-Hsuan Wang, Alexander Amini, Wilko Schwarting, Igor Gilitschenski, Sertac Karaman, and Daniela Rus. Learning interactive driving policies via data-driven simulation. In 2022 International Conference on Robolics and Automation (ICRA). IEEE, 2022a.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6629-6638, 2019.
Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, et al. Language models with image descriptors are strong few-shot langulan learners. arXiv preprint arXiv:2205.10747, 2022b.
Yue Wu. So Ycon Min. Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria. Yuanzhi Li, Tom Mitchell, and Shrimai Prabhumoye. Plan, eliminate, and language models are good teachers for embodied agents. arXiv preprint arXiv:2305.024122 2023.
Yury Zemlyanskiy. Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, and Fei Sha. Readtwice: Reading very large documents with memories. arXiv preprint 105442100,.02411, 2021.
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to novel environment dynamics via reading. arXiv preprint /910.0/9/0.0222 2019.
Victor Zhong, Austin W Hanjie, Sida I Wang, Karthik Narasimhan, and Luke Zettlemoyer. Silg: The multi-environment symbolic interactive language grounding benchmark. arXiv preprint 1:Xiv:2/10.10661 2021.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718, 2023.
13

---

A Additional Details
A.1 Scale of Auxiliary Rewards
In practice, the R&R framework is quite robust to the scale of auxiliary reward since reward clipping has been implemented by most RL algorithms (Hu) 2022). Any reward in the range of (2,50) should result in the same behavior.
A.2 Atari Baselines
A2C/PPO. A2C (Mnih al. 2016) and PPO (Schulman et al. 2017 among some of the earliest
Actor-Critic algorithms that brought success for deep RI. on Atari games. A2C learns a policy æ and value function V reduce the variance of the REINFORCE (Sutton al. 1999) algorithm. The algorithm optimizes the advantage function A(s, a) ((s, a) V(s) instead of the action value function (s, a) further stabilize training.
MuZero. MuZero algorithm (Schrittwisser et al. 2020) combines a tree-based search with a learned model of the environment. The model of MuZero iteratively predicts the reward. the sectonnselection policy, and the value function. When applied to the Atari benchmarks, MuZero achieves super-human performance in lot of games.
R2D2/R2D1. R2D2 KKpturowski et al. 2018) proposes an improved training strategy for RNN-
based RL agents with distributed prioritized experience replay. The algorithm the first exceed human-level performance in 52 of the 57 Atari games. R2D1 is a by (Stooke Abbeel 2019).
variant implemented
Agent57. Badia et al. 2020) train a neural network that parameterizes a family of policies with different exploration, and proposes mnetacontrrlllrr to choose which policy prioritize during training. Agent57 exceeds human-level performance on all 57 Atari games.
A.3 Explanation of Game Objectives
Skiing. The player controls the direction and speed of skier and must avoid obstacles, such as trees and moguls. The goal is to reach the end of the course as rapidly as possible, but the skier must pass through series of gates (indicated by a pair of closely spaced flags). Missed gates count as penalty against the player's time.
Tennis. The player plays the game of tennis. When serving and returning shots, the tennis players automatically swing forehand or backhand as the situation demands, and all shots automatically clear the net and land in bounds.
Paeman (Ms. Pac-Man). The player earns points by eating pellets and avoiding ghosts, which contacting them results in losing a life. Eating "power pellet" causes the ghosts to turn blue, allowing them to be eaten for extra points. Bonus fruits can be caten for increasing point values, twice per round.
Breakout Using a single ball, the player must use the paddle to knock down as many bricks as possible. If the player's paddle misses the ball's s rebound, the player will lose a life.
A.4 Outputs of QA Extraction Module
14

---

Game
Wikipedia
Official Manual
What is the objective of the game?
To reach the bottom of the ski course as rapidly as possible.
To reach the bottom of the hill in the fastes time Have fun and god bless!".
How succeed the game?
Learning to control the tips of your skis and anticipating and avoiding trouble.
N/A
Skiing
How to score at the game?
N/A
Elapsed time at the end of the run. Who are your enemies?
N/A
N/A
What the objective of the game?
To score as many points as you can prac- tice clearing the maze of dots before try- ing to gobble up the ghosts.
The player earns points by eating pellets and avoiding ghosts.
How succeed in the game?
The player carns points by cating pellets and avoiding ghosts.
Pacman
Score as many points as you can.
How to score at the game?
The player earns points by eating pellets and avoiding ghosts.
N/A
Who are your enemies?
Ghosts. stay close to energy pill be- fore eating it, and tease the ghosts.
N/A
What the objective of the game?
The first player to win one six six-game set is declared the winner of the match.
N/A
How to succeed in the game?
The first player to win one six-game set is declared the winner of the match.
precisely aim your shots and hit them out of reach of your opponent.
Tennis
How to score at the game?
The first player to win at least 6 games and be ahead by two games wins the set. Who are your enemies?
N/A
N/A
N/A
What is the objective of the game? Destroy the two walls using five balls To destroy the wall in little time
The player must knock down as many
possible The first player or team to com- pletely destroy both walls score the most points To smash their way through the wall and score points
bricks as possible.
Breakout
How to succeed in the game?
the player must knock down many bricks as possible by using the walls and/or the paddle below hit the ball against the bricks and eliminate them
The first team destroy wall or the most points after playing five balls wins the game To destroy the walls in little time as possible
How to score at the game?
Scores are determined by the bricks hit during game player scores points by hitting one of the wall's bricks Smash their way through the wall and score points.
N/A
Who are your enemies?
N/A
N/A
Table 5: Table showing the outputs of the QA Extraction module on Wikipedia instructions vs the official Atari manual. The Wikipedia manual significantly shorter, and contains less information, causing the extractive QA model to repeat answers. However, the overall information, across the questions, captured by the Extraction module in good agreement across Wiki and Official manuals.
15


Reviews:
[{"review_id": "qV7UUCwtl7", "reviewer": "Reviewer_3Ei6", "summary": "This paper presents a novel method for Single Agent Reinforcement Learning which utilises computer game instruction manuals to enhance learning efficiency and performance. The Atari game manuals are used to accelerate RL algorithms in learning to play four different games. The framework comprises a Question-Answer Extraction module and a Reasoning module.\n\nThe QA Extraction module summarizes salient information from the manual. The Reasoning module evaluates in-game events using the extracted information from the manual and then assigns auxiliary rewards when such interactions are detected. These auxiliary rewards are provided to standard RL agents.\n\nThe paper's results show improvements in both performance and training speed for various RL algorithms when aided by this novel framework. It is asserted that this is the first successful attempt to use instruction manuals in a fully automated framework for solving the Atari RL benchmarks.\n\nIt is noted that even small open-source QA language models could effectively extract useful gameplay information from human-written documentation for RL agents.", "questions": "Without a full analysis of the variation in results over a large number of random seeds, how can the results be trusted?", "limitations": "I believe that the limitations provided are fair.", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The results in this paper are certainly extremely promising, and the ability to use large language models to create additional rewards from the instruction manual is very promising. That such methods are so much more sample efficient is clearly going to be important in realms beyond simple game play.", "weaknesses": "The main, and very problematic issue with this paper is the lack of robust evaluation. There are no statistics given, no averaging over random seeds provided and no code made available. It is thus entirely unclear how much the results themselves can be trusted under scrutiny. Thorough and transparent evaluation is absolutely vital in all RL scenarios, and because this has been left out completely, this paper is, I believe, not ready for publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing that our method is novel and the results are promising, and thank you for pointing out concerns about the reliability of our evaluation (W1, Q1). We would like to reiterate that our *main contribution* is to sketch out a framework making use of LLMs for exploiting external textual data that future RL algorithms may build upon (also recognized by reviewer qAA2, 7mzB).\n\nThe read and reward framework is independent from the 4 RL algorithms we evaluated. Our experiments (Table 1) involving 3 games and 4 different algorithms already demonstrate the robustness of our framework.\n\nDue to the fact that our framework provides dense reward for the RL agent, we observe consistent performance across different trials. For completeness we will include additional experiments (Attached to rebuttal) involving 3 random seeds in the final paper. Our computational resources are limited so we will not be able to finish Agent57 experiments before the end of the rebuttal period. "}, {"review_id": "Emnf3unsnk", "reviewer": "Reviewer_qAA2", "summary": "This paper proposes Read and Reward (RR), a method to incorporate prior human knowledge about the environment to achieve performance and efficiency gains in RL environments. The paper instantiates RR in several Atari environments by reading the information from the instruction manual. A full end-to-end pipeline is demonstrated on Skiing and a pipeline without object detection is demonstrated on Tennis, Pacman, and Breakout.", "questions": "1. Would it be possible to have more results on other Atari games? I understand that the bounding-box detection was challenging, but are Tennis, Pacman, and Breakout the only other Atari games that allow for ground truth calculation of the objects from the RAM state?\n2. How robust is the automated reward shaping to the quality of the bounding box detector in Skiing? \n", "limitations": "While some limitations are addressed, it would be interesting if there were additional discussion about the limitations of extracting information from natural language data. For example, humans understanding of the physical world may not be adequately encoded in natural language. Does this limit the application scope of agents using the RR method? Additionally, if the LM's reasoning abilities are erroneous, will this irreparably break RR?\n\nFinally, it would be helpful to directly note that as RR is fundamentally a reward shaping method, it is difficult to apply it to situations where the reward itself is unclear. This happens often with delayed reward. In particular, RR would likely not improve performance on Go or Chess.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "*Originality*: While incorporating prior knowledge into RL training is a previously investigated topic, this paper provides an end-to-end pipeline and demonstrates its usefulness on Atari Skiing. The method introduced is more general than methods previously investigated.\n\n*Quality*: There are thorough experiments with ablation studies, such as training the QA module on a different set of instructions.\n\n*Clarity*: the paper is well-written.\n\n*Significance*: as RL agents are eventually deployed in real-world scenarios, it will be increasingly important to have them learn efficiently and incorporate prior knowledge, which is often in the form of text. This paper provides a complete demonstration of a method leveraging natural language data to improve performance. More generally, it sketches out a framework for exploiting external data that future algorithms may build upon. Such agents may be rapidly deployed to learn in an unsupervised fashion.", "weaknesses": "Although RR is an interesting proof-of-concept of automated reward shaping, it is unclear whether it scales to more complex environments or works on noisier sources of textual knowledge. In particular:\n1. Does RR generalize to noisier sources of textual data? While the Wikipedia ablation is interesting, the data itself is rather clean and contains similar information. One interesting experiment to try might be with text-based RL environments, such as the Jericho [1] or Machiavelli [2] environments. Oftentimes these environments have publicly available [walkthroughs online](https://forum.choiceofgames.com/t/guides-for-all-games/15569/4), although they are often incomplete and noisy, as the instructions are intended for humans. Is RR able to extract the rewards and guide the agents in this setting? (I realize that this would be a significant undertaking so am not expecting it in the rebuttal)\n2. The reward shaping only surrounds object detection. As a result, the QA prompt, even though it is hand-designed, still generalizes. However, there are several RL environments where the rewards are more heuristics rather than strict rules. For example, dialogue in the Diplomacy environment [3] does not have clear positive or negative reward but instead depends on the context. One could imagine gaining better understanding of dialogue in Diplomacy by leveraging knowledge on the internet, but it is unclear how RR might be applied in that scenario, as the reward shaping must center on text and the QA prompt likely could not be hand-designed.\n3. It seems like it was quite difficult to train a bounding-box detector for most games. However, this would imply that RR's capabilities are bottlenecked by limitations of such bounding-box detectors. This might make RR difficult to scale to more complex environments with more ambiguous objects.\n\n[1] Hausknecht, M., Ammanabrolu, P., Coté Marc-Alexandre, & Yuan Xingdi (2019). Interactive Fiction Games: A Colossal Adventure. CoRR, abs/1909.05398.\n[2] Pan, A., Chan, J., Zou, A., Li, N., Basart, S., Woodside, T., Ng, J., Zhang, H., Emmons, S., & Hendrycks, D. (2023). Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark. ICML.\n[3] Meta Fundamental AI Research Diplomacy Team (FAIR) et al. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science 378,1067-1074 (2022). DOI:10.1126/science.ade9097", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the novelty and significance of our work. Here are our responses to the questions and concerns:\n\nW1 Generalization to noisier data sources: \n\nCurrent implementation of RR uses small language models (RoBERTa and Macaw), which already demonstrate some degree of robustness to noisy manual information in the Atari official instruction manual intended for human readers (Section 3, line 125~127). To improve performance, one could adopt a more powerful language model like GPT-4. \n\nW2 Limitation of reward shaping for dialogue games:\n\nThank you for pointing out this important limitation. Our main contribution is a framework to transfer external knowledge to RL agents to games with well-defined observation and action space. It indeed takes more work to apply to specific scenarios like Diplomacy.\n\nW3 Bounding-box generation:\n\nThis is indeed one of the main challenges for future works. We hope that the advances in visual-language models (VLMs) could aid our progress.\n\nQ1 Additional Atari games: \n\nCurrently, reliable ground-truth object labels only exist for Tennis, Ms. Pacman, Breakout. However, we believe deeply in the impact of creating a general solution for all Atari games, and have been actively attempting to integrate recent visual-language models (VLMs). Although we do not have a working solution yet due to the lack of compositionality in VLMs, we believe that our approach will lay the ground for a future solution that reads the instruction manual and solves all Atari games.\n\nQ2 Robustness to bounding box detection error: Thank you for raising this important question:\n\nIn practice, we find RR quite robust to bounding box detection errors since the bounding box detector causes at least one observable error every 10 frames, and almost all detection errors result in missing automated reward. \n\nAssuming uniform probability of missing reward for any interaction, the expectation of reward is still positive/negative, and therefore the RR framework should demonstrate a reasonable degree of robustness to noise in detectors.\n\nLimitations:\n\nThank you for the insightful suggestions on scenarios where RR may not apply, and where LLMs may fail. We will include more discussions on scenarios where RR does not apply in the final version of our paper."}, {"review_id": "UhpQ4IKk1y", "reviewer": "Reviewer_LzHL", "summary": "The authors introduce Read and Reward framework, where RL agents accelerate their learning of a new environment by interpreting user manuals. Specifically, the framework consists of a QA Extraction module that extracts and summarizes relevant information and a Reasoning module that evaluates object-agent interactions based on the information. The algorithm accelerates learning by using auxiliary rewards, whenever the interactions are detected by the reasoning module. The method empirically boosts the efficiency and performance of many Atari games.", "questions": "- have you tried using VLMs to do the reasoning? instead of object detectors. This way the set of interactions can also be expanded\n- does the scale of the aux reward matter in the games investigated?\n- is there an intuition on the speedup ratio (Table 3) Vs. game? (Breakout appears to really benefit from R&R)", "limitations": "Both limitations and broader impact sections are included in the draft", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The topic in focus is very relevant because of the recent surge in the abilities of LLMs. The solution of utilizing LLMs to map auxiliary rewards in RL is novel. The performance of speeding up RL training for the four games investigated is apparent. The ablations performed are thorough and complete.", "weaknesses": "- document length and unstructuredness are overstressed in the paper as a method/contribution. However, a) many NLP papers have already addressed them, and the method employed in the paper is simple summarization; b) long context models are already available to ingest more than the text needed to reason\n- the overall idea is quite nice; however, due to the difficulty of grounding knowledge in the environment, the paper only considered \"hit\" interactions and required unsupervised object detectors (which might be unreliable) OR game states, and the interaction is quite primitive in many scenarios, therefore cannot describe most tasks/useful interactions.\n- Table 1 needs error bars/std\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the potential of our work. \n\nHere are our responses to the questions and concerns:\n\nW1 Document Length: We would like to reiterate that our *main contribution* is to sketch out a framework making use of LLMs for exploiting external textual data that future RL algorithms may build upon (also recognized by reviewer qAA2, 7mzB), by making use of recent NLP techniques and long context models. While recent works within a few months have somewhat transformed NLP and in-context reasoning, they are orthogonal to the main focus of this work and long context problems (for example, reading a paper) may still exist. Our RR framework is compatible with all recent LLMs and will likely result in more possibilities. \n\nW2 Grounding: As mentioned in Section 5, while there is a strict requirement on grounding, VLMs may open up more potential for better interaction tracking and wider the set of interactions. Although our internal experiments demonstrate VLMs still lack compositionality and cannot ground interactions as accurate as our current framework, we are hopeful that the community will soon discover more reliable VLMs that are suitable for this task. \n\nW3 Table 1: Thank you for pointing out the insufficiency in our evaluation. We have initiated experiments and have attached std for all algorithms except Agent57 in the rebuttal. We will update the table with full results for the final version.\n\nQ1 VLMs:\n\nWe have been actively experimenting with current VLMs. Although our internal experiments demonstrate VLMs still lack compositionality and cannot ground interactions as accurate as our current framework, we are hopeful that the community will soon discover more reliable VLMs that are suitable for this task. \n\nQ2 Scale of Aux reward: \n\nThe framework is quite robust to the scale of auxiliary reward since reward clipping has been implemented by the RL algorithms. Any reward in the range of (2,50) should result in the same behavior.\n\nQ3 Breakout benefitting from RR:\n\nDue to the mechanism of the game and the content of the instruction manual. An auxiliary reward is provided very densely every time the paddle hits the ball. Therefore, the RL algorithms learn fastest in the game Breakout because auxiliary reward is provided most often."}, {"review_id": "g83jGweKl5", "reviewer": "Reviewer_7mzB", "summary": "RL has a notorious sample efficiency problem, and the authors propose to tackle this problem by having RL agents read instruction manuals. The authors propose a novel framework called Read and Reward (R&R), which is composed of two modules. The first module is QA which extracts and summarizes relevant information in the manual, and the second module is reasoning which evaluates events in the game and provides appropriate rewards for the RL agents to learn with algorithms such as A2C and Agent 57. The authors found R&R to significantly improve the agent's sample efficiency in several challenging games, such as Skiing.\n\n\n\n", "questions": "I'm curious about the delayed reward schedule. Would it not artificially create a sparse reward problem? \n\n\nMisc:\n* The citation format seems slightly problematic. `However, in many real-world scenarios (Kolve et al., 2017)` feels more common and readable compared to `However, in many real-world scenarios Kolve et al. (2017);`\n* Lines 44 and 48 seem repetitive.", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and has the following strengths:\n\n* **Interetsing approch**: as the authors mentioned, this is the first work that demonstrates improving RL performance by reading instruction manuals. Many pure RL methods suffer significantly in Skiing, so seeing other approaches that improve training is encouraging.\n* **Further contribution to NLP + RL**: the authors further explore the space of using NLP to improve RL agents. This line of research will help reduce the need to train agents from scratch and provide useful priors to the agent.\n* **No labeling required**: the amount of human labor required in the proposed framework is minimal — they only provide some generic questions, such as \"what is the objective of the game?\" ", "weaknesses": "Despite its strengths, the paper has the following weaknesses:\n\n* **Adhoc setting**: the setting seems ad hoc, stitching together various modules such as QA and object detection. A multi-modality model such as GPT-4 can probably serve as a general replacement for these modules, unlocking new capabilities, such as learning from image/video tutorials, in addition to instruction manuals. That said, there has not been an excellent open-source multi-modality model available. \n* **Slightly insufficient evaluation**: the authors only tested their approach on four games, but maybe it's worthwhile to evaluate some other games, such as Montezuma Revenge, which RL algorithms usually struggle with. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing that our proposed framework is interesting and promising. Here are our responses to the questions and concerns:\n\nW1 Multi-modality models:\n\nThank you for recognizing the important future direction to train and incorporate multi-modal LMs. We have been actively experimenting with the latest VLMs. Although our internal experiments demonstrate VLMs still lack compositionality and cannot ground interactions as accurate as our current framework, we are hopeful that the community will soon discover more reliable VLMs that are suitable for this task.\n\nW2 Evaluation: Currently, reliable ground-truth object labels only exist for Tennis, Ms. Pacman, Breakout. However, we believe deeply in the impact of creating a general solution for all Atari games, and have been actively researching more general solutions for visual-language grounding.\n\nIn addition, we have initiated experiments and have attached std for all algorithms except Agent57 in the rebuttal. We will update the table with full results for the final version.\n\nQ1 Delayed reward schedule:\n\nThe delayed reward schedule introduces `sparse reward' for both Read & Reward and baselines (similar to the original setting for the Skiing game). Therefore, the effect of Read & Reward is more visible.\n\n\nMisc:\nThank you for pointing out minor improvements in our paper. We will correct them in the final version."}]

Author rebuttal:
We thank all the reviewers for their insightful comments. We are encouraged by *all reviewers'* appreciation that our proposed framework achieves promising results (Reviewers 3Ei6, qAA2, LzHL, 7mzB). In addition, we are encouraged by acknowledgements for our contributions on 1) an important connection between the rapidly advancing LLM field and RL. (Reviewers LzHL, qAA2, 7mzB) 2) getting an end-to-end system together that clearly makes use of text data (Reviewers qAA2, 7mzB).

## Our contribution
We would like to reiterate that our main contribution is to sketch out a framework making use of LLMs for exploiting external textual data that future RL algorithms may build upon (also recognized by reviewer qAA2, 7mzB). 

## On robustness of evaluation
Our experiments (Table 1) involving 3 games and 4 different algorithms already demonstrate the robustness of our framework on different tasks and algorithms.

|      | Tennis | Pacman | Breakout |
| :----------- | :-----------: | :-----------: | :-----------: |
| A2C | -23 (1.3) | 387.1 (66.2) | 2.1 (0.3) |
| A2C + R & R | -5.2 (2.2) | 455.2 (63.2) | 4.5 (3.0) |
| PPO | -23 (1.0) | 200.1 (65.4) | 1.9 (0.5) |
| PPO + R & R | -8.1 (2.3) | 284.2 (67.3) | 10.2 (0.1) |
| R2D1 | -23.0 (1.3) | 1999.2 (109.2) | 2.1 (0.2) |
| R2D1 + R & R | -2.2 (3.1) | 3001.3 (203.2) | 10 (3.0) |

To further address reviewer concerns, we have conducted experiments with 3 random seeds for each entry in the table and reported the average and (standard deviation of the experiments). Note that we were unable to attach results for Agent57 due to the fact that each Agent57 experiment takes on average 1 month to run on our machines.

We will update the final version of our paper with the results in this table.

Meta-review: The reviewers found the paper interesting, and the ability to guide an RL agent using an LLM promising. However, it is not clear if this approach can be generalized to other setups. Finally, during the rebuttal period, the authors provided an experimental table that included the results using multiple seeds. In this table, this approach was significantly better in 6 out of 9 cases, so it is clear a better experimental evaluation is needed, including more games and more seeds.
Decision: Accept (poster)