SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since Read and Reward only provides an auxiliary reward, it is compatible with most RL frameworks such as A2C, PPO, SAC, TD3, and Agent 57 (Mnih et al., 2016; Schulman et al., 2017; Haarnoja et al., 2018; Fujimoto et al., 2018; Badia et al., 2020). We use popular open-source implementations A2C,PPO,R2D1 (Stooke & Abbeel, 2019), and Agent57 (Hu, 2022).
4.6 Full Pipeline Results on Skiing with A2C
We compare the performance of our Read and Reward agent using the full pipeline to baselines in Figure 5, our agent is best on the efficiency axis and close to best on the performance axis. A2C equipped with Read and Reward outperforms A2C baseline and more complex methods like MuZero (Schrittwieser et al., 2020). We note a  $50\%$  performance gap between our agent and the SOTA Agent 57, but our solution does not require policy mixing and requires more than 10 times fewer frames to converge to performance  $60\%$  better than random.
Figure 5: Performance (in Game Score) vs Efficiency (in Frames Seen) of fully end-to-end Read and Reward (green) compared to an A2C baseline, and MuZero (Schrittwieser et al., 2020) on Skiing. Benefiting from the auxiliary rewards from the instruction manual, Read and Reward out-performs the A2C Baseline and achieves  $60\%$  improvement on random performance while using much fewer training frames compared to the SOTA mixed-policy Agent 57 (Badia et al., 2020) and GDI-H3 (Fan & Xiao, 2022).
When experimenting with a full end-to-end pipeline as described in Section 4.3, we notice instability with the detection and grounding models. As shown in Figure 4, missing bounding boxes cause lots of mislabeling when objects are close to each other. Such errors may lead to the auxiliary reward being incorrectly assigned.
Game	Wikipedia	Official Manual
Pacman	What is the objective of the game?
The player earns points by eating pellets and avoiding ghosts.	To score as many points as you can prac- tice clearing the maze of dots before try- ing to gobble up the ghosts.
How to succeed in the game?
The player earns points by eating pellets and avoiding ghosts.	Score as many points as you can.
How to score at the game?
The player earns points by eating pellets and avoiding ghosts.	N/A
Who are your enemies?
N/A	Ghosts. stay close to an energy pill be- fore eating it, and tease the ghosts.
Table 3: Table showing the outputs of the QA Extraction module on Wikipedia instructions vs the official Atari manual. The Wikipedia manual is significantly shorter, and contains less information, causing the extractive QA model to use the same answer for all questions. Full table for all 4 games is shown in Table 5 in the Appendix.
Although Skiing is arguably one of the easiest games in Atari for object detection and grounding, with easily distinguishable objects and white backgrounds, we still observe that current unsupervised object detection and grounding techniques lack reliability.
4.7 Results on Games with Delayed Reward Structure
Since the problem of object detection and grounding is not the main focus of this paper, we obtain ground-truth labels for object locations and classes as mentioned in Section 4.3. In addition, for a setting more generalizable to other environments (Kolve et al., 2017; Fan et al., 2022), we implement delayed reward schedule (Section 4.2) for Tennis, MsPacman, and Breakout. Various RL frameworks benefit significantly from Read and Reward, as shown in Table 1, 2.
We plot the Game Score and the auxiliary rewards of A2C with Read and Reward v.s. the A2C baseline in Figure 6. The A2C baseline fails to learn under the sparse rewards, while the performance of the Read and Reward agent continues to increase with more frames seen. In addition, we observe that the auxiliary reward from the Read and Reward framework has a strong positive correlation with the game score (game reward) of the agent, suggesting that the A2C agent benefits from optimizing the auxiliary rewards at training time.
4.8 Results without Delayed Reward
Without delayed reward, Read and Reward does not offer performance improvement in terms of game score, but still improves training speed. As shown in Table 4, Agent57 with Read and Reward requires up to  $112\mathrm{X}$  less training steps to reach the same performance as the Agent57 baseline without the instruction manuals.
Figure 6: Comparison of the training curves of Read and Reward + A2C v.s. A2C Baseline in terms of game score alongside Auxiliary Reward v.s. Frames Seen under our delayed reward setting for 3 different Atari games. Read and Reward (red) consistently outperforms the A2C baseline (brown) in all games. In addition, the auxiliary reward from Read and Reward demonstrates a strong positive correlation with the game score, suggesting that the model benefits from optimizing the auxiliary rewards at training time.
4.9 R&R Behavior on Manuals from Different Source
To illustrate the generalization of our proposed framework to a different source of information, we download the "Gameplay" section from Wikipedia<sup>4</sup> for each game and feed the section as manual to our model. In Table 3, we show a comparison of the answers on the set of 4 generic questions on the Pacman by our QA Extraction Module. Note that the QA Extraction module successfully extracts all important information about avoiding ghosts and eating pellets to get points. In addition, since the official manual contains more information than the Gameplay section of Wikipedia (2 pages v.s. less than 4 lines), we were able to extract much more information from the official manual.
Due to the high similarity in the output from the QA Extraction Module, the Reasoning module demonstrates good agreement for both the Wiki manual and the official manual. Therefore, we observe consistent agent performance between the official manual and the Wikipedia manual in terms of game score (Table 1).
Tennis	Pacman	Breakout
steps to reach base-line@1e6	6e5	5.1e5	8911
speed-up ratio	1.67	2	112
Table 4: Table showing the number of training steps required by Agent57 with Read and Reward (Official) to reach the same performance as the Agent57 baseline at 1e6 training steps and the speedup ratio, under delayed-reward.
One potential challenge and limitations is the requirement of object localization and detection as mentioned in Section 4.3. However, since Read and Reward does not need to detect all objects to assign reward, i.e., energy pills are not grounded in the game Pacman but Read and Reward is still achieves improvement. In addition, reliable backbone object detection models (He et al., 2017) have already shown success in tasks such as indoor navigation (Shridhar et al., 2020b) and autonomous driving. With recent progress on visual-language models (Bubeck et al., 2023; Li et al., 2022; Driess et al., 2023; Liu et al., 2023; Zou et al., 2023), we believe that our proposed framework can benefit from reliable and generalizable grounding models in the foreseeable future.
Another simplification we made in Section 4.4 is to consider only interactions where objects get sufficiently close to each other. While such simplification appears to be sufficient in Atari, where observations are 2D, one can imagine this to fail in a 3D environment like Minecraft (Fan et al., 2022) or an environment where multiple types of interaction could happen between same objects (Kolve et al., 2017). Future works would benefit by grounding and tracking more types of events and interactions following recent works on video scene understanding Wang et al. (2022b) or game progress tracking Wu et al. (2023).
Finally, the performance of Read and Reward may be limited by information within the natural language data it reads.
In this work, we propose Read and Reward, a method that assists and speeds up RL algorithms on the Atari challenges by reading downloaded game manuals released by the Atari game developers. Our method consists of a QA extraction module that extracts and summarizes relevant information from the manual and a reasoning module that assigns auxiliary rewards to detected in-game events by reasoning with information from the manual. The auxiliary reward is then provided to standard RL agents. To our knowledge, this work is the first successful attempt for using instruction manuals in a fully automated and generalizable framework for solving the widely used Atari RL benchmark (Badia et al., 2020).
Our results suggest that even small open-source QA language models could extract meaningful gameplay information from human-written documentations for RL agents. With recent break-trough in zero-shot language understanding and reasoning with LLMs (Bubeck et al., 2023), we hope that our work could incentivize future works on integrating human prior knowledge into RL training.
$^{4}$ wikipedia.org
Our research on using instruction manuals for RL could lead to both positive and negative impacts. The benefits include more efficient RL agents that could solve more real-world problems. The risks may involve more game cheating and exploitation, over reliance on the instruction manuals, and potential job loss.
A Additional Details
A.1 Scale of Auxiliary Rewards
In practice, the R&R framework is quite robust to the scale of auxiliary reward since reward clipping has been implemented by most RL algorithms (Hu, 2022). Any reward in the range of (2,50) should result in the same behavior.
A.2 Atari Baselines
A2C/PPO. A2C (Mnih et al., 2016) and PPO (Schulman et al., 2017) are among some of the earliest Actor-Critic algorithms that brought success for deep RL on Atari games. A2C learns a policy  $\pi$  and a value function  $V$  to reduce the variance of the REINFORCE (Sutton et al., 1999) algorithm. The algorithm optimizes the advantage function  $A(s, a) = Q(s, a) - V(s)$  instead of the action value function  $Q(s, a)$  to further stabilize training.
MuZero. MuZero algorithm (Schrittwieser et al., 2020) combines a tree-based search with a learned model of the environment. The model of MuZero iteratively predicts the reward, the action-selection policy, and the value function. When applied to the Atari benchmarks, MuZero achieves super-human performance in a lot of games.
R2D2/R2D1. R2D2 (Kapturowski et al., 2018) proposes an improved training strategy for RNN-based RL agents with distributed prioritized experience replay. The algorithm is the first to exceed human-level performance in 52 of the 57 Atari games. R2D1 is a single-machine variant implemented by (Stooke & Abbeel, 2019).
Agent57. Badia et al. (2020) train a neural network that parameterizes a family of policies with different exploration, and proposes a meta-controller to choose which policy to prioritize during training. Agent57 exceeds human-level performance on all 57 Atari games.
A.3 Explanation of Game Objectives
Skiing. The player controls the direction and speed of a skier and must avoid obstacles, such as trees and moguls. The goal is to reach the end of the course as rapidly as possible, but the skier must pass through a series of gates (indicated by a pair of closely spaced flags). Missed gates count as a penalty against the player's time.
Tennis. The player plays the game of tennis. When serving and returning shots, the tennis players automatically swing forehand or backhand as the situation demands, and all shots automatically clear the net and land in bounds.
Pacman (Ms. Pac-Man). The player earns points by eating pellets and avoiding ghosts, which contacting them results in losing a life. Eating a "power pellet" causes the ghosts to turn blue, allowing them to be eaten for extra points. Bonus fruits can be eaten for increasing point values, twice per round.
Breakout Using a single ball, the player must use the paddle to knock down as many bricks as possible. If the player's paddle misses the ball's rebound, the player will lose a life.
A.4 Outputs of QA Extraction Module
Game	Wikipedia	Official Manual
Skiing	What is the objective of the game?
To reach the bottom of the ski course as rapidly as possible.	To reach the bottom of the hill in the fastest time Have fun and god bless!".
How to succeed in the game?
N/A	Learning to control the tips of your skis and anticipating and avoiding trouble.
How to score at the game?
N/A	Elapsed time at the end of the run.
Who are your enemies?
N/A	N/A
Pacman	What is the objective of the game?
The player earns points by eating pellets and avoiding ghosts.	To score as many points as you can practice clearing the maze of dots before trying to gobble up the ghosts.
How to succeed in the game?
The player earns points by eating pellets and avoiding ghosts.	Score as many points as you can.
How to score at the game?
The player earns points by eating pellets and avoiding ghosts.	N/A
Who are your enemies?
N/A	Ghosts. stay close to an energy pill before eating it, and tease the ghosts.
Tennis	What is the objective of the game?
The first player to win one six-game set is declared the winner of the match.	N/A
How to succeed in the game?
The first player to win one six-game set is declared the winner of the match.	precisely aim your shots and hit them out of reach of your opponent.
How to score at the game?
N/A	The first player to win at least 6 games and be ahead by two games wins the set.
Who are your enemies?
N/A	N/A
Breakout	What is the objective of the game?
The player must knock down as many bricks as possible.	Destroy the two walls using five balls To destroy the wall in as little time as possible The first player or team to completely destroy both walls or score the most points To smash their way through the wall and score points
How to succeed in the game?
the player must knock down as many bricks as possible by using the walls and/or the paddle below to hit the ball against the bricks and eliminate them	The first team to destroy a wall or score the most points after playing five balls wins the game To destroy the walls in as little time as possible
How to score at the game?
N/A	Scores are determined by the bricks hit during a game A player scores points by hitting one of the wall's bricks Smash their way through the wall and score points.
Who are your enemies?
N/A	N/A
Table 5: Table showing the outputs of the QA Extraction module on Wikipedia instructions vs the official Atari manual. The Wikipedia manual is significantly shorter, and contains less information, causing the extractive QA model to repeat answers. However, the overall information, across the 4 questions, captured by the Extraction module is in good agreement across Wiki and Official manuals.