SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SPRING: Studying the Paper and Reasoning to Play Games
Yue Wu $^{14*}$ ; Shrimai Prabhumoye $^{2}$ , So Yeon Min $^{1}$ , Yonatan Bisk $^{1}$ , Ruslan Salakhutdinov $^{1}$ ,
Amos Azaria<sup>3</sup>, Tom Mitchell<sup>1</sup>, Yuanzhi Li<sup>1,4</sup>
$^{1}$ Carnegie Mellon University,  $^{2}$ NVIDIA,  $^{3}$ Ariel University,  $^{4}$ Microsoft Research
Open-world survival games pose significant challenges for AI algorithms due to their multi-tasking, deep exploration, and goal prioritization requirements. Despite reinforcement learning (RL) being popular for solving games, its high sample complexity limits its effectiveness in complex open-world games like Crafter or Minecraft. We propose a novel approach, SPRING, to read Crafter's original academic paper and use the knowledge learned to reason and play the game through a large language model (LLM). Prompted with the LATEX source as game context and a description of the agent's current observation, our SPRING framework employs a directed acyclic graph (DAG) with game-related questions as nodes and dependencies as edges. We identify the optimal action to take in the environment by traversing the DAG and calculating LLM responses for each node in topological order, with the LLM's answer to final node directly translating to environment actions. In our experiments, we study the quality of in-context "reasoning" induced by different forms of prompts under the setting of the Crafter environment. Our experiments suggest that LLMs, when prompted with consistent chain-of-thought, have great potential in completing sophisticated high-level trajectories. Quantitatively, SPRING with GPT-4 outperforms all state-of-the-art RL baselines, trained for 1M steps, without any training. Finally, we show the potential of Crafter as a test bed for LLMs. Code at github.com/holmeswww/SPRING
Open-world survival games like Minecraft (Fan et al., 2022) and Crafter (Hafner, 2021) pose significant challenges for AI algorithms due to a combination of factors: procedural generation requires strong generalization; diverse action space requires multi-task capabilities; technology tree requires long-term planning and deep exploration; diverse and conflicting objectives require goal prioritization. In particular, Crafter is designed for efficient simulation and fast iteration. Similar to Minecraft, Crafter features key challenges such as multi-tasking, exploration with a deep and wide tech-tree, requiring the agent to craft multiple tools and interact with multiple objects to survive in the game.
Reinforcement learning (RL) has been the go-to approach for game-based problems, with numerous successes in games like Go (Silver et al., 2017), robotics (Fu et al., 2020; Hafner et al., 2023) and various video games (Vinyals et al., 2019; Schrittwieser et al., 2020; Badia et al., 2020; Hafner et al., 2023). While RL demonstrated impressive performance, it still suffers from certain limitations, such as high sample complexity and difficulty in incorporating prior knowledge. Such drawbacks make it exceptionally challenging to apply RL to diverse and complex open-world benchmarks like Crafter (Hafner, 2021) or Minecraft (Fan et al., 2022). Addressing the benefits and drawbacks of RL is therefore crucial for achieving a sample-efficient solution.
*Work done during internship at Microsoft. For correspondence, contact ywu5@andrew.cmu.edu
Figure 1: Overview of SPRING. The context string, shown in the middle column, is obtained by parsing the LATEX source code of Hafner (2021). The LLM-based agent then takes input from a visual game descriptor and the context string. The agent uses questions composed into a DAG for chain-of-thought reasoning, and the last node of the DAG is parsed into action.
On the other hand, large language models (LLMs) (Brown et al., 2020; Smith et al., 2022; Chowdhery et al., 2022) have shown remarkable success when prompted for various tasks, including embodied planning and acting (Ahn et al., 2022; Du et al., 2023; Wang et al., 2023; Shinn et al., 2023), QA or dialogue (Ouyang et al., 2022; Bubeck et al., 2023), and general problem-solving (Brown et al., 2020; Bubeck et al., 2023). Their unique planning (Ahn et al., 2022), reasoning (Shinn et al., 2023), and problem-solving (Bubeck et al., 2023; Madaan et al., 2023) ability makes them a promising candidate for incorporating prior knowledge and in-context reasoning for game-based problems, particularly when it comes to addressing the aforementioned limitations of RL.
Hence, in this work, we study the possibility and reliability of LLMs for understanding and reasoning with human knowledge, in the setting of games. We consider a two staged approach SPRING (Figure 1): (1) studying the paper: the first stage reads the  $\mathrm{LATEX}$  of the paper of (Hafner, 2021) and (2) reasoning: the second stage involves reasoning about that knowledge through a QA framework to take an environment action. Note that the Crafter environment was released after the data collection date of GPT-3.5 and GPT 4 (OpenAI, 2023) models $^2$ , the environment is unseen to them. We first use LLM to extract prior knowledge from the  $\mathrm{LATEX}$  source code of the original paper by Hafner (2021). We then use a similar QA summarization framework as Wu et al. (2023) which produces QA dialogue on game mechanics. SPRING handles significantly more diverse contextual information than (Wu et al., 2023), making use of all 17 action/interaction types and even information about desirable behaviors documented in the paper.
We focus on reading the relevant academic paper in the first stage of SPRING, by first deciding the paragraphs that are relevant for playing the game. Then we extract key information through a series of questions such as "Write all information helpful for the game in a numbered list". In the second stage, we promote and regulate in-context chain-of-thought reasoning in LLMs to solve complex games. The reasoning module is a directed acyclic graph (DAG), with questions as nodes and dependencies as edges. For example, the question "For each action, are the requirements met?" depends on the question "What are the top 5 actions?", creating an edge from the latter to the former. For each environment step, we traverse the DAG computing LLM answers for each node in the topological order of the graph. The final node of the DAG is a question about the best action to take and the LLM answer for the question is directly translated to environment action.
Qualitatively, our experiments show that LLMs, when prompted with consistent chain-of-thought, can execute sophisticated trajectories independently in Crafter (Hafner, 2021). Quantitatively, SPRING's zero-shot performance with GPT-4 surpassing all state-of-the-art RL algorithmstrained for 1M steps (Table 2).
- SPRING is the first to tackle a competitive RL benchmark by explicitly extracting multiple interactions and tech-tree dependencies directly from an academic paper.
$^{2}$ GPT-3.5/4 training data ends in September 2021 according to OpenAI API
- We are the first to show SOTA performance in a challenging open world game with a zero-shot LLM-based (GPT-4) policy
- We study the quality of in-context reasoning induced by different prompts and propose a controlled chain-of-thought prompting through a DAG of questions for decision making.
This section is structured as follows. We first describe how we generate the context from the LATEX source code of Hafner (2021) in Section 2.1. Then we describe our SPRING framework and how we compute the action in Section 2.2.
Problem Setting Our goal is to show that LLMs can plan and act reasonably well in an environment where control tasks are less required. In the setting of Crafter, we define the states,  $s$ , as samples from state distribution  $S$ . We are interested in creating a goal-conditioned policy  $\pi$  which maps state  $s$  to action  $a \in A$ ,  $\pi : S \to A$ . Due to the use of LLM, we further break the policy down into two parts: a descriptor  $\mathcal{D}$  which describes key aspects the visual observation in plain text  $(d = \mathcal{D}(s))$ . And an LLM-based actor which takes state description  $d$  and outputs action  $a$ .
In addition, we define  $S_{\mathrm{para}}^j$  to be the  $j^{\mathrm{th}}$  paragraph in the  $\mathrm{LATEX}$  source of the environment paper (Hafner, 2021), and  $\mathcal{M}_{LLM}$  to be the LLM which takes a context string and a question string as input and outputs an answer to the question.
2.1 Studying the paper: Context from  $\mathbf{L}\mathbf{A}\mathbf{T}_{\mathrm{E}}\mathbf{X}$  source
Figure 2: Paper Studying Mould. The 3-step approach for computing  $C_q$  from the LATEX source code of Hafner (2021). First, as shown in the left column, for each paragraph we compute LLM answer for all relevancy questions in  $Q_{\mathrm{rel}}$ , and keep only the relevant paragraphs. Second, as shown in the middle column, we compute paragraph-level LLM answer to  $q$ . Third, we summarize the answer into  $C_q$  with a summary prompt; we concatenate  $C_q$  across  $q \in Q_{\mathrm{game}}$  and obtain  $C$ .
Similar to Wu et al. (2023), we compose gameplay specific questions and then compute LLM answer to the questions for each subsection in the latex files. Since a considerable amount of the paper is irrelevant to the gameplay, we use a set of 2 questions  $Q_{\mathrm{rel}} = \{\text{"Would this paragraph help me succeed in this game?"}$  "Does this paragraph contain information on the game mechanics, or game strategies?}\} to identify relevance, and a set of 4 questions  $Q_{\mathrm{game}} = \{\text{"Write all information helpful for the game in a numbered list."}$  "In plain text. List all objects I need to interact/avoid to survive in the game. Use "I would like to X object Y" in each step. Replace Y by the actual object, X by the actual interaction.", "Write all game objectives numbered list. For each objective, list its requirements.", "Write all actions as a numbered list. For each action, list its requirements." \} to summarize gameplay and action space relevant information. We add the prompt "DO NOT answer in LaTeX." to all of  $Q_{\mathrm{game}}$  to prevent the LLM from outputting the list in LATEX format.
For a specific gameplay specific question  $q \in Q_{\text{game}}$ , our goal is to compute  $C_q$ , the answer to  $q$  conditioned on the paper. However, since the length of the paper exceeds input length constraints for most LLMs, we have to break the paper down into paragraphs individual  $S_{\text{para}}^j$ . We provide an illustration of the process in Figure 2.
First, we filter the paragraphs for relevance and keep only paragraphs identified as relevant by at least one question from  $Q_{\mathrm{rel}}$ . We set  $P_q^{\mathrm{rel}}$  to be the set of relevant paragraphs.
$$
P _ {q} ^ {\text {r e l}} = \left\{\mathcal {S} _ {\text {p a r a}} ^ {j} \mid \exists q _ {r} \in Q _ {\text {r e l}} s. t. \mathcal {M} _ {L L M} \left(\mathcal {S} _ {\text {p a r a}} ^ {j}, q _ {r}\right) = \text {" Y e s "} \right\} \tag {1}
$$
Second, we compute the set,  $A_{q}$ , of answers to  $q$  for each relevant paragraph from  $P_{q}^{\mathrm{rel}}$ , from the LATEX source code.
$$
A _ {q} = \left\{\mathcal {M} _ {L L M} \left(\mathcal {S} _ {\text {p a r a}}, q\right): \mathcal {S} _ {\text {p a r a}} \in P _ {q} ^ {\text {r e l}} \right\} \tag {2}
$$
Third, to obtain the answer string  $C_q$  from the set  $A_q$ , we query an LLM with a summarization prompt  $q_{\text{summarize}} =$  "Remove duplicate items."
$$
C _ {q} = \mathcal {M} _ {L L M} \left(\operatorname {c o n c a t} \left(A _ {q}\right), q _ {\text {s u m m a r i z e}}\right) \tag {3}
$$
Finally, we concatenate (with the linebreak character) all question-context pairs to form the context string  $C$  for SPRING.
$$
C = \operatorname {c o n c a t} \left(\{\text {" Q u e s t i o n :} q \text {A n s w e r :} C _ {q} \mid \forall q \in Q _ {\text {g a m e}} \}\right) \tag {4}
$$
2.2 Reasoning: QA-DAG for SPRING
Node	Question
q1	List objects in the current observation. For each object, briefly answer what resource it provides and its requirement.
q2	What was the last action taken by the player?
q3	For each object in the list, are the requirements met for the interaction?
q4	Did the last player action succeed? If not, why?
q5	List top 3 sub-tasks the player should follow. Indicate their priority out of 5.
q6	What are the requirements for the top sub-task? What should the player do first?
q7	List top 5 actions the player should take and the requirement for each action. Choose ONLY from the list of all actions. Indicate their priority out of 5.
q8	For each action in the list, are the requirements met?
qa	Choose the best executable action from above.
Table 1: List of all 9 questions in  $Q_{\mathrm{act}}$ . The questions are designed to promote consistent chain-of-thought. Experimentally, we find the LLM robust to different phrasing of the questions.
Figure 3: Reasoning. (a) The visual descriptor takes the last two gameplay screens as input, and outputs their descriptions in language  $(d^t,d^{t - 1})$ . (b) SPRING traverses a DAG of questions from Table 1 in topological order. Answer to the final question  $q_{a}$  is mapped to environment action using sub-string matching. (c) The LLM answer for each question (node) is conditioned on the previous 2 steps of observation, the context  $C$ , and answers to the immediate parents of the current node.
For LLMs to be able to understand the gameplay, we first follow Du et al. (2023); Wang et al. (2023) to define an visual descriptor  $\mathcal{M}_{\mathrm{desc}}$  which converts state  $s \in S$  to textual description  $d$  (Figure 3 a).
To achieve consistent chain-of-thought reasoning (Wei et al., 2021) throughout hundreds of steps within one round of gameplay, we compose a fixed set of questions  $Q_{\mathrm{act}} = \{q_1,\dots ,q_a\}$  to query
the LLM at every step of the game, with question-question dependencies as  $D = \{(q_u,q_v)|q_u,q_v\in Q_{\mathrm{act}}$  and answering  $q_{v}$  requires the answer of  $q_{u} \}$ . Note that the above specification forms a directed acyclic graph (DAG) with nodes  $Q_{\mathrm{act}}$  edges  $D$  (Figure 3 b).
For any question (node)  $q_v \in Q_{\mathrm{act}}$ , we compute the answer  $A_{q_v}^t$  for time step  $t$ , conditioned on the gameplay context  $C$ , most recent 2 steps of game description  $d^{t-1}, d^t$ , and answers to its dependencies (Figure 3 c).
$$
A _ {q _ {v}} ^ {t} = \mathcal {M} _ {L L M} \left(\operatorname {c o n c a t} \left(C, d ^ {t - 1}, d ^ {t}, \left\{A _ {q _ {u}} ^ {t} \mid \left(q _ {u}, q _ {v}\right) \in D \right\}\right), q _ {v}\right) \tag {5}
$$
Experimentally, we find that prompting the LLM with only the direct parents of a question greatly reduces the context length, and helps LLM to focus on the most relevant contextual information.
We traverse the DAG using a modified topological sort algorithm to compute LLM answer for each question based on its topological order. Finally, we map the answer to the last question in the node  $q_{a}$  directly to one of the 17 named actions in the environment with sub-string matching  $(a = A_{a}^{t})$ . We take the default action "Do" on sub-string matching failure.<sup>3</sup>
3 Experiments and Results
We present our experiments as follows. First, we explain our experimental setup and baselines for our experiments. Then, we compare SPRING to popular RL methods on the Crafter benchmark. Finally, we conduct experiments and analysis on different pieces of our architecture to study the influence of each part over the in-context "reasoning" capabilities of the LLM.
3.1 Experimental Details
The Crafter environment (Hafner, 2021) is a procedurally generated open-world survival game for benchmarking RL algorithms with 22 achievements in a tech tree of depth 7. The environment is a grid-world features top-down observation and discrete action space of size 17. The observation also shows the current inventory state of the player, including its health points, food, water, rest levels, and inventory. The game is inspired by Minecraft and features a similar get-to-diamond challenge. In comparison, Crafter captures many key research challenges of Minecraft in a simpler and faster environment, thus speeding up experiments and result collection.
Environment Descriptor The gameplay screen (top left of Fig 3.) consists of a  $9 \times 9$  grid ( $\{(i,j) \mid 1 \leq i, j \leq 9\}$ ). The top 7 rows consist of the local view of the world; each cell  $(i,j)$  is associated with a pre-defined background (e.g., "grass", "water", "none") and possibly with an object "asset" (e.g., "tree", "health", "player"). The bottom 2 rows represent agent status (e.g., "health") and item inventories, which include images of assets (e.g., "stone sword"), and the number of each ascent in the inventory.
Our environment descriptor accepts as input the gameplay screen and outputs a text description of the screen. We first create combinations of background and object (appearance) assets. Then we add number assets to recognize the quantity of inventory/ status. We match these combinations with the gameplay screen, using cv2.filter with a matching threshold of 0.9. We disable the detector during nights when observations are unreliable. Finally, for each  $(i,j)$ , we filter the matched combinations, and select the one with the highest matching score. From this information, we can measure the distance and direction of each object relative to the player; simultaneously, we can count the agent status and inventory item.
The environment descriptor then obtains the set of objects in observation  $\mathcal{O} = \{(obj, dist, direction)\}$ , the set of inventory items  $\mathcal{I} = \{(object, count)\}$ , and the agent status  $\mathcal{H} = \{(\text{attribute}, value, max)\}$ . Including only the closest object of each kind, we compose the observation description  $d$  as: "You see : - <obj> <dist> steps to your <direction>. Your status: <attribute>: <value>/<max>. Your inventory: - <object>: <count)". We describe direction of objects using "north", "south", "east", "west".
Evaluation Metrics Agents in Crafter are evaluated primarily based on two metrics: reward and score. The game assigns a sparse  $+1$  reward each time the agent unlocks a new achievement in an
<sup>3</sup>We will release code for our agent at github.com/anonymous
Method	Score	Reward	Training Steps4
Human Experts	50.5 ± 6.8%	14.3 ± 2.3	N/A
SPRING + paper (Ours)	27.3 ± 1.2%	12.3 ± 0.7	0
DreamerV3 (Hafner et al., 2023)	14.5 ± 1.6%	11.7 ± 1.9	1M
ELLM (Du et al., 2023)	N/A	6.0 ± 0.4	5M
EDE (Jiang et al., 2022)	11.7 ± 1.0%	N/A	1M
DreamerV2 (Hafner et al., 2020)	10.0 ± 1.2%	9.0 ± 1.7	1M
PPO (Schulman et al., 2017)	4.6 ± 0.3%	4.2 ± 1.2	1M
Rainbow (Hessel et al., 2018)	4.3 ± 0.2%	5.0 ± 1.3	1M
Plan2Explore (Sekar et al., 2020)	2.1 ± 0.1%	2.1 ± 1.5	1M
RND (Burda et al., 2018)	2.0 ± 0.1%	0.7 ± 1.3	1M
Random	1.6 ± 0.0%	2.1 ± 1.3	0
Table 2: Table comparing SPRING and popular RL algorithms in terms of game score, reward, and training steps. Results for SPRING is summarized over 5 independent trials. SPRING out-performs the previous SOTA in terms of all metrics. In addition, since SPRING gathers knowledge from reading the paper, it requires no training.
episode, and assigns reward of  $-0.1 / 0.1$  when the agent loses/gains one health point. The score metric (Hafner, 2021) is computed by aggregating the success rates for each achievement:
$$
S = \exp \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \ln \left(1 + s _ {i}\right)\right) - 1,
$$
where  $s_i$  is the agent's success rate on achievement  $i$  and  $N = 22$  is the number of achievements. Note that RL agents only train on the reward, and SPRING does not require any training.
RL Baselines We include results from popular actor-critic methods like PPO (Schulman et al., 2017); DQN variants like Rainbow (Hessel et al., 2018); intrinsically motivated methods like RND (Burda et al., 2018), Plan2Explore (Sekar et al., 2020), EDE (Jiang et al., 2022); LLM assisted solutions like ELLM Du et al. (2023); model-based methods like DreamerV2 (Hafner et al., 2020); DreamerV3 (Hafner et al., 2023), which currently holds the state-of-the-art.
LLMs. For LLM access, we use GPT-3.5-turbo (OpenAI, OpenAI) and GPT-4 (OpenAI, 2023) from OpenAI's API.
3.2 Overall Results
We compare the performance of RL baselines to SPRING with GPT-4 conditioned on the environment paper (Hafner, 2021) in Table 2.
SPRING out-performs the previous SOTA, including previous attempts at using LLMs for Crafter by large margins, achieving an  $88\%$  relative improvement on game score and a  $5\%$  improvement in reward on the best performing RL method (Hafner et al., 2023). Since the model obtains knowledge from reading the paper, SPRING requires 0 training steps, while RL methods generally require millions of training steps<sup>4</sup>.
We include a plot of unlock rate by task, comparing our method to popular RL baselines in Figure 4. SPRING assisted by prior knowledge out-performs RL methods by more than  $10\mathrm{x}$  on achievements like "Make Stone Pickaxe", "Make Stone Sword", and "Collect Iron", which are up to depth 5 down in the tech tree and significantly harder to reach through random exploration. For achievements "Eat Cow" and "Collect Drink", SPRING achieves perfect performance, whereas model-based RL framework like Dreamer-V3 has more than  $5\mathrm{x}$  lower unlock rate for "eat cow" since cows are moving and harder to reach through random exploration. Finally, we note that SPRING did not take the action "Place Stone", which can be reached easily by random exploration, since placing a stone was not discussed as beneficial for the agent in the paper (Hafner, 2021).
4We base our comparison on the hard 1M cap set by Hafner (2021).
Figure 4: Ability spectrum showing the unlocking percentages for all 22 achievements. Rainbow manages to drink water and forage for food. DreamerV3 collects coal, iron, stone, and forges more advanced tools and weapons. Since SPRING starts off with knowledge about the game, it achieves more than  $10\mathrm{x}$  higher unlock rate on previously hard-to-reach tasks like "Eat Plant", "Make Stone Pickaxe", "Make Stone Sword", and "Collect Iron".
Method	Achievement Depth	Reward	Questions per Step
SPRING + Full Paper	6	12.3 ± 0.7	9
SPRING + Paper w/ modified C	4	9.4 ± 1.8	9
SPRING + Action Description	4	8.2 ± 0.2	9
SPRING + w/o C	1	0.5 ± 0.2	9
SPRING + Full Paper	6	12.3 ± 0.7	9
Step-by-step prompt + Full Paper	5	7.3 ± 4.4	2
QA w/o DAG + Full Paper	4	4.3 ± 3.9	9
w/o QA + Full Paper	2	2.4 ± 1.3	1
SPRING + Full Paper	6	12.3 ± 0.7	9
SPRING + Full Paper w/ GPT-3.5	2	3.3 ± 2.9	9
Table 3: Analysis on how different parts of SPRING contribute to its performance, comparing the max achievement depth in the tech tree, the reward, and the number of human-written questions in the prompt. Results are summarized over 5 independent trials. The first 4 rows study the necessity of prior knowledge from the context string  $C$ . The middle 4 rows study different chain-of-thought prompting techniques. The last 2 rows study the role of LLMs. All three aspects are important for SPRING to achieve best reported performance.
3.3 Component Analysis
We study how the different aspects of the framework contribute to the behavior of the agent through a series of ablations as shown in Table 3.
Studying the LATEX Paper In the first 4 rows of Table 3, we investigate the contribution of gameplay context from the LATEX paper toward performance of the agent. We report the performance of SPRING with no contextual information (w/o  $C$ ) (row 4); SPRING conditioned on only the action descriptions and dependencies from (Hafner, 2021) Table F.1 (only question 4 from  $Q_{\mathrm{game}}$ ) (row 3); SPRING conditioned on the context manually modified to exclude the "crafting table" dependency for wooden_pickaxe by removing two corresponding lines from the context  $C$  (row 2); SPRING conditioned on the full context from the paper (row 1).