SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As expected, since Crafter environment is unseen for GPT, the agent achieves performance similar to random agent without any game context. When provided with only action descriptions and action dependencies, using only question 4 from  $Q_{\mathrm{game}}$  in section 2.1, SPRING achieves strong  $67\%$  performance comparable to DreamerV2 (Silver et al., 2017).
For the next piece of the experiment, we manually remove "near crafting table" dependency for wooden_pickaxe from its context, which is required for 11 later achievements. SPRING with GPT-4 incurs a  $24\%$  performance drop. Interestingly, we find that the LLM has some ability to recover
from the inaccurate context information. We observe that after failing to craft the wooden_pickaxe without a table, the agent instead tries to craft a wooden_sword first to maintain survival. Eventually, the agent was able to identify the missing requirement through guessing and trying after some unsuccessful trials, and craft the wooden_pickaxe. However, the confusion delayed the agent's progress and therefore causes the performance gap with the agent conditioned on the full context (row 5).
Reasoning In the middle 4 rows of Table 3, we investigate the contribution of different prompting methods toward performance of the model. Conditioned on the full context from the LATEX paper, we report the performance of GPT-4 directly prompted to output the action using the last question  $q_{a}$  only (row 8); GPT-4 prompted with all questions from  $Q_{\mathrm{act}}$  but in a list without the DAG dependencies  $D$  (row 7); GPT-4 prompted "Let's think step-by-step" (Kojima et al., 2022) about the next action, and prompted to choose a permissible action  $q_{a}$  with let's think step-by-step followed by  $q_{a}$  again (row 6); GPT-4 with SPRING (row 5).
Relative to our method, we observe that directly prompting the LLM for the action leads to a  $80\%$  performance drop, and therefore does not result in a meaningful agent. The popular chain-of-thought reasoning prompt "Let's think step-by-step" (Kojima et al., 2022) achieves reasonable reward with a  $40\%$  drop, but with a high  $60.27\%$  standard deviation. Qualitatively, we observe that the LLM produces inconsistent outputs across time steps, due to the fact that the model's chain-of-thought is not directed or controlled through the prompt. Therefore, LLMs prompted with "Let's think step-by-step" alone cannot reliably follow a good policy. Controlling the chain-of-thought with 9 questions from  $Q_{\mathrm{act}}$  (section 2.2) successfully controls the consistency of LLM outputs across time qualitatively. However, we observe that the LLM often ignores earlier questions at later stages of QA when all previous questions are presented in a list, leading to random disagreements in answers. For example, the LLM may correctly identify that it needs "wooden pickaxe" to mine the stone ahead in the first few questions, but forgets about the requirement later when it's prompted for actions. Quantitatively, the model performs  $65\%$  worse with  $90\%$  variance without the DAG. The introduction of DAG eliminates this problem by reducing the QA context length to only a question's immediate parents.
Overall, SPRING achieves the best performance and a small  $6\%$  performance standard deviation, due to more consistent reasoning over time steps with better focus and fewer distractions.
LLM In the last two rows of Table 3, we show that the same architecture does not work well with GPT-3.5-turbo. We believe the observed  $73\%$  performance gap mainly comes from GPT-3.5-turbo's worse performance at following fine-grained instructions in each of the questions, which are required for chain-of-thought reasoning with SPRING.
3.4 Cost for running SPRING
The number of queries per step is 9 (same as the number of questions). Each game could take around 300 steps, but can go up to 500 steps in the worst case. Therefore, the maximum number of queries per game can go up to 4500. According to the public price of GPT-4 API, each query costs around  $0.06^{5}$ . The total cost should be less than 270 (USD) per game with GPT-4. Given the progress in chip-set development, we are hopeful that the inference costs will lower, making LLMs more accessible for the public.
3.5 Potential for Benchmarking LLMs
In Table 4, we compare popular publicly available LLMs including GPT-4 (OpenAI, 2023), GPT-3.5 (text-davinci-003) (OpenAI, OpenAI), Bard (Manyika, Manyika), Claude (Anthropic, Anthropic), Alpaca-30b (Taori et al., 2023) under the same setting on Crafter, following the same step-by-step prompt as Section 3.3 and Table 3. We observe a clear separation in performance under our setting.
RL vs. LLMs Comparing LLM-based agents against RL agents brings forth a intriguing discussion. RL algorithms do not require prior knowledge such as instruction manuals, and could
5 Pricing estimated according to OpenAI API as of October 17 2023
Method	Achievement Depth	Reward	Questions per Step
Step-by-step prompt + GPT-4	5	7.3 ± 4.4	2
Step-by-step prompt + text-davinci-003	4	4.5 ± 2.1	2
Step-by-step prompt + Bard	0	-0.9 ± 0	2
Step-by-step prompt + Claude	1	0.1 ± 0.1	2
Step-by-step prompt + Alpaca-30b	1	0.1 ± 0.1	2
Random	1	2.1 ± 1.3	0
Table 4: Comparison of different LLMs under the same setting using the context  $C$  generated with text-davinci-003 following the same step-by-step prompt as Section 3.3 and Table 3.
continually improve given enough trials. However, RL algorithms are typically trained with reward functions deliberately engineered to cover all in-game achievements (Hafner, 2021; Hafner et al., 2023). Such reward functions often require a lot of expert knowledge and careful formulation.
On the other hand, LLM agents like SPRING does not need the reward (we report reward for comparison purpose, SPRING does not use the reward during inference), but instead uses external knowledge from the LATEX source code. In addition, current LLM agents lack the capabilities of improving from interactions.
We hope future works would be able to leverage the benefits of both paradigms in order to achieve efficient planning with fine-grained control.
Policy Informed by Natural Language Instructions In the instruction following setting, step-by-step instructions have been used to generate auxiliary rewards, when environment rewards are sparse. Goyal et al. (2019); Wang et al. (2019) use auxiliary reward-learning modules trained offline to predict whether trajectory segments correspond to natural language annotations of expert trajectories.
There has been many attempts to go beyond instruction following to learning from unstructured natural language (Branavan et al., 2012; Goldwasser and Roth, 2014; Zhong et al., 2021; Wang and Narasimhan, 2021). Zhong et al. (2021); Wang and Narasimhan (2021) make use of special architectures to learn reasoning on grid worlds with template-generated instructions. However, the model requires 200 million training samples from templates identical to the test environments. Such a training requirement limiting the generalization of the model and causes performance loss even on slightly bigger grid worlds with identical mechanics.
Wu et al. (2023) proposes a summary (Read) and reasoning (Reward) through a QA prompting framework with an open-source QA LLM (Tafjord and Clark, 2021). The framework demonstrates the possibility of an using real-world human-written manuals to improve RL performance on popular games, despite limiting the interaction types to only "hit". Our framework handles all 17 kinds of interactions available in the game. Moreover, our framework makes use of information on tech-tree dependencies, and suggestions on desired policies extracted from the academic paper.
LLMs for Planning LLMs have shown promising results at high-level planning in indoor embodied manipulation environments. Huang et al. (2022); Ahn et al. (2022) primarily explores generating plans for embodied tasks, with limited actions space and trajectory length. Song et al. (2022); Wu et al. (2022) enhances Ahn et al. (2022) with greater action diversity and real-time re-planning. However, a lot of the high-level plans lack executability and has to be post-processed to meet specific task requirements, thus limiting the generalization to complex open world tasks. In addition, all prior works along this line operates on few-shot human/expert generated demonstrations containing up to 17 trajectories to provide context for LLMs, which requires more manual labor, and may limit the generalization to unseen scenarios. In comparison, our SPRING framework requires no demonstration.
LLMs for Open World Games Compared to popular indoor manipulation tasks, planning in open-world game environments poses the following additional challenges. 1) Long horizon. Due to the nature how in-game achievement/technology progresses, a successful gameplay can easily go beyond 200 steps (Hafner, 2021). 2) Parallel objectives. Open-world environments contain objectives that can be pursued in parallel and often require prioritization (Wang et al., 2023). Therefore, open world games are significantly more challenging than current indoor embodied manipulation environments.
Du et al. (2023) applies LLMs as high-level planners to assist RL exploration in Crafter. Wang et al. (2023); Yuan et al. (2023) use LLMs as high-level planner and goal selector to control a low level-policy in Minecraft. Tsai et al. (2023) studies the capabilities of ChatGPT on text games. Notably, all prior works require expert or human generated example trajectories as context for the LLMs. Since the example trajectories do not cover all scenarios, all prior works may encounter unseen situation during evaluation, leading to an overall performance inferior to state-of-the-art RL algorithms (Hessel et al., 2018; Guss et al., 2021; Hafner et al., 2023), trained without the use of LLMs. To our knowledge, we are the first to show an LLM (GPT-4) achieving performance surpassing the state-of-the-art RL algorithms in a challenging open world game.
A primary limitation in using an LLM to support interaction with the environment is the need for object recognition and grounding. However, these limitations do not exist in environments that offer accurate object information, such as contemporary games (Fan et al., 2022) and virtual reality worlds (Kolve et al., 2017). While pre-trained visual backbones (He et al., 2017) perform poorly on games, they have shown reasonable performance for environments closer to the real-world (Shridhar et al., 2020). In addition, with recent progress on visual-language models (Bubeck et al., 2023; Driess et al., 2023; Liu et al., 2023; Zou et al., 2023), we believe there will be reliable and generalizable solutions to visual-language understanding in the foreseeable future. Future works could focus on addressing the requirement for a separate visual descriptor with large visual-language models.
In this work, we explore solving the Crafter (Hafner, 2021) RL benchmark using the latest LLMs by reading the LATEX source code of an academic paper about the benchmark. We study the quality of in-context "reasoning" and "planning" induced by different forms of prompts under the setting of the Crafter open-world environment. To enforce consistent planning and execution over hundreds of environment steps, we introduce SPRING, an innovative prompting framework for LLMs designed to enable in-context chain-of-thought planning and reasoning. Quantitatively, SPRING with GPT-4 outperforms all state-of-the-art RL baselines, trained for 1M steps, without any training.
Our work demonstrates the reliability of LLMs for understanding and reasoning with human knowledge. We hope that our work points to a new way of integrating human prior knowledge into RL training through intrinsic rewards (Wu et al., 2023), hierarchical RL (Shu et al., 2017), or sub-goal planning (Wang et al., 2023; Wu et al., 2023).
Our research on LLM holds potential for both positive and negative impacts. The benefits include better understanding of the powers of LLM and enhanced integration of prior knowledge, which could lead to advancement in various AI topics. However, the risks may involve reliance on computationally demanding models, game cheating or exploitation, and reliance on prior knowledge.
Ahn, M., A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng (2022). Do as i can, not as i say: Grounding language in robotic affordances.
Ahn, M., A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. (2022). Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691.
Anthropic. Introducing claude. https://www.anthropic.com/index/introducing-claude. Accessed: May 27, 2023.
Tafjord, O. and P. Clark (2021). General-purpose question-answering with macaw. arXiv preprint arXiv:2109.02593.
Taori, R., I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto (2023). Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. Accessed: May 27, 2023.
Tsai, C. F., X. Zhou, S. S. Liu, J. Li, M. Yu, and H. Mei (2023). Can large language models play text games well? current state-of-the-art and open questions. arXiv preprint arXiv:2304.02868.
Vinyals, O., I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. (2019). Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature 575(7782), 350-354.
Wang, H. and K. Narasimhan (2021). Grounding language to entities and dynamics for generalization in reinforcement learning. arXiv preprint arXiv:2101.07393.
Wang, X., Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y.-F. Wang, W. Y. Wang, and L. Zhang (2019). Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6629-6638.
Wang, Z., S. Cai, A. Liu, X. Ma, and Y. Liang (2023). Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560.
Wei, J., M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le (2021). Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
Wu, Y., Y. Fan, P. P. Liang, A. Azaria, Y. Li, and T. M. Mitchell (2023). Read and reap the rewards: Learning to play atari with the help of instruction manuals. arXiv preprint arXiv:2302.04449.
Wu, Y., S. Y. Min, Y. Bisk, R. Salakhutdinov, A. Azaria, Y. Li, T. Mitchell, and S. Prabhumoye (2023). Plan, eliminate, and track-language models are good teachers for embodied agents. arXiv preprint arXiv:2305.02412.
Wu, Y., S. Y. Min, Y. Bisk, R. Salakhutdinov, and S. Prabhumoye (2022). Tackling alfworld with action attention and common sense from language models.
Yuan, H., C. Zhang, H. Wang, F. Xie, P. Cai, H. Dong, and Z. Lu (2023). Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303.16563.
Zhong, V., A. W. Hanjie, S. I. Wang, K. Narasimhan, and L. Zettlemoyer (2021). Silg: The multi-environment symbolic interactive language grounding benchmark. arXiv preprint arXiv:2110.10661.
Zou, X., J. Yang, H. Zhang, F. Li, L. Li, J. Gao, and Y. J. Lee (2023). Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718.
A Example Trajectory
$= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Step: 0, Cumulative Reward: 0.0$
You took action make_wood_sword.
You see:
- tree 5 steps to your north-east
- cow 6 steps to your north-west
You face grass at your front (south).
Your status:
-health:9/9
-food:9/9
drink: 9/9
energy: 9/9
You have nothing in your inventory.
Question: Action
Move North
Question: List objects in the current observation. For each object, briefly answer what resource it provides and its requirement.
1. Tree: Provides wood which is used to craft tools and structures like the table. Requirement: An axe or hands to chop down the tree.
2. Cow: Provides food when killed, restoring your health points. Requirement: A weapon like a sword to kill the cow or hands to punch it.
3. Grass: Generally provides no resources, but it can occasionally drop seeds for planting. No requirements to interact with grass.
Question: What was the last action taken by the player?
The last action taken by the player was "make_wood_sword."
Question: For each object in the list, are the requirements met for interaction?
1. Tree: Yes, the requirements are met as you can use your hands to chop down the tree, even though it will be slower than using an axe.
2. Cow: Yes, the requirements are met as you have crafted a wood sword to kill the cow, which is more efficient than using your hands.
3. Grass: Yes, there are no specific requirements to interact with grass, and you can simply walk on it or punch it with your hands if necessary.
Question: List top 3 sub-tasks the player should follow. Indicate their priority out of 5.
1. Chop down the tree to collect wood (Priority: 5/5) - Wood is essential for crafting more tools and structures, as well as potentially making a shelter.
2. Kill the cow for food (Priority: 4/5) - Ensuring a sufficient food supply is crucial for maintaining health and being able to accomplish other tasks.
3. Explore the environment (Priority: 3/5) - It is essential to gain more information about available resources and potential dangers in the surrounding area so that you can plan effectively.
Priority level is indicated out of 5, where 5 is the highest priority and 1 is the lowest priority.
Question: What are the requirements for the top sub-task? What should the player do first?
Requirements for the top sub-task (Chop down the tree to collect wood):
1. Approach the tree: Navigate to the tree located 5 steps to your north-east.
What the player should do first:
1. Move North-East: Move in the direction of the tree to eventually reach it and then prepare to chop it down.
Question: List top 5 actions the player should take and the requirement for each action. Choose ONLY from the list of all actions. Indicate their priority out of 5.
Priority 1: Move North - Requirement: Flat ground above the agent.
Priority 2: Move East - Requirement: Flat ground right to the agent.
Priority 3: Move East - Requirement: Flat ground right to the agent.
Priority 4: Do (Chop down the tree) - Requirement: Facing the tree; have necessary tool (wooden sword can be used, but a wooden pickaxe would be preferred).
Priority 5: Collect Wood - Requirement: Successfully chopped the tree, and wood is available to pick up.
Question: For each action in the list, are the requirements met?
Priority 1: Move North - Yes, the requirement is met as flat ground is above the agent.
Priority 2: Move East - Yes, the requirement is met as flat ground is available to the right of the agent.
Priority 3: Move East - Yes, the requirement is met as flat ground is available to the right of the agent.
Priority 4: Do (Chop down the tree) - The requirement is partially met; you are facing the tree and have a wooden sword, but a wooden pickaxe is preferred for chopping down trees.
Priority 5: Collect Wood - No, the requirement is not met yet, as you have not successfully chopped down the tree or accessed the wood.
Question: Did the last player action succeed? Why?
I cannot determine if the last player action succeeded or not because there is no information provided about the result of the action "make_wood_sword." Normally, if the action was successful, the player would have a wood sword in their inventory or an updated game state would include the new item.
Question: Choose the best executable action from above.
The best executable action from the above list is Priority 1: Move North, as the requirement of having flat ground above the agent is met.
$= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Step: 1, Cumulative Reward: 0.0$
You took action move_north.
You see:
- tree 4 steps to your east
- cow 6 steps to your north-west
You face grass at your front (north).
Your status:
-health:9/9
- food: 9/9
- drink: 9/9
energy: 9/9
Move East
1. Tree: Provides wood, requires a wood or better pickaxe to collect.
2. Cow: Provides meat (food), requires a sword or other weapon to defeat and collect the resource.
3. Grass: No direct resources, can be used to build a shelter or pathway with no specific tool requirements.
The last action taken by the player was move_north.
1. Tree: Yes, your recent action was making a wood sword so you can interact with the tree to collect wood.
2. Cow: Yes, you have a wood sword to defeat the cow, so you can interact with it.
3. Grass: Yes, there are no specific tool requirements to interact with the grass.
1. Collect wood from the tree. Priority: 5/5
2. Defeat the cow and collect meat (food). Priority: 4/5
3. Interact with grass to clear a path or build a shelter. Priority: 3/5
The requirements for the top sub-task (collect wood from the tree) are to have a wood pickaxe in the inventory and be close enough to the tree.
The player should first move closer to the tree located 4 steps to the east. Once the player is nearby, they should use their wood pickaxe to collect wood from the tree.
1. Priority 1: Move East (Requirement: Flat ground to the right of the agent) - to get closer to the tree.
2. Priority 2: Move North (Requirement: Flat ground above the agent) - to be in front of the tree.
3. Priority 3: Make Wood Pickaxe (Requirement: Nearby table; wood in inventory) - to be able to collect wood from the tree.
4. Priority 4: Do (Requirement: Facing tree or material; have a wood pickaxe) - to collect wood from the tree.
5. Priority 5: Make Wood Sword (Requirement: Nearby table; wood in inventory) - to defend yourself against potential threats (such as monsters).
1. Priority 1: Move East (Flat ground to the right of the agent) - Yes, the requirement is met as the agent is on flat ground.
2. Priority 2: Move North (Flat ground above the agent) - Yes, the requirement is met as the agent is on flat ground.
3. Priority 3: Make Wood Pickaxe (Nearby table; wood in inventory) - No, the requirement is not met as the agent does not have a nearby table and wood in their inventory.
4. Priority 4: Do (Facing tree or material; have a wood pickaxe) - No, the requirement is not met as the agent does not have a wood pickaxe in their inventory.
5. Priority 5: Make Wood Sword (Nearby table; wood in inventory) - No, the requirement is not met as the agent does not have a nearby table and wood in their inventory.
Yes, the last player action (move_north) succeeded because there was flat ground available to the north of the player's position, allowing them to move in that direction.
The best executable action from the list above is Priority 1: Move East, as the requirement of flat ground to the right of the agent is met.
\(= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Step: 2, Cumulative Reward: 0.0 \(= \equiv\) \(= \equiv\) \(= \equiv\) \(= \equiv\) You took action move_east.
- tree 3 steps to your east
- cow 4 steps to your east
You face grass at your front (east).
- health: 9/9
- food: 9/9
drink: 9/9
energy: 9/9
Question: List objects in the current observation. For each object, breifly answer what resource it provides and its requirement.
1. Tree: Provides wood when chopped down; requires having a wood pickaxe or better to chop down.
2. Cow: Provides food (meat) when killed; requires having a wood sword or better to kill.