LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
Authors: Zhuoling Li
Xiaogang Xu
Zhenhua Xu
Ser-Nam Lim
Hengshuang Zhao
Venue: ICML 2025 poster (2025)
Keywords: large auto-regressive model
referee reinforcement learning
embodied intelligence
Primary area: reinforcement_learning->planning
TL;DR: We introduce the large auto-regressive model and referee reinforcement learning algorithm.

Abstract: Recent embodied agents are primarily built based on reinforcement learning (RL) or large language models (LLMs). Among them, RL agents are efficient for deployment but only perform very few tasks. By contrast, giant LLM agents (often more than 1000B parameters) present strong generalization while demanding enormous computing resources. In this work, we combine their advantages while avoiding the drawbacks by conducting the proposed referee RL on our developed large auto-regressive model (LARM). Specifically, LARM is built upon a lightweight LLM (fewer than 5B parameters) and directly outputs the next action to execute rather than text. We mathematically reveal that classic RL feedbacks vanish in long-horizon embodied exploration and introduce a giant LLM based referee to handle this reward vanishment during training LARM. In this way, LARM learns to complete diverse open-world tasks without human intervention. Especially, LARM successfully harvests enchanted diamond equipment in Minecraft, which demands significantly longer decision-making chains than the highest achievements of prior best methods.

LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
Zhuoling Li
Xiaogang Xu² Zhenhua Xu Ser-Nam Lim" Hengshuang Zhao
https: http:///izzuoiingo gith.b..hhhh... gwthtb.../LARM htt.i///__aaaaaaaaeeeeeee
Abstract
specific tasks such as object grasping and indoor naviga- tion (Savva et al., 2019). While notable advancements are achieved, these agents tend to be specialist models confined to isolated tasks (Huang et al., 2023). To overcome this limi- tation, recent studies, including this work, employ Minecraft (Baker et al., 2022; Fan et al., 2022; Guss et al., 2019; Wang et al., 2024; Li et al., 2023) benchmark to explore em- bodied agents with open-ended objectives and long- long-horizon reasoning chains.
Recent embodied agents are primarily built based on reinforcement learning (RL) or large language models (LLMs). Among them, RL agents are efficient for deployment but only perform very few tasks. By contrast, giant LLM agents (of- ten more than 1000B parameters) present strong generalization while demanding enormous com- puting resources. In this work, we combine their advantages while avoiding the drawbacks by con- ducting the proposed referee RL on our developed large auto-regressive model (LAARM). Specifi- cally, LARM is built upon a lightweight LLM (fewer than 5B parameters) and directly outputs the next action to execute rather than text. We mathematically reveal that classic RL feedbacks vanish in long-horizon embodied exploration and introduce a giant LLM based referee to handle this reward vanishment during training LARMM. In this way, LARM learns to complete diverse open-world tasks without human intervention. Es- pecially, LARM successfully harvests enchanted diamond equipment in Minecraft, which demands significantly longer pacision-making chains than the highest achievements of prior best methods.
The early methods for developing such agents primarily rely on reinforcement learning (RL) (Fan et al., 2022). Due to the limited exploration efficiency of RL, these methods require careful reward engineering for different tasks, and the de- rived RL policies can mostly only complete a single simple task (Yuan et al., 2023). The advantage of RL policies is that they are usually lightweight for real-time deployment. Dif- ferently, recent embodied works begin to investigate large language models (LLMs) (Brown et al., 2020). Owing to the extensive general knowledge and formidable reasoning capabilities of LLMs, these methods demonstrate promis- ing results with significantly reduced domain-specific engi- neering efforts (Wang et al., 2023a). Nevertheless, LLMs continue to exhibit several limitations. First of all, the out- puts of LLMS are usually sentences or code (Zhao et al., 2024) generated through iterative token prediction, necessi- tating N inference operations for N tokens. Therefore, the response speeds of LLMs are restricted. Secondly, recent research suggests that huge model size is important for an LLM generalize well (Achiam et al., 2023), while the computing resource for embodied agents is usually very lim- ited, Our analysis reveals that while giant LLMs with more than 1000B parameters like GPT-4 (Achiam et al., 2023) can answer questions about exploration and crafting issues in Minecraft well, the performance of lightweight LLMs such as LLaVA-7B (Liu et al., 2024a) is limited.
1. Introduction
In recent years, remarkable progress has been achieved in various artificial intelligence (AI) topics (LeCun et al., 2015) like computer vision (He et al., 2016) and natural language processing (Kenton & Toutanova, 2019), but most of them lack the capacity to physically interact with the real world. To address this disconnect, the concept of embodied Al is introduced (Chrisley, 2003). Early embodied agents are predominantly developed on simulation platforms for
As illustrated in Fig. 1, we aim to combine the advantages of both RL methods and LLM methods while avoiding their drawbacks. To this end, we first propose Large Auto- Regressive Model (LARM), the main body of which shares the same structure as lightweight LLMs like TinyLLa VA (Zhou al., 2024). This choice enables us first pre-train it utilizing numerous webpage data to provide it with ba- sic general knowledge. Taking environmental observation
'The University of Hong Kong "The Chinese University of Hong Kong TTinnguu University University of Central Florida. Correspondence to: Hengshuang Zhao <hszhao@ cs. hk..
Proceedings of the 4 International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
as input, LARM predicts the next action to perform in auto-regressive manner. Instead of generating descriptive sentence composed of multiple tokens, LARM directly pro- duces a single token to select the next action, which makes LARM respond more swiftly than common LLMs.
practice is devising the policy into hierarchical architec- ture, where complex task is first decomposed into many simple sub-tasks. Different models are trained for various sub-tasks and a leader model is built to decide the order of performing these sub-tasks (Liu et al., 2024b).
The following problem is how to train LARM. We find that classic RL algorithms cannot train LARM effectively and mathematically reveal this is because the reward feedback gradually vanishes in long-horizon embodied exploration. This phenomenon can be empirically understood as even though a policy selects the correct action, it obtains positive feedback only after the target task is completed, meaning many iterations of delay. In addition, any wrong decision- making in future iterations will cause the policy to get no positive reward, which hides the value of the current correct action. To handle this problem, we introduce referee RL. Its core idea is that we employ a referee (like a giant LLM) to provide immediate feedback about whether the just per- formed action brings positive contribution to realizing the final target. In this way, we efficiently distill the concerned generalizable knowledge of giant LLMMs into our lightweight end-to-end LARM policy during online exploration with- out human supervision. This marks the first attempt that optimizes LLM-style embodied policy through making it directly interact with the environment online.
Due to its open-world characteristic, Minecraft is suitable for exploring how to develop open-ended embodied intelli- gence (Feng et al., 2024). To concentrate on studying this problem, there are plentiful works that take skill (e.g., chopping down a tree or crafting table) as the basic model output (Wang et al., 2023a). The skill could be modeled as a well-trained policy based on reinforcement learning (Yuan et al., 2023), an action generation model based on language prompt (Lifshitz et al., 2023), or provided APIs. Among these works, LLM-based methods achieve the most impres- sive results thanks to their rich general LLM knowledge (Achiam et al., 2023; Wang et al., 2023b), especially for giant LLMs with more than 1000B parameters like GPT-4. However, due to the huge model sizes, these LLMs can only be deployed in remote computing clusters. There are also works that try tuning a lightweight L.LM like LLaMA using Minecraft relevant text and then prompting the tuned LLM to say what skill should be performed in inference (Feng et al., 2024). However, the text used for tuning contains much irrelevant information and is not specialized for task execution. As the embodied text tuning data volume is also limited, the tuned LLMs often fail to describe what skill should be performed correctly.
We validate our method in both MineDojo (Fan et al., 2022) and Mineflayer (PrismarineJS., 2013) environments. The experimental results suggest that our method completes di- verse challenging tasks with a single model, indicating its promising generalization. LARM achieves higher success rates than previous counterparts, although these counter- parts may employ a special network for each task. Notably, LARM is the first method that harvests enchanted diamond equipment in Minecraft. In addition, evaluated with an RTX4090 GPU, LARM runs with a speed of 0.58 second per inference, which meets the speed requirement of online high-level action scheduling.
Large language models. LLMs draw broad attention from the research and industrial communities due to their rich general knowledge and the ability to generate the answers to diverse kinds of questions (Chang et al., 2024). GPT- 3 emerges as a milestone in the evolution of L.LMs, as it takes the next token prediction problem as the pre-training task and showcases remarkable open-world generalization capabilities (Brown et al., 2020). Subsequently, the fine- tuning of GPT-3 using reinforcement learning with human feedback leads to the creation of ChatGPT (OpenAI, 2023) and GPT-4 (Achiam et al., 2023). However, a significant limitation of LLMs is their inability to interpret information in images, which are vital for humans to perceive the world. To overcome this problem, researchers devise strategies that inject vision information into LLMs and enable LLMs to perceive images. A common method is fine-tuning a small number of network parameters using numerous language- image data pairs to bridge the representation gap between text and images (Ding et al., 2023). In this way, some large vision-language models like LLaVA (Liu et al., 2024a) and Flamingo (Alayrac et al., 2022) are derived.
2. Related Work
Minecraft agents. Compared with other embodied bench- marks, Minecraft is an open-ended platform suitable for exploring building agents with long-horizon planning ca- pabilities (Fan al., 2022). It simulates diverse weather, biomes, and mobs in an unlimited 3D virtual world. Early methods in Minecraft are mostly based on reinforcement learning (Frazier& & Riedl, 2019) or imitation learning (Baker et al., 2022). Their model outputs are atom actions, e.g., a short movement, mouse click, or keyboard press. However, due to the huge decision space, such atom-based agents are quite challenging for optimization. Thus, these works pay their main attention to devising strategies for alleviating the optimization complexity (Scheller et al., 2020). An effective
RL with LLMs in embodied AL RL can search promis- ing decision-making policies without human intervention, and LLMs are able to provide suitable search start points

---

LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
Efficient but Task specialized
Efficcintt and
Generalizoble but slow
Generolizoble
Agent
LLM Agent
LARM Agent (Ours)
JU
Common RL Agent Achievements
Common LLM Agent Achievements
LARM Agent Achievements
More Advanced Achievements
Figure 1. Comparison among agents based on RL, LLM, and LARM. As shown, RL agents are usually task specialized, and LLM agents are computationally expensive to deploy. By contrast, the LARM agent is efficient and generalizable. Besides, LARM presents better performance. As shown, LARM is the first method that achieves enchanted diamond equipment in Minecraft.
based on their rich general knowledge (Laleh & Ahmad- abadi, 2024). Therefore, it is natural to design ways to combine them to build advanced embodied intelligence. In previous works, a popular choice is training a network for each basic skill based on RL, and then prompting the LLM to say what skill should be used according to the task target and environment observation (Wang et al., 2023b). Never- theless, this paradigm is not only slow, it requires the LLM to have sufficient knowledge about Minecraft. According to our analysis, giant LLMS own such an ability but the capa- bilities of lightweight LLMs are limited. Another possible choice utilizing L.LM to generate the code for calculating RL reward (Xie al., 2024). Nevertheless, it is not always feasible to define reward function by writing code. For ample, in Minecraft, the information is represented as image and agent status information, which cannot be mapped as reward based on rules. To handle this problem, we propose to directly employ GPT-4 to read the agent status before and after executing a skill and judge whether the outcome brought by this skill contributes realizing the given target. In addition, to the best of our knowledge, this is the first work that directly optimizes an LLM-style policy based online exploration and reinforcement learning. Our results suggest that the rich general knowledge in LLM favors this exploration and self-learning process.
and is effectively framed as a Markov Decision Process symbolized by a tuple m = A, T, T). Specif- ically, S is the set of all potential states. A is the action set, and every action is also called as a skill this work. P : S x A X + [0, 1] represents a probability distribution that governs the state transitions given states and actions. T is the set of all task targets. R : S + R, 7, and T de- note the reward function, discount factor, and initial state distribution, respectively. At any discrete time step t, the environment resides in a state St E S, and the correspond- ing observation o, by a policy π is function of this state, expressed as o = [(t).. This observation o, is then utilized to select the subsequent action according to at ~ ( 4), where ag A and E T denotes the given target task.
In tackling the studied long-horizon embodied task, the ob- jective is to navigate through sequence of intermediate states T, $1, $...... ST-1 to ultimately reach the target state ST at the final time step T. This requires the policy to gener- ate a series of actions ag, 01, ..... or-1 such that each action a transitions the environment from state 8, to the next state $t+1 correctly, adhering to the dynamics prescribed by the transition probability distribution P. It is crucial that each intermediate state 8t is accurately achieved in sequence to ensure the policy attains the target state 8T.
3.2. PPO
3. Preliminary
Proximal Policy Optimization (PPO) (Schulman et al., 2017) is a model-free reinforcement learning algorithm widely adopted for training policies in complex environments, and our referee RL. is developed based on this algorithm. A PPO
3.1. Problem Formulation
What we study in this work can be conceptualized an auto- regressive prediction problem involving long sequences,

---

LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
policy и mainly consists of two components, the actor Va and critic Te- *e is estimate the value function Vo, (8t), the expected cumulative discounted reward starting from state and following т. The optimization objective of is as follows:
Algorithm 1 Referce RL
Require: Target task 1.
1: Initialize the actor Fa critic Tes referee Tr
2: Initialize policy exploration step T, policy update steps Nr
3: for each iteration iter do
L.. = (s) (r: + yVo, (sc+1))))
(1)
4: Initialize data buffer B +- 0
5: for t= to 7 do
where T2 E R. is the reward received after taking action a, in state St. To further reduce the variance of the value estimate and improve stability, PPO employs the generalized advantage estimation (GAE), which defined as:
6: Get the actor observation or + f(x)
Get state S1+1 and environment reward ra by tak- ing ag app(0;
8:
Get auxiliary reward P. t- TT,(,, St, ae, a++1)
9: Add transition B +- BU Ut, aes F1, Pt)
T-1 A, = EGTA) 5x.
10:
end for
11:
for n = 1 N₂ do
Jema
12:
Sample a random training data batch
where À is factor balancing between temporal differ- ence (TD) learning and Monte Carlo estimation, and 6g denotes the TD-error, which is formulated as 6% = rh + VV,, (sx+1) (sx). With A, the objective function for the critic na can be given as:
{(1,, 0t, de; 8t41, Ft, ft) ~ B
13:
Optimize Ta and with respect to Eq. (1)~(4)
14:
end for
15:
end for
Ta (ae (82) E,
T.g.
Ar]:
(3)
task. As we train the critic Tr using the very imbalanced
Ta Told (as) st))
reward set {rw] described in Eq. (5) with respect the optimization objective in Eq. (1), can infer that the output
where and wold is the old actor policy before weight update and na (ay]s;) is the current actor. To improve the training stability, PPO further develops clipped surrogate objective based on Eq. (3), which can be formulated as:
of q. gradually converges to Vo, (sx) VVo. (sk+1) R -E.
In this way, the corresponding TD error set (d: (= is:
T-1
0, if
T T
(6)
igg. [mimin(k A2, (ke, 1 - e, +c),)), (4)
R
where k, Ta (eclee)
- (a,ls.) and is small positive parameter. By optimizing ne and Ta with respect to Eq. (1) and Eq. (4), the policy gradually learns to perform the target task.
According to the definition of GAE in Eq. (2), we can observe that the first T as the elements in )T are nearly zero. For the last item (sA)T-1-tō7-1 we have ( because 7 E 10, and A E (0. Hence, when the task needs long-horizon action chain execution, the obtained GAE value A, is close to zero, which suggests that the opti- mization objective for training Ta in Eq. (4) becomes zero. In this way, even though the policy makes the right action decision, it cannot get any positive feedback.
/ items are close zero
- 0
4. Method
4.1. Referee Reinforcement Learning
In long-horizon embodied task exploration, the policy can usually only get positive feedback after the target task is completed successfully (Fan et al., 2022). Following the notations in Section 3.1, we can assume that there is an exploration trajectory {sk, og. T where T is a large integer, and 8k, ag. and rk denote the state, action, and environment reward at the k step, respectively. The agent completes the target task at the final step T. Therefore, we can get that:
To handle this problem, we introduce referee RL. Specif- ically. we introduce a referee Tp to provide an auxiliary reward feedback to the trained policy n, and this reward at the step & represented as ft. = =Tp(t, 8k; 0k, 81+1). This means Tp takes the target task information i, initial state sk- selected action ag, and new state s as input and provides feedback based on whether the selected action is correct and outcome caused by this action. In this work, we split the feedback into four categories: (a) The selected action is correct and brings a positive outcome to realizing the target. (b) The selected action is correct but does not bring positive outcome. (c) The selected action is incorrect but
[ e,, 1 ... TT-1 R if
Fk
(5)
where -E denotes a small negative constant due to time penalty and R is the positive reward of completing the target

---

LARM: Large Auto- Rebrosserr Model for Long- Hong-Horizon Embodied Intelligence
LARM Policy
LoRA
Environment Exploration
Task Description
Agent Status:...
Text Observation
Actor Head
Action
MIHECRA
LLM Decoders
Agent
Environment
Critic Head
Value
Vision Observation
Skill Token
Agent State
Environment Reward
Referee
Update
Auxiliary Reward
Total Reward
based Policy Optimization
Reward Generation
Figure The overall pipeline our method. As illustrated, we parametrize actor T,, and critic Tं using single LARM model with two separate prediction heads, ie.. the action head and critic head. We train LARM based proposed referee RL algorithm, which utilizes both environment feedback and referee generated auxiliary reward to guide the optimization of LARM.
does not lead to negative outcome. (d) The selected action is incorrect and results in negative outcome. For the four categories, Ty correspondingly returns the reward value r", rh, r9, and rd, where nd > pb > 0 > r > pd By adding this auxiliary reward F to the original reward described in Eq. (5), the feedback to ı before T does not remain as the constant s. this way, the TD error òg and correspond- ing GAE value A, do not converge zeros, being able to provide effective optimization guidance to Ta-
design has two key benefits: (i) The model is initialized with the general knowledge and reasoning ability of LLMs while maintaining an acceptable parameter volume. In this way, LARM can be deployed based on the restricted computing resources embodied applications and achieve real-time response. (ii) As LARM adopts a similar model architecture as LLMs, we can first pre-train it using numerous question- answer data related to the concerned embodied AI topics. This kind of data is much easier to collect and scale up than embodied data with action execution. We have tried pre-training LARM using a 34G webpage dataset crawled from Wiki (Fan et al., 2022), and the results indicate that the training convergence improved.
In this work, we model the referee Tp based on GPT-4 (Achiam et al., 2023), which is a giant LLM and owns extensive generalizable knowledge. As mentioned before, the information provided GPT-4 includes i (target task description), 0g (the executed skill), 8ı and 8k4-1 (the inven- tory list and environment resource surrounding the agent before and after executing ag), and then we prompt it to judge the situation and response a reward among r", rb, re, and rd The full detailed procedure of the referee RI. algorithm is elaborated in Algorithm I.
The input to the LARM model consists of four parts, i.e., task description, text observation, vision observation, and skill token. The task description specifies the target task to conduct. The text information primarily includes the in- ventory list, historical action, and blocks surrounding the agent. The vision information is the real-time image per- ceived by the agent. We encode text and image information tokens based CLIP (Radford et al., 2021), and these tokens with additional learnable skill token input to the LARM decoders to conduct feature interaction. After the decoders, the skill token is input to the action head and critic head depicted in Fig. 2 to output the action and state value. Therefore, LARM parametrizes the actor тa and critic n, in Section 4. 1. based on a single model with two different trainable prediction heads.
4.2. Large Auto-Regressive Model
In this part, we explain how to parametrize स. and w. using our designed LARM policy. As shown in Fig. 2, the main body of LARM is the decoders of lightweight decoder- only LLM, TinyLLaVA VA-3..1 in this work. The parameters of these decoders frozen during training and trainable LoRA (Hu et al., 2021) module is applied to help the model learn new knowledge in the applied task domain. This

---

LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
Similar to previous literature (Wang et al., 2023a; Liu al., 2024b), the action predicted by LARM is skill, such as chopping down a tree or searching for a cobblestone. LARM selects a skill to perform by conducting feature matching between the action head output and skill description like STEVE (Zhao et al., 2024). In this work, we test LARM with two kinds of skills, the RL-based skills in MineDojo and API-based skills based on Mineflayer. The former cat- egory is easier to be extended to real-world applications and the latter class presents higher execution success rates. According to the given task requirements, new skills can be dynamically added based on the strategies developed in previous works (Wang et al., 2023a) and how to generate these skills not the research focus of this work.
action details. In this way, several significantly more ad- vanced achievements obtained by previous works based on Mineflayer, like harvesting diamonds with high success rates ( (Wang et al., 2023a). In addition, the agents are usually spawned without initial tools in Mineflayer.
5.2. Main Results
We compare LARM with previous methods in this part. As the basic actions in MineDojo and Mineflayer are different, we conduct the comparison separately.
Comparison on MineDojo. The methods compared on MineDojo include MineAgent (Fan et al., 2022), Plan4MC (Yuan al., 2023), LLaMA-Rider (Feng et al., 2024), and RL-GPT (Liu et al., 2024b). Among them, MineAgent is the baseline method provided by Minddojo. It first fine-tunes CLIP (Radford et al., 2021) based on numerous web data and uses the fine-tuned CLIP to guide the training of rein- forcement learning algorithms. PIan4MC is a reinforcement learning based method. It splits a task into basic skills and trains agent to learn them one by one in hierarchical way. LLaMA-Rider is an LLM obtained by fine-tuning LLaMA. It first makes the agent explore the environment to collect data. Then, the collected data is adopted to fine-tune LLaMA in a supervised manner. RL-GPT builds two LLM based agents (a slow agent and a fast agent) to schedule the agent actions. For an action step. this method first queries the agents whether this action step can be completed via code generation. If the code generation is infeasible, an RL based action is performed.
Combining the aforementioned techniques, LARM is op- timized by iterating between environment exploration and policy update described in Algorithm 1. The training de- tails like the optimizer choice follow PPO (Schulman et al., 2017). For learning to complete the most challenging task in this work (craft an enchanted diamond tool), about 42 hours of exploration is taken using a single RTX4090 GPU.
5. Experiments
5.1. Environment
In this work, we validate our method using the MineDojo and Mineflayer environments. Both these two environments are developed based on Minecraft, but there exist differences in their testing protocols. In addition, the commonly used strategies of generating skills in these two environments are also different. For fair comparison, we compare our method with the counterparts in these two environments separately using the same testing protocol and skill generation strategy.
We compare LARM with these methods on diverse tasks, and the detailed settings of these tasks follow previous works (Feng et al., 2024). Notably, we train a single LARM model to complete all the tasks, which is different from many pre- vious works that employ separate models for various tasks. This characteristic suggests the promising generalization ability of LARM. To compute success rates, we test LARM for 30 times on every task. The experimental results are re- ported in Table 1. As shown, LARM presents higher success rates than the compared counterparts in all the test tasks, and the promising performance of LARM is mainly attributed to our designed referee RI. algorithm, which addresses the reward vanishment problem in long-horizon embodied ex- ploration. In addition, we can observe that LARM achieves higher success rates on tasks with shorter action chains, indicating the great challenge in developing long-horizon embodied intelligence. For example, the Harvest bucket task requires three iron ingots, and the Harvest iron sword demands two iron ingots and one stick. Therefore, Harvest iron sword needs one more step (Harvest stick) than Harvest bucket, which causes that LARM obtains a higher success rate in the task of crafting bucket.
MineDojo. MineDojo (Fan et al., 2022) is a pioneering benchmark suitable for studying how to develop open-ended and generally capable embodied agents. It features a simula- tion suite with thousands of tasks specified through natural language prompts. The behaviors that can be conducted in MineDojo primarily include navigation, harvest, combat, and craft. When testing a policy, an agent is randomly ini- tialized in biome with some initial tools. The policy needs to control the agent to explore and harvest resources in the environment and gradually realize the given target. The ac- tion space in MineDojo is a compound multi-discrete space that allows the agent to select a movement action or an op- tional functional action each step, encompassing diverse range of arguments to facilitate complex interactions.
Minellayer: Compared with MineDojo, the basic actions in Mineflayer (PrismarineJS., 2013) are provided APIs, such as harvesting a log or finding the nearest stone. Compared with MojeDojo, these APIs help researchers concentrate more on high-level decision making rather than repetitive
Comparison on Mineflayer. To fully reveal the superiority

---

LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
Table I. Performance comparison with previous methods based on MineDojo.
Task
MineAgent
Plan4MC
LLaMA Rider Base
LLaMA-Rider
RI.GGT
LARM (Ours)
Harvest stick
0,00
0.30
0.23
0,43
0.65
0,93
Harvest crufting table
0.03
0.30
0.37
0.67
0.65
0.87
Harvest bowl
0.00
0.47
0.73
0.97
0.97
Harvest chest
0.00
0.23
0.67
0.77
0.83
Harvest wooden pickaxe
0.00
0.03
0.00
0.37
0.67
0,70
Harvest wooden sword x
0.00
0.47
0.63
0.10
0.70
Harvest furnace
0.00
0.37
0.00
0.17
0.67
0.73
Harvest stone stairs
0.00
0.47
0.00
0.57
0.67
Harvest stone sword
0.00
0.10
0.00
0.00
0,40
Harvest ingot
0.00
0.47
0.03
0.13
0.60
Harvest bucket
0.00
0.20
0.00
0.00
0.37
Harvest iron sword
0.00
0.20
0.00
0,00
0.27
Harvest beef
0.33
0.43
0.03
0.03
0.46
0.60
Harvest mutton
0.35
0.33
0.00
0.03
0.38
0.63
Table Performance comparison based on Mineflayer.
Table 4. Ablation Study on LLM base selection.
Achievement
AutoGPT Voyager STEVE LARM (Ours)
LLM Base
Stick /
Wooden x
Stone x
Iron x
Wooden sword
3/3
3/3
30/30
TinyLIaVA-0.58
0.80
0.50
0.27
0.13
Stone sword
3/3
3/3
3/3
30/30
TinyLLaVA-3.IB
0.83
0.57
0.33
0.13
Iron sword
3/3
3/3
3/3
30/30
TinyLLaVA-3.IB'
0.93
0.70
0.40
0.27
Diamond sword
1/3
3/3
28/30
Table S. Analysis on noisy reward.
Enchanted sword
0/3
0/3
0/3
16/30
Noise Ratio
Stick /
Wooden x
Stone x
Iron x
Table 3.
Ablation Study on Reward Design.
0%
0.93
0.70
0.40
0.27
Reward
Stick
/
Waoden x
Stone x
Iron x
10%
0.93
0.67
0.30
0.17
30%
0.77
0.33
0.17
0.07
0.20
0.13
0.10
0.00
50%
0.50
0.13
0.00
0.00
ER+LAR
0.30
0.23
0.13
0,00
ER+AR2
0.80
0.53
0.20
0,07
in Table 2. We observe that LARR outperforms the compared methods in obtaining different levels of achieve- ments. Notably, LARM is the first method that harvests an enchanted diamond sword in Minecraft successfully.
ER+AR4
0.93
0.70
0.40
0.27
of LARM, we further evaluate LARM using the Mineflayer based environment. The compared methods include Auto- GPT (autogpt). Voyager (Wang et al., 2023a), and STEVE (Zhao et al., 2024). In this work, Voyager is training-frees method implemented based on GPT-4. Its main contribution is designing a multi-step prompt generation pipeline. When a target task is given, Voyager prompts GPT-4 to know which skill should be executed and gradually realize the target. AutoGPT an LLM being able to reason which skill should be performed through multi-step question answering. model. In STEVE, dataset including both videos and text-image pairs is gath- ered and utilized to fine-tune LLaMA (Touvron et al., 2023), and then the fine-tuned model can invoke pre-defined skills.
5.3. Ablation Study
Analysis on reward design. The key difference of Ref- eree RL from the classic PPO implementation is the reward design. Therefore, we ablate different reward settings in this part. We compare four reward choices in MineDojo with four tasks, i.e., Harvest stick, Harvest wooden sword, Harvest stone sword, and Harvest iron sword. The four re- ward choices ER (environment reward only), ER+LAR (environment reward plus auxiliary reward produced by LLaVA-BB, a lightweight LLM), ER+AR2 (environment reward plus auxiliary reward produced by GPT-4o, but the auxiliary reward is determined based on only whether the ac- tion outcome is positive), and ER+AR2 (the standard setting of our method described in this paper). The experimental results are presented in Table 3.
STEVE is a large
In this experiment, the agent spawned a random biome without initial inventory. The test tasks include Harvest wooden sword, Harvest stone sword, Harvest iron sword, Harvest diamond sword, and Harvest enchanted diamond sword. As shown in Fig. 1, Harvest enchanted diamond sword is significantly more complex than the other achieve- ments. The previous methods usually test their models three times in Mineflayer. To reduce randomness, we run LARM for 30 times. The experimental results are reported
As shown, when only the environment reward is provided, the agent presents a significantly better success rate on Har- vest stick than Harvest stone sword, where the latter task demands longer action chain. This phenomenon is because that the environment reward gradually vanishes with the in-

---

LARM: Large Auto- Rello-evgeeeiv Model for Long-Horizon Embodied Intelligence
Search a Village
t3
Multi-agent Combat Enter the Nether
t
Figure 3. More behavior example illustrations of LARM, which include traveling long distance to find village, building nether portal and then entering the nether, multiple agents collaborate with each other to combat zombies.
crease of the action chain length. When the auxiliary reward is generated by LLaVA-7B, the result is also poor, which is because that VA-7B does not understand the Minecraft world well and the provided reward is often incorrect. This result confirms our aforementioned claim that only giant LLMS serve as referees well. In addition, the performance of ER+AR2 worse than ER+AR4, which because se- lecting the correct action does not always bring a positive outcome. For example, the agent decides to search tree but does not find it successfully. In this situation, ER+AR2 returns a negative reward, which punishes the searching a tree decision even though this choice may be correct. By contrast, ER+AR4 will give a small positive reward.
general knowledge. Then, we compare the performances of models without and with the webpage data pre- pre-training, and the results are reported in the 2nd and 3rd rows of Table 4. It can be observed that webpage data pre-training improves the success rates on embodied task execution. This result sug- gests that the numerous text and image data on the Internet has the potential to benefit embodied agents.
Analysis LLM capability. We have mentioned that giant LLMs answer questions about the Minecraft world well while the understanding of lightweight LLMS is poor. In this part, we demonstrate this by providing a question- answer example and highlighting the key content in bold:
Prompt: Minecraft, you need to craft stone pickaxe. What additional resources do you need to gather if you have only got cobblestones in your inventory?
Analysis on LLM base selection. As mentioned before, we utilize the weight of lightweight LLM to initialize the parameters in LARM to provide initial general knowl- edge. Then, we fine-tune LARM using Minecraft related webpage data to enhance its understanding. In this experi- ment, we analyze how the lightweight LLM choice affects the results and whether the webpage data pre-training de- scribed in Section 4.2 is beneficial. To this end, we first compare the performances of LARM using TinyLLaVA- 0.5B and TinyL.LVVA-3.I experiment results are presented in the 1st and 2nd row of Table 4. As shown, adopting an LLM with more parameters as the LARM model base favors the embodied performance, which is mainly attributed to that a larger LLM has stronger
GPT-4o: you already have cobblestone in your inventory, you need to gather wooden planks to craft sticks. You need two sticks and three cobblestones to craft a stone pickaxe.
Llama3-8B: You need a stick.
TinyLLaVA33, IB: To craft a stone pickaxe in Minecraft, you will need the following additional resources: cobblestone, stone, wood, leaves, dirt, grass, pillar, shovel, and sword.
the LLM base, where the
TinyLLaVA-3.15 IBB after Webpage data pre-training: You ad- ditionally need two sticks.
Comparing the answers, we can observe that webpage data

---

LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
pre-training significantly improves the concerned domain of knowledge in TinyL.LaVA-3.IB.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I. Aleman, F. L., Almeida, D., Altenschmidt, J., Alt- man, S., Anadkat, S., et al. Gpt-4 technical report. Mixiv:2303.08774 2023.
Analysis on noisy reward. In the previous experiments, we show that incorporating a referee into the RL framework improves the exploration efficiency very significantly. How- ever, this is partly because GPT-4 can provide high-quality feedback about whether the executed skill is beneficial to realizing the final target in Minecraft, The effectiveness of referee RL under circumstances that the referee feed- back is noisy (many false referee rewards) is not studied. In this experiment, we randomly replace different ratios of referee rewards generated by GPT-4 as other incorrect referee rewards to test the performances of LARM under noisy rewards. The results are presented in Table 5. We can find that when the noise ratio exceeds 10%, the LARM performance drops significantly.
Alayrac, J.-B.. Donahue, J., Luc, P., Micch, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. NeurlPS, 2022.
autogpt.
saanficant-grviiaa/auto--gp An ex- perimental open-source attempt to make gpt- 4 fully autonomous., 2023. URL https:
/ //github.ccm http:/iigniiintt-rraraaa Auto-GPT/ Sttiip////eeeeeeeeeeeeeeeee
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J.,
5.4. Case Study
Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (vpt): Learning to act by watching unlabeled online videos. NeurIPS, 2022.
The previous experiments mainly show the performances of LARM on harvesting various categories of materials and crafting tools. In Fig, 3, we visualize more examples of other behaviors in the Mineflayer based environment, such as exploring the open world, constructing building with a specific structure, and multiple agents cooperate to combat dangerous creatures. These behaviors suggest that our proposed techniques in this work have the potential to be further generalized to other diverse domains.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. NeurIPS, 2020.
Chang. Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation large language models. ACM Transactions on Intelligent Systems and Technology, 2024.
6. Conclusion
Chrisley, R. Embodied artificial intelligence. Artificial Intelligence, 2003.
In this work, we have proposed LARM, which is effi- cient and possesses general knowledge. To train it, we have revealed the feedback vanishment problem in apply- ing classic RLs long-horizon embodied exploration. To address this feedback vanishment, we have developed the referee RL technique. By optimizing LARM with referee RL, our method can learn to complete diverse embodied tasks without human supervision. Especially, LARM is the first method that obtains the enchanted diamond equipment achievement the Minecraft benchmark successfully.
Ding, N., Qin, Y., Yang, G. Wei, F., Yang. Z., Su, Y., Hu, S. Chen, Y., Chan, C.-M., Chen, W., et al. Parameter- efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 2023.
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo: Building open-ended embodied agents with internet-scale knowledge. NewrlPS, 2022.
Impact Statement
Feng, Y., Wang, Y., Liu, J., Zheng, S., and Lu. Z. Llama- rider: Spurring large language models to explore the open world. In NAACL, 2024.
This paper presents work whose goal is to advance the field of embodied intelligence. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Frazier, S. and Riedl, M. Improving deep reinforcement learning in minecraft with action advice. In AAAI, 2019.
Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., and Salakhutdinov, R. Minerl: a large-scale dataset of minecraft demonstrations. In IJCAI, 2019.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (No. 62422606, 62201484).
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.

---

LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
Hu, E. J., Wallis, Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv:2302.13977, 2023.
Huang, C., Mees, O., Zeng, A., and Burgard, W. Visual language maps for robot navigation. ICRA, 2023.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L.,, and Anandkumar, A. Voyager: An open-ended embodied agent with large language model. NeurIPS, 2023a.
Kenton, J. M.-W. C. and Toutanova, Bert: Pre- training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Wang. Z., Cai, S., Chen, G., Liu, A., Ma, X., Liang, Y., and CraftJarvis, T. Describe, explain, plan and select: interactive planning with large language models enables open- world multi-task agents. In NeurlPS, 2023b.
Laleh, A. R. and Ahmadabadi, M. N. A survey on enhanc- ing reinforcement learning in complex environments: In- sights from human and Ilm feedhack. arXiv:2411.134 2024.
Wang. Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., al. Jarvis-I: woen-world multi task agents with memory- aegmete- multimodal language models. TPAMI, 2024.
LeCun, Y. Bengio, Y., and Hinton, G. Deep learning, Na- lure, 2015.
J., Li, D., Savarese, S., and Hoi, S. 2 Bootstrapping language-image pre-tree vre-training with frozen image encoders and large language models. In ICML, 2023.
Xie, T. Zhao, S., Wu, H., Liu, Y., Luo, Q., Zhong. V., Yang, Y., and Yu, Tex(2reward: Automated dense reward function generation for reinforcement learning. In ICLR, 2024.
Lifshitz, S., Paster, K., Chan, H., Ba, J., and Mcllraith, S. Steve-l: A generative model for text-to-behavior in minecraft. NeurIPS, 2023.
Yuan, H., Zhang, C., Wang, H., Xie, F., Cai. P., Dong, H., and Lu, Z. Skill reinforcement learning and planning for open-world long-horizon tasks. In NeurIPS Workshop, 2023.
Liu, H., Li, C., Wu, Q. and Lee, Y. J. Visual instruction tuning. NeurlPS, 2024a.
Liu, S., Yuan, H., Hu, M., Li, Y., Chen, Y., Liu, S., Lu, Z.,, and Jia, J. Rl-gpt: Integrating reinforcement learning and code-as-policy. In NeurIPS, 2024b.
Zhao, Z., Chai, W., Wang, X., Li, B., Hao, S., Cao, S., Ye, T., and Wang, G. See and think: Embodied agent in virtual environment. In ECCV, 2024.
OpenAI. Chatgpt. https:/ com/blog/ chatgpt/, 2023.
Zhou, B., Hu, Y., Weng. X., Jia, J., Luo, J., Liu, X., Wu, J., and Huang, L. Tinyllava: framework of small-scale large multimodal models, 2rXiv:2402.1288, 2024.
PrismarineJS.
Prismarinejs/mineffaeer Create minecraft bots with powerful, stable, and high level javascript api. 2013.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et Learning transferable visual models from natural language supervision. In ICMI., 2021.
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. Habitat: A platform for embodied ai research. In ICCV, 2019.
Scheller, C., Schraner, Y., and Vogel, M. Sample efficient reinforcement learning through learning from demonstra- tions in minecraft. In NeurIPS Workshop, 2020.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv: orXiv:1707.000 170,:077776647,220 2017.


Reviews:
[{"review_id": "hRSQiRUyf8", "reviewer": "Reviewer_xu7T", "summary": "The paper introduces LARM (Large Auto-Regressive Model), a lightweight LLM-based embodied agent designed for long-horizon decision-making in open-world environments.\n\nLARM is built on a lightweight auto-regressive model (fewer than 5B parameters) and directly predicts actions instead of generating text like traditional LLMs, enabling faster inference in real-time settings.\nThe paper identifies the reward vanishment problem in classic RL, where long-horizon credit assignment becomes ineffective.\n\nTo address this, the authors propose Referee RL, a technique where a giant LLM (GPT-4) provides immediate feedback on the quality of actions, distilling generalizable knowledge into LARM without human supervision.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We have addressed all concerns of the Reviewer in the following. The paper will be revised accordingly.\n\n## Q1: Referee RL stable update proof\n\nWe thank the Reviewer for this reminder and will add this proof to the paper. Due to the reply character limit, we cannot provide the whole proof here, but we can give a concise description.\n\nPPO can be treated as an extension of TPRO, and the stable update of TPRO has been proved. Thus, to prove the stable update criteria of referee RL, we need to focus on the key difference between referee RL and PPO, the reward noise. To imitate this noise, we replace the correct reward as false reward with a ratio $\\sigma$ with respect to a random distribution. Empirically, if $\\sigma$ exceeds a threshold $\\epsilon$, the policy cannot converge stably. We can analyze the value of $\\epsilon$ based on the gradient bias analysis in stochastic optimization theory.\n\n## Q2: The impact of referee RL and relation with RL techniques\n\nReferee RL can be applied to various RL algorithms beyond PPO.\n\nReferee RL combines the techniques of RL and LLM to address the key problems in these two communities, therefore benefiting both communities.\n\nThe key problem of RL is its exploration inefficiency. LLM has general knowledge. Our work shows that LLM can provide guidance to RL and reduce the exploration cost very significantly (like the experiment results in the Reply to Q1 of Reviewer 4ZZV).\n\nThe key problem of LLM is it lacks the knowledge of direct environment interaction. Based on RL exploration, we tune a lightweight LLM into a SOTA embodied agent.\n\nOur method does not conflict with the RL techniques mentioned by the Reviewer, such as hierarchical RL. They handle the long-horizon exploration problem from different perspectives and can be applied simultaneously.\n\n## Q3: Exploration on more RL methods\n\nThe RL methods mentioned by the Reviewer include hierarchical RL, inverse RL, reward relabling, Decision Transformer, more RL baselines. These methods do not conflict with our method and can be applied simultaneously. We will add discussion on the relation with these methods to the paper.\n\nSpecifically, our LARM model is for high level scheduling, and a part of the low-level skills are implemented based on PPO or deep Q-learning. So hierarchical RL has been used in our work.\n\nInverse RL assumes there are optimal expert demonstrations. However, the demonstrations are unavailable in our studied task. For reward relabling, it assumes the historical actions are optimal under the reward function. However, this assumption is often too strong to meet, and the reward is also difficult to define explicitly.\n\nWe have tried the idea of Decision Transformer, which relies on training on expert demonstration data. To realize this idea, we first collect abundant demonstration data, and the method achieves similar performance as referee RL with less computing cost (4 hours of training). However, the expert demonstration data is often unavailable.\n\nWe have tried replacing the PPO in referee RL as deep Q-learning, and the result comparison is as follows:\n\n|  | Stick | Wooden | Stone | Iron |\n| :-: | :-: | :-: | :-: | :-: |\n| Deep Q-learning | 0.87 | 0.63 | 0.30 | 0.17 |\n| PPO | 0.93 | 0.70 | 0.40 | 0.27 |\n\nWe can see that PPO based referee RL gets better performance.\n\n## Q4: Influence of referee reward noise\n\nWe have analyzed reward noise influence theoretically in the reply to Q1. This part analyzes it using experiments. We randomly modify the reward generated by GPT-4 with a ratio $\\sigma$, and the success rates on different tasks are as follows:\n\n| $\\sigma$ | Stick | Wooden | Stone | Iron |\n| :-: | :-: | :-: | :-: | :-: |\n| 0% | 0.93 | 0.70 | 0.40 | 0.27 |\n| 10% | 0.93 | 0.67 | 0.30 | 0.17 |\n| 30% | 0.77 | 0.33 | 0.17 | 0.07 |\n| 50% | 0.50 | 0.13 | 0.00 | 0.00 |\n\nWe can observe that (1) Stronger noise deteriortes the sucess rates. (2) Tasks requiring longer execution chains are less noise tolerant.\n\nIn addition, we design two new experiments using the VirtualHome simulator and a real robot, where the referee LLM sometimes provide incorrect rewards. Refer to the replies to Q1 and Q2 of Reviewer 4ZZV for experiment details and results.\n\n## Q5: Results on smaller referee LLMs\n\nAs suggested by the Reviewer, we study the success rates adopting different sizes of referee LLMs. Due to the reply character limit, refer to the replies to Q5 of Reviewer 2LZg for details.\n\n## Q6: Computing efficiency analysis\n\nThe reason we did not provide quatitative efficiency comparison with previous LLM based methods is these methods are often based on large LLMs deployed on remote servers, like GPT-4. We cannot know their specific parameter numbers, FLOPs, etc. What we can make sure is their computing costs are enormous as publicly known and far more than ours. We report the efficiency metrics of our method as follows:\n\nInference Time | Inference Memory | FLOPs | Training Time \n| :-: | :-: | :-: | :-: |\n0.58s | 3.8G | 614.8G | 42 hours", "claims_and_evidence": "LARM balances efficiency and generalization by combining RL’s efficiency with LLM’s reasoning (Figure 1). It directly predicts actions instead of generating text, enabling faster inference.\n\nLARM outperforms RL and LLM-based baselines (MineAgent, Plan4MC, LLaMA-Rider, RL-GPT), achieving higher success rates in Minecraft tasks and becoming the first AI to craft enchanted diamond equipment (Tables 1 & 2).\n\nReferee RL mitigates reward vanishment, providing immediate feedback via GPT-4, improving long-horizon learning stability (Equation 5, Algorithm 1). Ablations confirm that removing Referee RL hurts performance (Table 3).", "methods_and_evaluation_criteria": "Reasonable methods and evaluation criteria.\n\nLARM uses Referee RL, where GPT-4 provides auxiliary rewards to address reward vanishment in long-horizon tasks.\n\nBuilt on TinyLLaVA-3.1B with LoRA, it directly predicts actions instead of generating text, enabling faster inference.\n\nPretrained on a 34GB Wiki dataset for better long-horizon planning.\n\nEvaluation Setup:\nTested in: MineDojo (open-ended AI) and Mineflayer (API-based Minecraft).\nMetrics: Task success rates, inference speed (0.58s per step on RTX 4090).", "theoretical_claims": "LARM identifies the \"reward vanishment\" problem in long-horizon RL, where TD errors approach zero over time, making standard RL inefficient for credit assignment (Equation 5).\n\nReferee RL mitigates this issue by injecting auxiliary rewards from GPT-4, providing immediate feedback on action quality, which stabilizes long-horizon learning (Algorithm 1, Figure 2).\n\nGAE formulation (Equation 6) supports the claim that standard PPO struggles with long-horizon tasks, while Referee RL preserves meaningful TD errors, enabling more effective optimization.\n\nDiscussions: \n\nMissing formal convergence analysis – The paper does not provide a proof that Referee RL leads to stable policy updates over time.", "experimental_designs_or_analyses": "-Strengths:\n\nComprehensive evaluation of LARM across multiple long-horizon tasks in MineDojo and Mineflayer, demonstrating its effectiveness in open-ended embodied AI.\n\nComparison with RL-based and LLM-based baselines (MineAgent, Plan4MC, LLaMA-Rider, RL-GPT, Voyager, AutoGPT, STEVE) ensures fair benchmarking (Tables 1 & 2).\n\n-Weaknesses and Missing Analyses:\n\nReferee RL is the key contribution, but its impact is not fully analyzed.\n\nThe paper shows that GPT-4-based referee feedback improves LARM, but does not explore alternative reward shaping methods (e.g., inverse RL, reward relabeling).\n\nMore experiments are needed to analyze how different referee models (e.g., smaller LLMs vs. GPT-4) affect policy training and success rates.\n\nHow sensitive is Referee RL to noisy or incorrect feedback? The paper does not test scenarios where the referee gives suboptimal or misleading rewards.", "relation_to_broader_scientific_literature": "LARM contributes to RL-LLM hybrid learning.\n\nKey novelty:\n\nReferee RL introduces LLM-based auxiliary rewards to mitigate reward vanishment in long-horizon RL.\n\nLARM is an example of multi-modal input models for direct action prediction, showing how LLMs can enhance policy learning without text-based prompting.\n\nWhat’s missing?\n\nComparison to hierarchical RL approaches\n\nDiscussion on alternative reward shaping methods – hindsight experience replay, etc.\n\nRelation to decision transformers or sequence modeling in RL", "essential_references_not_discussed": "Decision Transformer (DT) (Chen et al., 2021) is not referenced or discussed.\n\n-DT treats reinforcement learning as sequence modeling, using autoregressive token prediction similar to LARM’s auto-regressive action selection.\n\n-Why it matters: Comparing LARM’s LLM-based policy with DT’s transformer-based decision-making would clarify how Referee RL improves credit assignment compared to DT’s return-conditioned training.\n\nHierarchical Reinforcement Learning (HRL) methods are not discussed.\n\n-LARM avoids hierarchical task decomposition by relying on an LLM-based policy, but methods like FeUdal Networks (Vezhnevets et al., 2017) or Option-Critic (Bacon et al., 2017) explicitly structure long-horizon tasks with sub-goals.", "other_strengths_and_weaknesses": "-Strengths\n\nNovel approach to handling reward vanishment – LARM introduces Referee RL, where a GPT-4-based referee provides auxiliary rewards, improving long-horizon credit assignment.\n\nCombines RL and LLM advantages while mitigating their drawbacks – Unlike task-specific RL agents or slow, text-generating LLM agents, LARM achieves both efficiency and generalization by directly predicting actions (Figure 1).\n\n-Weaknesses:\n\nThe paper is highly specific to VLMs and LLMs, limiting broader RL insights.\n\nThe system design and evaluation focus heavily on LLM-driven decision-making, but a more general RL perspective (e.g., comparisons with hierarchical RL, multi-step planning, or alternative reward shaping methods) would provide broader applicability.\n\nFew traditional RL baselines – While the paper compares against LLM-based baselines, it does not compare LARM’s learning efficiency against traditional RL methods beyond PPO.\n\nNo discussion on whether Referee RL could benefit non-LLM policies – Could a transformer-based decision model or hierarchical RL agent also benefit from Referee RL?\n\nNo computational efficiency analysis.\n\nThe paper claims LARM is more efficient than traditional LLM agents, but does not provide FLOP comparisons, memory usage, or training time benchmarks to quantify efficiency gains.", "other_comments_or_suggestions": "Referee RL needs further analysis.\n\nHow sensitive is Referee RL to incorrect or noisy feedback? The paper does not analyze how GPT-4 errors affect policy learning.\nNo study on different referee models – The paper assumes GPT-4 is necessary, but smaller LLMs or alternative feedback mechanisms could be explored.", "questions_for_authors": "How does LARM compare to traditional RL methods beyond PPO?\n\nThe paper primarily compares LARM to LLM-based approaches, but how does it compare to hierarchical RL or model-based RL in long-horizon planning?\n\nImpact on Evaluation: If LARM is significantly better than traditional RL, it strengthens the argument for LLM-driven decision-making in embodied AI.\nHow sensitive is Referee RL to incorrect or noisy feedback?\n\nThe paper assumes GPT-4 always provides accurate auxiliary rewards, but what happens if the referee makes errors or inconsistent judgments?\n\nImpact on Evaluation: If Referee RL is highly sensitive to noise, additional robustness measures may be needed.\nCould a smaller LLM serve as an effective referee?\n\nThe paper uses GPT-4 as the referee, but has the impact of smaller LLMs (e.g., LLaMA-7B, Mistral-7B) been tested?\nImpact on Evaluation: If smaller models perform similarly, LARM could be more computationally efficient.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YzvYDOxvec", "reviewer": "Reviewer_4ZZV", "summary": "This paper focusses on the long-horizon embodied intelligence, specifically, the MineCraft tasks. Previous works generally rely on the strong generalization of giant LLM agents, since the performance of lightweight LLMs such as LLaVA-7B is limited. However, this requires huge computing resources. In this paper, the authors aim to combine the advantages o both RL methods and LLM methods while avoiding their shortcomings. To achieve this, they first propose Large Auto-Regressive Model (LARM), with the main body using the same lightweight LLMs   as TinyLLaVA. LARM is equipped with basic knowledge about the game it is playing by using numerous WiKi webpage data for pre-training. It predicts the next action to perform in an auto-regressive manner by taking environmental observation as input. To train LARM, this paper introduce referee RL instead of using traditional RL which will lead to reward vanishment during long-horizon embodied exploration. The core idea is to employ a referee (like a giant LLM) to provide immediate feedback about whether the just performed action brings positive contribution to realizing the final target.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We have test our method in more environments as suggested, and the details are in the following. The paper will be revised accordingly.\n\n## Q1: Experiment in a household simulator\n\nWe thank the Reviewer for this suggestion. As suggested, we conduct experiments in VirtualHome to further validate the effectiveness of our method. We design 10 tasks that each task requires 4 to 6 steps of low-level skill execution. An example of the designed task is \"watch TV\". To complete this task, the agent needs to complete the following five steps in sequence: (1) Walk to TV. (2) Turn on TV. (3) Walk to a sofa. (4) Sit on the sofa. (5) Watch TV. An anonymous video link describing how LARM performs this \"watch TV\" task is provided here: [VirtualHome Video](https://drive.google.com/file/d/1Dp0ivtO3e-8xa6fsBtbQD7wLtSl6cYtp/view?usp=sharing).\n\nIn this experiment, we remove the Wiki data pre-train step for LARM to show that our algorithm can also work well without Wiki data pre-train. we employ Qwen-VL-Max as the referee model. Notably, Qwen-VL-Max does not always provide correct referee reward (about 10% error rate). Therefore, this experiment shows the robustness of our method based on noisy reward. Refer to the replies to Q1 and Q4 of Reviewer xu7T for more in-depth analysis on the robustness of our method under noisy reward.\n\nIn this experiment, we compare our method with classic RL methods including Deep Q-learning, TPRO, and PPO to reveal how much efficiency is improved by combining RL and LLM using our method. The input to policies includes task target description,agent initial state, historical actions. As VirtualHome provides the function to execute each low-level skill, we do not need to implement low-level policies like in Minecraft. For each method, we report the avarage times of explorations the policy needs to complete the designed tasks for the first time (all actions are predicted without random exploration). The results are as follows:\n\nMethod | Deep Q-learning | TPRO | PPO | LARM\n| :-: | :-: | :-: | :-: | :-: |\nAverage Times | 4876.3 | 3925.8 | 3766.0 | 85.4\n\nThe above results sufficiently show the efficiency advantage of LARM. For tasks needing longer action chains (like tens to thousands of steps are needed for the tasks in Minecraft), classic RL methods like Deep Q-learning cannot complete a task for the first time after millions of exploration times. By contrast, LARM learns to complete the task efficiently.\n\n## Q2: Experiment in the real world \n\nAs suggested by the Reviewer, we design an experiment in the real world to further validate the practical value of our method. In this experiment, we train a LARM policy to control a robot to build blocks. \n\nSpecifically, several blocks in various shapes are randomly placed on a table. The LARM policy is trained based on our proposed referee RL algorithm to stack blocks as a building similar to a house. To complete this task, the policy needs to learn to select suitable blocks and decide their relative positions to place. The basic skills are implemented based on imitation learning, where we adopt the robotic manipulation algorithm VIRT to grasp and place blocks.\n\nWe use Qwen-VL-Max as the referee model. It judges whether the building is becoming more like a house based on image observation. Notably, we do not clearly define how a house should look like. Therefore, there are multiple ways of stacking different blocks to be similar to a house. Correspondingly, the referee reward provided by the referee policy is noisy.\n\nInterestingly, after 100 iterations of random exploration, the policy learns to select and stack different blocks as a house. The anonymous videos of two stacking block process examples are provided here: [Robot Stack Blocks 1](https://drive.google.com/file/d/15dB0SLVZBckz6WzJfIEq7GKv7n_Mugdg/view?usp=sharing) and [Robot Stack Blocks 2](https://drive.google.com/file/d/1SofuQErTk8qn51ChDwSVmTTklY9WpG5X/view?usp=sharing).\n\nAccording to the results of this experiment, we can conclude that (1) Our proposed method can be applied to a real robot task. (2) By combining LLM and RL, our method can derive policies that are able to perform creative tasks.", "claims_and_evidence": "Yes. this paper provide solid experiments to valid their calim: using a lightweight LARM for long-horizon embodied tasks while using referee RL as the training method.", "methods_and_evaluation_criteria": "Yes. This paper carries out experiments on Minedojo and Mineflayer, which are commonly used to assess an embodied intelligence's ability of long-term tasks. The author also provides extensive comparisons with previous SoTA methods.", "theoretical_claims": "Yes. they are correct.", "experimental_designs_or_analyses": "Yes. the experiments are extensive.", "relation_to_broader_scientific_literature": "The method proposed in this paper could be potentially adopted to promote AI agent in game environments.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "This paper use a tiny LLM to finish the long-term diamond task. Comparing with previous works which rely on giant LLMs (e.g., gpt-4o) or finetuned on larger LLMs (e.g., 70B LLaMA), LARM proposed in this paper is time-efficient, which only takes 40 GPU hours on a single RTX4090 GPU.", "other_comments_or_suggestions": "It will be better if the authors can provide more results in addition to the MineCraft environments. For example, do the authors ever try their methods on simulated household environments like VirtureHome, or even the real world? It will show more value compared to the current bit-style MineCraft, considering the rules of MineCraft are highly structured.", "questions_for_authors": "No additional questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jSn3pMgQMZ", "reviewer": "Reviewer_2LZg", "summary": "This paper introduces a lightweight LLM-based agent that balances efficiency and generalization for long-horizon tasks. Using Referee RL, which employs a giant LLM for immediate feedback, LARM overcomes reward vanishment in reinforcement learning. Tested in Minecraft, it outperforms previous methods, achieving complex goals like enchanted diamond equipment.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We have addressed the concerns of the Reviewer one by one in the following. The paper will be revised accordingly.\n\n## Q1: Real-time inference\n\nThe speed of 0.58 second per inference is the inference time of the high-level scheduling policy. Each time of high-level schedule corresponds to seconds of low-level skill execution. The inference speed of the low-level skill policy is more than 1000 FPS. Therefore, we claim that our method meets the online inference requirement.\n\nTo avoid misunderstanding, we change the claim in the paper as \"meet the speed requirement of online high-level action scheduling\".\n\n## Q2: Similar to two previous works\n\nThe two works mentioned by the Reviewer are both about low-level action learning, while ours is about high-level scheduling. The motivation, method, and results are all very different:\n\n* CLIP RL: This work uses CLIP attention map to replace text instruction and provides an invariant representation for the policy to approach different objects. This work does not have significant similarity with ours, which combines LLM and RL.\n\n* Auto MC-Reward: This work relies on LLMs to write code to generate reward functions. Its idea can be applied to learn low-level skills where reward functions can be defined explicitly. However, in many tasks (like the real robot block building task in the Reply to Q2 of the Reviewer 4ZZV), the rewards are too subjective and complex to define explicitly, so this work is inapplicable. By contrast, our method does not have this limitation.\n\nAs suggested by the Reviewer, we add comparison to these two works. For fair comparison, all three works adopt the train and test settings of Auto MC-Reward. The success rates on different key achievements are as follows:\n\n| Method | Wood | stone | iron | diamond |\n| :-: | :-: | :-: | :-: | :-: |\n| CLIP RL | 0.64 | 0.23 | 0.02 | 0.00 |\n| Auto MC-Reward | 0.85 | 0.78 | 0.63 | 0.29 |\n| LARM | 0.98 | 0.96 | 0.81 | 0.56 |\n\n## Q3: Test in more environments\n\nAs suggested by the Reviewer, we have test our method in more environments, including VirtualHome (a household activity simulator) and the Cobot Magic robot (a robot in the real world). Due to the reply character limit, please refer to the replies to Q1 and Q2 of Reviewer 4ZZV for experiment details and results. We believe these experiments sufficiently confirm the practical value of our work.\n\n## Q4: How skills are generated\n\nThe skills are generated by GPT-4. Our fine-tuned model is used for online high-level action scheduling. It does not serve as the referee model.\n\n## Q5: Generalization to more LLMs\n\nOur method can be generalized to different LLMs. As suggested by the Reviewer, we add experiments that test using different LLMs as the referee and LARM base model. \n\nFirst, we study the success rates adopting different sizes of referee LLMs, including TinyLLaVA-3.1B, Fuyu-8B, Llama3-8B, and Qwen2-14B. Notably, these models do not always provide correct referee reward, meaning the reward quality is different. The success rates are as follows:\n\n| referee | Stick | Wooden | Stone | Iron |\n| :-: | :-: | :-: | :-: | :-: |\n| TinyLLaVA-3.1B | 0.73 | 0.50 | 0.17 | 0.00 |\n| Fuyu-8B | 0.87 | 0.57 | 0.27 | 0.03 |\n| Llama3-8B | 0.90 | 0.67 | 0.33 | 0.13 |\n| Qwen2-14B | 0.90 | 0.67 | 0.37 | 0.20 |\n| GPT-4o | 0.93 | 0.70 | 0.40 | 0.27 |\n\nAccording to the results, we can observe that generally LLMs with more parameters lead to better performance. The key factor is the quality of the generated referee reward. Refer to the replies to Q1 and Q4 of Reviewer xu7T for more in-depth analysis on the reward noise tolerance of LARM.\n\nThen, we study replace the LARM based model from TinyLLaVA-3.1B\nto Fuyu-8B and Qwen2-14B. The success rates and speed are as follows:\n\n| Base Model | Stick | Wooden | Stone | Iron | Inference Time\n| :-: | :-: | :-: | :-: | :-: | :-: |\n| TinyLLaVA-3.1B | 0.93 | 0.70 | 0.40 | 0.27 | 0.58 |\n| Fuyu-8B | 0.97 | 0.77 | 0.43 | 0.30 | 1.19 |\n| Qwen2-14B | 1.00 | 0.80 | 0.47 | 0.33 | 2.88 |\n\nAccording to the results, we can observe that replacing the base model with a larger one generally benefits the performance. However, the inference time is also increased significantly.\n\n## Q6: Missing reference\n\nAs suggested, we will add all these references to the paper and discuss the relation with them.\n\n## Q7: Wiki Pre-train effect\n\nThe Wiki pre-train improves model convergence speed and success rates on different tasks significantly. Actually, we have reported the success rate gains caused by Wiki pre-train in Table 4 of the paper. For the convenience of review, we present the results in the following table. Besides success rates, we also report the exploration iteration number for arriving convergence.\n\n| Wiki Pre-train | Exploration Iterations | Stick | Wooden | Stone | Iron |\n| :-: | :-: | :-: | :-: | :-: | :-: |\n| No  | 8500 | 0.83 | 0.57 | 0.33 | 0.13 |\n| Yes | 5000 | 0.93 | 0.70 | 0.40 | 0.27 |\n\nThe Wiki data preprocessing pipeline is the same as the work MineDojo.", "claims_and_evidence": "The paper claims that \"LARM runs at 0.58 seconds per inference, meeting online inference requirements.\" However, with an interaction FPS of less than 2, this is insufficient for real-time deployment. The claim should be more accurately framed.", "methods_and_evaluation_criteria": "The proposed method is highly similar to prior works [2] and [3], which also employ multimodal large models as reward models to assist agent learning. The contributions are incremental rather than groundbreaking.\n\nThe paper fails to compare LARM with essential baselines, particularly [2] and [3], which are closely related. A fair comparison is necessary to establish the advantages of the proposed approach.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "The method is only evaluated on Minecraft, whereas related works typically demonstrate robustness and generalization across multiple gaming environments [4]. Testing on other domains, including games with incomplete Wiki resources (e.g., Montezuma’s Revenge), would strengthen the claims.\n\nWhen test in MineFlayer, are the skill code are generated by GPT-4 or your fine-tuned models?\n\nThe paper lacks evaluation on other vision-language models (e.g., Fuyu, BLIP, Qwen2-VL), making it unclear if LARM's improvements generalize across architectures.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "This paper does not cite some related works:\n\n[1] Wang et al. Jarvis-1: Open-world Multi-task Agents with Memory-Augmented Multimodal Language Models. T-PAMI 2024.\n\n[2] Jiang et al. CLIP-Guided Reinforcement Learning for Open-Vocabulary Tasks. ECCV 2024.\n\n[3] Li et al. Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft. CVPR 2024.\n\n[4] Wang et al. OmniJarvis: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents. NeurIPS 2024.", "other_strengths_and_weaknesses": "The method relies heavily on Minecraft Wiki for learning. However, the paper does not quantify how much this pretraining improves performance over the original model, leaving a significant gap in evaluation. Additionally, the preprocessing details of the Minecraft Wiki corpus are missing, making replication difficult.", "other_comments_or_suggestions": "See in weakness.", "questions_for_authors": "See in Weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GivCCNixSM", "reviewer": "Reviewer_xmFZ", "summary": "In the open-world environment of Minecraft, this paper proposes the Large Auto-Regressive Model (LARM), which leverages the instruction-following and generalization capabilities of large language models to construct a Minecraft agent. Additionally, the paper introduces Referee RL to provide immediate feedback for training LARM. Experimental results on MineDojo and Mineflayer environments show that LARM outperforms baselines.\n\n## update after rebuttal\nI have carefully read all the reviewers' comments as well as the authors' rebuttal. Most of my concerns can be addressed through a thorough revision of the manuscript. However, my primary concern remains with the evaluation on the MineRL environment.\n\nWhile I appreciate the additional experiments provided by the authors, I find the following issues problematic:\n\n1) I am surprised by the authors were able to adapt their method to the MineRL environment, train the LARM model, and complete evaluations on 200 tasks (each with 30 trials) within just one day.\n\n2) Due to the significant lack of implementation details and the experiments on MineRL, the paper requires substantial revision.\n\nSo I keep the scores.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We believe the Reviewer has significant misunderstandings of this work. In the following, we address the concerns one by one using more precise explanations and sufficient experiments.\n\n## Q1: Inference speed analysis\n\nWe did not explicitly compare the speed of our method with the counterparts because many of them are based on LLMs deployed on remote servers like GPT-4. Their speeds are largely influenced by network latency, so it is difficult to compare fairly.\n\nTo conduct a comprehensive speed comparison as suggested, we first report the efficiency metrics of our method as follows:\n\nInference Time | Inference Memory | FLOPs | Training Time \n| :-: | :-: | :-: | :-: |\n0.58s | 3.8G | 614.8G | 42 hours\n\nThen, we study how the success rates and inference times are changed by adopting different base LLMs. Due to the reply character limit, please refer to the reply to Q5 of Reviewer 2LZg for results.\n\nFor the online inference claim, it is because LARM is for high-level schedule. Each time of high-level schedule corresponds to several seconds of low-level skill execution. The inference speed of the low-level skill policy is more than 1000 FPS. So the LARM speed of 0.58 second per inference meets the online inference requirement. To avoid such misunderstandings, we change the claim in the paper as \"meet the speed requirement of online high-level action scheduling\".\n\n## Q2: Evaluated task number\n\nActually, we have test our method in other tasks of MineDojo. Our method is still very effective. \n\nWe do not report results on them because all previous works in MineDojo choose to select representative tasks to report results, although there are totally 1581 tasks in MineDojo. If we report the results on other tasks, we have no method to compare. The reported tasks in Table 1 of the paper follow previous published works.\n\nTo further show the effectiveness of our method, we add experiments based on a household simulator VirtualHome and a real-world robot. Refer to the replies to Q1 and Q2 of Reviewer 4ZZV for details.\n\n## Q3: Missing comparison\n\nAs reminded by the Reviewer, we will add all the missing references to the paper and discuss the relation with them.\n\nIn the following, we compare success rates with them. As their experiment settings are different from ours, we unify all methods using the setting of DEPS. The success rates on key achievements are as follows:\n\n| Method | Wooden | Stone | Iron | Diamond |\n| :-: | :-: | :-: | :-: | :-: |\n| MP5 | 0.89 | 0.76 | 0.52 | 0.22 |\n| DEPS | 0.80 | 0.69  | 0.17 | 0.02 |\n| JARVIS-1 | 0.89 | 0.89 | 0.35 | 0.09 |\n| Optimus-1 | 0.98 | 0.92 | 0.47 | 0.12 |\n| LARM | 1.00 | 0.97 | 0.57 | 0.20 |\n\n## Q4: Significance from previous works\n\nWe believe this work shows great difference and advantages over the works mentioned by the Reviewer.\n\n* The works mentioned by the Reviewer are also based on LLMs. This means if the LLM does not have accurate knowledge about this environment, the method fails. By contrast, ours combines LLM and RL. These two parts both contribute to learning new tasks. If LLM does not have accurate knowledge (the referee reward is noisy), the RL still conducts learning through environment reward. To show this, we have added experiments using a household simulator and a real robot. Refer to the replies to Q1 and Q2 of Reviewer 4ZZV for results.\n\n* The methods mentioned by the Reviewer are all based on a stack of heavy models, meaning slow inference and high deployment cost. However, embodied applications mostly require fast response and deployment on local devices. Our real robot experiment shows the trained policy can be deployed using the local resource of a robot to achieve good success rates.\n\n## Q5: Unfair comparison\n\nThe comparison is absolutely fair. Comparing methods in MineDojo and Mineflayer, our method does not use any extra information than the compared methods. If we call an API, the compared methods also call this API to get information. The inventory list and environment information used by our method are also used in other methods.\n\n## Q6: How skills are generated\n\nWe use a skill generation pipeline similar to Voyager, but the overall method framework is very different. Voyager completely relies on GPT-4, a remote LLM, to complete tasks in an offline way. Ours supports online learning and is lightweight.\n\n## Q7: Skills in MineDojo\n\nAlthough the basic actions in MineDojo are low-level actions like moving forward a step, almost all previous works in MineDojo use RL policies and rule-based methods to build higher level skills, like search a tree. The Reviewer can refer to the code of Plan4MC to know how these skills are built. Our skills used in MineDojo completely follow them, therefore the comparison is absolutely fair. We do not use any extra information.\n\n## Q8: Success Rate 0.65\n\nWe test our own method for 30 times to compute the success rate. The 0.65 is the success rate of a compared method and is obtained from its original paper.", "claims_and_evidence": "One of the key claims of this paper is leveraging the advantages of RL methods and LLMs while mitigating their limitations. Regarding the slow inference speed of LLMs, the paper asserts that LARM achieves an inference speed of 0.58 seconds per inference. However, the paper does not provide a comprehensive experimental comparison for inference speed. Moreover, this speed does not meet the 20Hz inference requirement of MineDojo.", "methods_and_evaluation_criteria": "1. Table 1 presents experiments conducted on MineDojo; however, the number of evaluated tasks is too limited, significantly fewer than the MineDojo benchmark, which includes 1,581 tasks for Programmatic Tasks.\n\n2. There is a lack of comparison to some powerful baselines on MineDojo, such as DEPS [1], Jarvis-1 [2], MP5 [3], etc.\n\n[1] Wang, Zihao, et al. \"Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents.\" NeurIPS 2023.\n\n[2] Wang, Zihao, et al. \"Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models.\" TPAMI 2024.\n\n[3] Qin, Yiran, et al. \"Mp5: A multi-modal open-ended embodied system in minecraft via active perception.\" CVPR 2024.", "theoretical_claims": "No obvious errors were found in the theoretical claims.", "experimental_designs_or_analyses": "As stated in the Methods and Evaluation Criteria section, the paper lacks a sufficient number of evaluation tasks and up-to-date baselines.", "relation_to_broader_scientific_literature": "The core objective of this paper is to leverage large language models to generate appropriate skills (code) for interacting with the environment to complete tasks. The fundamental idea is derived from prior work, Voyager. The reviewer did not find significant innovations in terms of model architecture, training methods, or skill implementation. However, the reward design approach may contribute to the community.", "essential_references_not_discussed": "The paper does not discuss comparisons between the proposed LARM and key baselines [1] [2] [3] [4] in current Minecraft research. The authors need to clarify the differences or advantages of LARM compared to the aforementioned agents.\n\n[1] Wang, Zihao, et al. \"Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents.\" NeurIPS 2023.\n\n[2] Wang, Zihao, et al. \"Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models.\" TPAMI 2024.\n\n[3] Qin, Yiran, et al. \"Mp5: A multi-modal open-ended embodied system in minecraft via active perception.\" CVPR 2024.\n\n[4] Li, Zaijing, et al. \"Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks.\" NeurIPS 2024.", "other_strengths_and_weaknesses": "Strength\n\n1. Using a large language model to provide rewards is an interesting idea. Although the reward implementation in this paper is relatively simple, it offers new insights into reward design in the field of reinforcement learning.\n\nWeakness\n\n1. As stated in line 218, the paper provides GPT-4 with an inventory list and information about environmental resources surrounding the agent to generate appropriate rewards. At the same time, these information serve as input of LARM. However, this implementation relies on internally integrated environment APIs, making it an unfair comparison against most existing works.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. As stated in line 271, the paper follows Voyager’s skill update strategy but does not explain how these skills are generated. Does this imply that these skills originate from Voyager or that LARM is based on the Voyager framework?\n\n2. Could the authors provide more implementation details on how skills interact with the environment in MineDojo? As far as I know, the action space in MineDojo consists of low-level actions rather than code.\n\n3. How was the 0.65 success rate in Table 1 obtained? Line 340 states that each task was executed 30 times—how does this translate to a success rate of 0.65?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes the Large Auto-Regressive Model (LARM), an autoregressive LLM-based agent for open-world decision-making problems such as Minecraft. The main aspect of the LLM training is Referee RL, which uses a large language model (GPT-4 class) to provide feedback on action quality. Results are demonstrated on Minedojo and Mineflayer. 

  The paper received borderline scores, with appreciation for the interesting method and analysis as well as thorough comparison. Some weaknesses were mentioned, however, for example lack of in-depth analysis/ablation of the components (e.g. Referee RL), comparison to other reward shaping and RL methods, analysis on suboptimal feedback, and additional experiments. The authors provided a comprehensive rebuttal, including additional comparisons (CLIP RL, MC-Reward, etc.) and new experiments in VirtualHome and real-world experiments. After significant back and forth, the review scores remain borderline. 

  Having considered the paper, reviews, rebuttal, and significant discussion, I lean towards acceptance of this paper. The method provides an interesting analysis and method with RL-based policy learning, and the rebuttals address most of the concerns. Therefore, I believe that the positives outweigh the remaining weaknesses, and this paper would be an interesting contribution. I strongly encourage the authors to include the results of the discussion, including new experiments. Further, I find that the paper overall lacks a situation with respect to vision-language-action models, which also finetune (often larger) VLMs and besides supervised fine-tuning have begun to include RL-based finetuning as well to yield generalist embodied agents.
Decision: Accept (poster)