Scaling Autonomous Agents via Automatic Reward Modeling And Planning
Authors: Zhenfang Chen
Delin Chen
Rui Sun
Wenjun Liu
Chuang Gan
Venue: ICLR 2025 Poster (2025)
Keywords: agents
large language models
planning
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across a range of text-generation tasks. However, LLMs still struggle with problems requiring multi-step decision-making and environmental feedback, such as online shopping, scientific reasoning, and mathematical problem-solving. Unlike pure text data, collecting large-scale decision-making data is challenging. Moreover, many powerful LLMs are only accessible through APIs, which hinders their fine-tuning for agent tasks due to cost and complexity. To address LLM agents' limitations, we propose a framework that can automatically learn a reward model from the environment without human annotations. This model can be used to evaluate the action trajectories of LLM agents and provide heuristics for task planning. Specifically, our approach involves employing one LLM-based agent to navigate an environment randomly, generating diverse action trajectories. Subsequently, a separate LLM is leveraged to assign a task intent and synthesize a negative response alongside the correct response for each trajectory. These triplets (task intent, positive response, and negative response) are then utilized as training data to optimize a reward model capable of scoring action trajectories. This reward model can be integrated with LLM-based agents and various planning algorithms to enhance task-solving performance. The effectiveness and generalizability of our framework are demonstrated through evaluations conducted on different agent benchmarks. In conclusion, our proposed framework represents a significant advancement in enhancing LLM agents' decision-making capabilities. By automating the learning of reward models, we overcome the challenges of data scarcity and API limitations, potentially revolutionizing the application of LLMs in complex and interactive environments. This research paves the way for more sophisticated AI agents capable of tackling a wide range of real-world problems requiring multi-step decision-making.

Published as a conference paper at ICLR 2025
ARMAP: SCALING AUTONOMOUS AGENTS VIA AUTOMATIC REWARD MODELING AND PLANNING
Zhenfang Chen MIT-IBM Watson AI Lab
Delin Chen"
Rui Sun"
UMass Amherst
University California, Los Angeles
Wenjun Liu" UMass Amherst
Chuang Gan
UMass Amherst and MIT-IBM Watson AI Lab
ABSTRACT
Large language models (LLMs) have demonstrated remarkable capabilities across range of text-generation tasks. However, LLMs still struggle with problems requiring multi-step decision-making and environmental feedback, such as online shopping, scientific reasoning, and mathematical problem-solving. Unlike pure text data, collecting large-scale decision- makinn data is challenging. Moreover, many powerful LLMs are only accessible through APIs, which hinders their fine-tuning for agent tasks due to cost and complexity. To address LLM agents' limitations, we propose a framework that can automatically learn reward model from the environment without human annotations. This model can be used to evaluate the action trajectories of LLM agents and provide heuristics for task planning. Specifically, our approach involves employing one LLM- agent to navigate an environment randomly, generating diverse action trajectories. Subsequently, separate LLM is leveraged to assign task intent and synthesize a negative response alongside the correct response for each trajectory. These triplets (task intent, positive response, and negative response) to optimize a reward model capable of scoring action trajectories. This reward model can be integrated with L.LM-based agents and various planning algorithms to enhance task-solving performance. The effectiveness and generalizability of our framework are demonstrated through evaluations conducted on different agent benchmarks. In conclusion, our proposed framework represents a significant ad- vancement enhancing LLM agents' deeision-makingg capabilities. By automating the learning of reward models, we overcome the challenges of data scarcity and API limitations, potentially revolutionizing the application of LLMs in complex and interactive environments. This research paves the way for more sophisticated Al agents capable of tackling a wide range weal-world problems requiring multi-step decision- maassinommaiig.....
then utilized training data
INTRODUCTION
Developing AI agents capable of perceiving environments, understanding instructions, and acting to accomplish a wide range of tasks in interactive settings (Brooks, 1986) have many real-world applications, including virtual human assistants (Reed al., 2022: Casheekar et al., 2024), business process management (Kirchdorfer et al., 2024), and robotic process automation (Hong et al., 2023b; Rana et al., 2023; Palo al., 2023).
The recent advent of large generative models has revolutionized numerous applications, such as question answering (Rajpurkar et al., 2016), text summarization (Hermann al., 2015), and multi- modal understanding (Goyal et al., 2017; Yu et al., 2016; Chen et al., 2024). However, while these models excel in comprehension and generation tasks, their performance in decision-makin scenarios- such online shopping and scientific reasoning falls relative short of human capabilities. This disparity likely stems from the nature of the training data. Large generative models are typically
"Equal contribution.
Project page: https: //t:///amaa /ar/aaammapaeeee thub.io

---

Published as a conference paper at ICLR 2025
Existing Methods
Trijecooy
(e) Tree Planning
I'm looking for varo spent carnaa faxil aeccckee with laco cutrarcinn
Thought:
use swarch bar
Action: search)eans
Actort home furfie
Ownertton Hodetr
price
20AIGEUH
DOCCCAOFON
valan
Search
Search
Duoc Scursh
Act en (4
Dabuus
Poticy
Thought: should select
Prec 19941 10 B3AT
Coatty to fine-ture Pelicy Prccel (LLM)
Pequire messiee humenn arrectated data
Thought: should
SAXITh
Action ciequayNaad Nond
Maryy Parlcy Models guaing and close-astureed
NoW!
Has upree by
(b) Our Method
Reword
Trajectory
Ouidc Than Slanning
Thought shooud Mazr resuits...
Bookss Scondh
Model
Reword Model
Irpul Trajedtoiis
Autometic Reward Modell
right)
Delout Poiccy Model
Action: (INMB060D4CPPM)
Six 125
Figure I: In Fig. (a), we show that is difficult for LLM agents to generate multi-step plans in an interactive environment to achieve the instruction goal. However, it is relatively easy for an LLM to learn a reward model that can evaluate whether the trajectories meet the task instructions, as shown in Fig. 1 (b). In Fig. 1 (c), we show that learned reward model can be used to guide the default policy models to improve action planning.
pre-trained on readily available image and text corpora from the internet. In contrast, trajectory data for agent tasks, which require multi-step interaction with the environment, is more challenging to collect and does not naturally occur on the internet. Furthermore, current state-of-the-ar commercial LLMs, such as GPT-4V (OpenAI et al., 2024) and Gemini (Reid et al., 2024), only provide limited APIs for general users. This restriction renders it either infeasible rost-prohibitive to fine-tune these models for specific agent tasks, further impeding progress in this field.
Previous studies have explored the development of autonomous agents for decision- making tasks using large language models (LLMs). Early research (Yao et al., 2023a; Zheng et al., 2024; Deng et al., 2024) utilized instruction prompts with few-shot examples to direct LLMs in handling various agent tasks. These methods do not require task-specific fine-tuning but have shown limited performance on benchmarks requiring interaction with environments and precise action prediction. A different research direction involves collecting human preference data (Hong et al., 2023a) or distilling trajectory data from advanced commercial LLM APIs (Zeng et al., 2023; Deng et al., 2024) and fine-tuning smaller open-source L.MSs to create new policy models for agent tasks. However, this distillation process relies on advanced pre-trained agent models for trajectory data extraction, which are often unavailable, expensive, or subject to commercial restrictions. For instance, data from models such as GPT-4 or Gemini cannot be used for commercial purposes.
A fundamental premise of our approach is that, in most agent applications, evaluation is easier than generation (Karp, 1975; Naor, 1996). As illustrated in Fig. (a), generating correct multi-step solution to navigate to the target page is challenging since it needs to predict multiple actions and interact with the environment. However, it is relatively simple to evaluate whether the output action trajectories and environment states meet the provided intent to find a "vans sport canvas fashion sneaker". ". Building on this premise, we suggest that developing a reward model is more feasible than creating a policy model for agent tasks. With an effective reward model, becomes possible guide LLMs in planning tasks both effectively and efficiently. For instance, as depicted in Fig. (c), by integrating the reward model with an LLM-based agent and the Monte Carlo Tree Search (MCTS) algorithm (Silver et al., 2017; Coulom, 2006), we can simulate and evaluate the future states of agent tasks, thereby making better decisions for subsequent actions. This approach is analogous to mental simulation (Hegarty, 2004; Lake et al., 2017) in cognitive science, where humans envision the outcomes of potential actions to make better decisions in problem-solving.
While reward models can assist LLM agents in planning, developing these reward models presents significant challenges. Some prior studies have utilized powerful commercial LLM APIs as evaluators for tasks (Kwon et al., 2023a). Although these approaches have demonstrated effectiveness in certain applications, they rely on sfate-of-the- the-the-rt art LLM models for evaluation, which are often expensive and difficult to scale. In this paper, we introduce automated method to learn without relying on state-of-the-aa LLMs for guidance. Furthermore, previous work has not considered integrating the learned reward models with various planning algorithms for problem-solving.
reward models
2

---

Published as a conference paper at ICLR 2025
The process of learning the reward model involves three steps. Initially, we utilize LLM-based agent (e.g., Dubey et al. (2024)) to navigate in the environments, aiming to achieve a randomly proposed intent while collecting extensive action trajectory demonstrations. Subsequently, the LLM examines the collected trajectories and proposes a refined intent that the sampled trajectories actually accomplish. Additionally, we prompt the L.M generate negative trajectories that fail achieve the intended task. Finally, based on the synthetic data (intents, positive trajectories, and negative trajectories) collected, we train customized reward model using widely adopted vision-language models such as VILA (Lin et al., 2023) to evaluate whether the user's intent has been fulfilled by the action trajectories. With this automatic reward model, we enhance the performance of LLM-based agents in conjunction with various planning algorithms such as best of n, reflexion, and MCTS.
In summary, we introduce a novel framework ARMAP (autonomous Agents from automatic Reward Modeling And Planning) for LLM-based agents incorporating an automatic reward model that evaluates task completion, analogous to mental simulation in human cognition. This framework offers several advantages: (1) Effectiveness: It enhances the performance of various LLM agents across different tasks. (2) Flexibility: It eliminates the need for fine-tuning the LLMs themselves and allows for optimization of custom reward targets during inference, enabling more controllable generation. (3) Practicality: The training of the automatic reward model does not rely on labor- intensive labeling or state-of-the-art commercial LLMs, making it more feasible and widely applicable.
RELATED WORK
LLMs for Agent tasks. Our research is related to deploying large language models (LLMs) as agents for decision-making tasks in interactive environments (Liu al., 2023b; Zhou et al., 2023; Shridhar et al., 2020; Toyama et al., 2021). Earlier works, such as (Yao et al., 2023a), fine-tuned models like BERT (Devlin et al., 2019) for decision-making in simplified environments, such as online shopping mobile phone manipulation. With the advent of large language models (Brown et al., 2020; OpenAl et al., 2024). it became feasible to perform decision decision-making tasks through zero-shot few-shot in-context learning. To better assess the capabilities of LLMs as agents, several models have been developed (Deng et al., 2024; Xiong et al., 2024; Hong et al., 2023a; Yan et al., 2023). Most approaches (Zheng al., 2024; Deng al., 2024) provide the agent with observation and action history, and the language model predicts the next action via in-context learning. Additionally, some methods (Zhang al., 2023a; Li et al., 2023; Song al., 2024) attempt distill trajectories from state-of state-of-the-art the- language models to train more effective policy models. In contrast, our paper introduces novel framework that automatically learns reward model from LLM agent navigation, using it to guide the agents in making more effective plans.
LLM Planning. Our paper is also related to planning with large language models. Early re- searchers (Brown et al., 2020) often prompted large language models to directly perform agent tasks. Later, Yao et al. (2022) proposed ReAct, which combined LLMs for action prediction with thain-of-thought prompting (Wei et al., 2022). Several other works (Yao et al., 2023b; Hao et al., 2023; Zhao et al., 2023; Qiao et al., 2024) have focused on enhancing multi-step reasoning capabilities by integrating LLMs with tree search methods. Our model differs from these previous studies in several significant ways. First, rather than solely focusing on text generation tasks, our pipeline addresses multi-step action planning tasks in interactive environments, where we must consider not only historical input but also multimodal feedback from the environment. Additionally, our pipeline involves automatic learning of the reward model from the environment without relying on human-annotater data, whereas previous works rely on prompting-based frameworks that require large commercial LLMs like GPT-4 (OpenAl et al., 2024) to learn action prediction. Furthermore, ARMAP supports a variety of planning algorithms beyond tree search.
Learning from AI Feedback. In contrast to prior work on LLM planning, our approach also draws on recent advances in learning from AI feedback (Bai et al., 2022; Lee et al., 2023; Yuan et al., 2024; Sharma et al., 2024; Pan et al., 2024; Koh et al., 2024b). These studies initially prompt state-of-the-an large language models to generate text responses that adhere to predefined principles and then potentially fine-tune the LLMS with reinforcement learning. Like previous studies, we also prompt large language models to generate synthetic data. However, unlike them, we focus not on fine-tuning a better generative model but on developing a classification model that evaluates how well action trajectories fulfill the intended instructions. This approach is simpler, requires no reliance on ofate-of-the. art LLMs, and is more efficient. We also demonstrate that our learned reward model can integrate with various LLMS and planning algorithms, consistently improving their performance.
3

---

Published as a conference paper at ICLR 2025
Synthesize Initialized Task Instructions
Sampled Trajectories
Automatically Generated Dataset
Positive Trajectories
looking for jeans x size, and price lower than 200 dolltars.
Task Instructions
Refine Task Instructions
Negative Trajectories
am looking straight leg jeens sandblasst light color, also 40w 341 sire, price lower 170 dollars.
Sample Trajectories
Training Data
Training Phase
Reward
Actions
Environment
1.0
Searc
Sample Negative Trajectories
0.0
scarch(..jeans...
Actions
Environment
B00K1E04EC Straightt Jeans, $16.44 to $45.61
BOODNBRZKW Regular-Fit Jean, $48.66 to $219. 21
Inference Data
Inference Phase
searchl..jeans..]
Reward
0,9
click(600K1
$15.44 $48.61
clicKBOOK1E04EC]
Size:
G 321
0.1
click[4Dw x 32I]
click[ADw x 341]
Sine: O 40 37W
$15.44
$16.44
Agent Model
Beward Model
Task Instruction
Size: @ 40w 321 40w 341 341...
click[Buy Now]
Now
cliclllyy Now]
Positive Trajectories
Negative Trajectories
Query Trajectories
Figure 2: The pipeline of our ARMAP framework. We first generate an initial task instruction using LLMs with in-context learning and sample trajectories aligned with the initial language instructions in the environment. Next, we use the LLM to summarize the sampled trajectories and generate refined task instructions that better match these trajectories. We then modify specific actions within trajectories to perform new actions in the environment, collecting negative trajectories in the process. Using the refined task instructions, along with both positive and negative trajectories, train a lightweight reward model to distinguish between matching and non-matchin non-matching trajectories. The learned reward model can then collaborate with various L.M agents improve task planning.
Inference- Time Scaling, Snell et al. (2024) validates the efficacy of inference- iiference-time scaling for language models. Based on inference-time scaling, various methods have been proposed, such as random sampling (Wang et al., 2022b) and tree-search methods (Hao et al., 2023; Zhang et al., 2024a; Guan al., 2025; Zhang et al., 2023b). Concurrently, several works have also leveraged inference- time scaling to improve the performance of agentic tasks. Koh et al. (2024b) adopts training- free approach, employing MCTS to enhance policy model performance during inference and prompting the LLM to return the reward. Gu et al. (2024) introduces a novel speculative reasoning approach to bypass irreversible actions by leveraging LLMs or VLMs. It also employs tree search to improve performance and prompts LLM output rewards. Yu et al. (2024) proposes Reflective-MCTS to perform tree search and fine-tune the GPT model, leading to improvements in Koh et al. (2024a). Putta al. (2024) also utilizes MCTS to enhance performance et al. (2023a) and real-world booking environments. Lin et al. (2025) utilizes the stepwise reward to give effective intermediate guidance across different agentic tasks. Our work differs from previous efforts in two key aspects: (1) Broader Application Domain. Unlike prior studies that primarily focus on tasks from a single domain, our method demonstrates strong generalizability across web agents, mathematical reasoning, and scientific discovery domains, further proving its effectiveness. (2) Flexible and Effective Reward Modeling. Instead of simply prompting an LLM as reward model, we finetune a small scale VLM (Lin et al., 2023) to evaluate input trajectories.
web-based tasks such as Yao
MODEL
In this section, we provide detailed introduction to our framework, autonomous Agents from automatic Reward Modeling And Planning (ARMAP). The framework includes automated reward data generation in section 3.2, reward model design in section 3.3, and planning algorithms in section 3.4.
3.1 BACKGROUND
The planning tasks for LLM agents can be typically formulated as a Partially Observable Markov Decision Process (POMDP): (X,S, A, O, T), where:
4

---

Published as a conference paper at ICLR 2025
X is the set of text instructions;
S is the of environment states;
A is the set of available actions at each state;
O represents the observations available to the agents, including text descriptions and visual information about the environment in our setting;
. : x A S is the transition function of states after taking actions, which is given by the environment in our settings.
Given a task instruction z E X' and the initial environment state sg E S, planning tasks require the LLM agents to propose sequence of actions Ja=1 that aim to complete the given task, where an E represents the action taken at time step n, and N is the total number of actions executed in trajectory. Following the n-th action, the environment transitions to state 8n, and the agent receives a new observation On, Based on the accumulated state and action histories, the task evaluator determines whether the task is completed.
An important component of our framework the learned reward model R, which estimates whether trajectory h has successfully addressed the task:
h),
(1)
where h: = ((a, 1N for) o- {a, NN corresponding environment observations, and is the predicted reward from the reward model. By integrating this reward model with LLM agents, we can enhance their performance across various environments using different planning algorithms.
are the actions taken in the trajectory, (oz 1.W In=0 are the
3.2
AUTOMATIC REWARD DATA GENERATION.
To train a reward model capable of estimating the reward value of history trajectories, we first need to collect set of training language instructions (rmm m== where M represents the number of instruction goals. Each instruction corresponds to set of positive trajectories {A J that 1M match the instruction goals and a set of negative trajectories [m that fail meet the task requirements. This process typically involves human annotators and is time-consuming and labor- intensive (Christiano et al., 2017; Rafailov al., 2024). As shown in Fig. of the Appendix. we automate data collection by using Large Language Model (LLM) agents to navigate environments and summarize the navigation goals without human labels.
Instruction Synthesis. The first step in data generation is to propose a task instruction for given observation. We achieve this using the in-context learning capabilities of LLMs. The prompt for instruction generation is shown in Fig. 9 of the Appendix. Specifically, we provide some few-shot examples in context along with the observation of an environment state to an LLM, asking it to summarize the observation and propose instruction goals. In this way, collect set of synthesized language instructions { Im=1s where M represents the total number synthesized instructions.
af
Trajectory Collection. Given the synthesized instructions x and the environment, an LLM--based agent is instructed to take actions and navigate the environment to generate diverse trajectories (xraw Am J aimed accomplishing the task instructions. Here, h.m represents the m-th history trajectory, which consists of N actions {a, J.M and N + 1 environment observations {o,, J. I==a- Due to the limited capabilities of current LLMs, the generated trajectories fim may always align well with the synthesized task instructions E,. To address this, we ask the LLM to summarize the completed trajectory hn and propose a refined goal x",, This process results in a set of synthesized demonstrations (1", hm m= where M, the number of refined task instructions.
Pairwise Data Construction. To train a reward model capable of distinguishing between good and poor trajectories, we also need trajectories that do not satisfy the task instructions. To create these, we sample additional trajectories that differ from { ) and do not meet the task requirements by modifying actions hm and generating corresponding negative trajectories {h J. For clarity, refer to the refined successful trajectories {Imm h and the unsuccessful ones {Im h }. These paired data will be used to train the reward model described Section 3.3, allowing it to estimate the reward value of any given trajectory in the environment.
5

---

Published as a conference paper at ICLR 2025
3.3 REWARD MODEL DESIGN.
Reward Model Architectures. Theoretically, we can adopt any vision-language model (Liu et al., 2023a; Li et al., 2024b;a; Lin et al., 2023) that can take a sequence of visual and text inputs as the backbone for the proposed reward model. In our implementation, we use the recent VILA model (Lin et al., 2023) as the backbone for reward modeling since it has carefully maintained open-source code, shows strong performance on standard vision-language benchmarks like (Fu et al., 2023; Goyal et al., 2017; Hudson& & Manning, 2019), and support multiple image input.
The goal of the reward model is to predict reward score to estimate whether the given trajectory (Im: (m)) has satisfied the task instruction or not, which is different from the original goal of VILA models that generate series of text tokens respond to the task query. To handle this problem, additionally add a comneccon layer for the model, which linearly maps the hidden state of the last layer into scalar value.
Optimazation Target. Given the pairwise data that is automatically synthesized from the envi- ronments in Section 3.2, optimize the reward model by distinguishing the good trajectories (Im. ) from bad ones (Im, h ). Following standard works of reinforcement learning from human feedback (Bradley & Terry, 1952; Sun et al., 2023b;a),
treat the optimization problem of the reward model as a binary classification problem and adopt a cross-entropy loss. Formally, we have
[()) - E (2. Am. [log a (Ro (Iim hith Ro(Em hm)),,
(2)
where a is the sigmoid function and 0 are the learnable parameters in the reward model R. By optimizing this target, the reward model is trained give higher value scores the trajectories that are closer to the goal described in the task instruction.
3.4 PLANNING WITH LARGE VISION-LANGAUGE REWARD MODEL.
After getting the reward model to estimate how well a sampled trajectory match the given task instruction, we are able to combine it with different planning algorithms to improve LLM agents' performance, Here, we summarize the typical algorithms we can adopt in this paper.
Best of N. This is simple algorithm that can adopt the learned reward model improve the LLM agents' performances. We first prompt the LLM agent to generate n different trajectories independently and choose the one with the highest predicted reward score the prediction for evaluation. Note that this simple method is previously used in natural language generation (Zhang al., 2024b) and we adopt it in the context of agent tasks to study the effectiveness of the reward model for agent tasks.
Reflexion. Reflexion (Shinn al., 2024) is a planning framework that enables large language models (LLMs) to learn from trial-and-error without additional fine-tuning. Instead of updating model weights, Reflexion agents use verbal feedback derived from task outcomes. This feedback is converted into reflective summaries and stored in an episodic memory buffer, which informs future decisions. Reflexion supports various feedback types and improves performance across decision- making, coding, and reasoning tasks by providing linguistic reinforcement that mimics human self-reflection and learning.
MCTS. We also consider tree search- planning algorithms like Monte Carlo Tree Search (MCTS) (Coulom, 2006; Silver et al., 2017) to find the optimal policy. There is a tree structure constructed by the algorithm, where each node represents a state and each edge signifies action. Beginning at the initial state of the root node, the algorithm navigates the state space to identify action and state trajectories with high rewards, as predicted by our learned reward model.
The algorithm tracks 1) the frequency of visits to each node and 2) a value function that records the maximum predicted reward obtained from taking action a in state s. MCTS would visit and expand nodes with either higher values (as they lead to high predicted reward trajectory) or with smaller visit numbers (as they are explree.... We provide more details in the implementation details and the appendix section.
4 EXPERIMENTS
In this section, we conduct series of experiments demonstrate the effectiveness of the proposed framework for agent tasks. First, we evaluate the framework's performance on standard agent benchmarks (Yao et al., 2023a; Wang al., 2022a; Yao et al., 2023b), detailed in Section 4.2. Next,

---

Published as a conference paper at ICLR 2025
show how customizing the reward target during inference allows us generate more tailored action plans, as described in Section 4.3. Finally, we conduct ablation studies Section 4.4. Before delving into the experimental results, we provide overview of our experimental setup.
4.1 EXPERIMENTAL. SETUP
Environments.
We evaluate the ARMAP framework in three different environments:
Wehshop well-known environment for online shopping (Yao al., 2023a), where the agent must search for and select products on the website to obtain a final result. Following the setup of AgentBench (Liu al., 2023b) for LLM evaluation, we test the model on the validation split, using the default matching reward as the evaluation metric.
SedenceWorree (Wang et al., 2022a) an interactive benchmark designed for embodied science experiments. It places agents in simulated text-based environment where they must perform elementary science experiments by navigating the environment, manipulating objects, and observing outcomes. The aim assess whether AI models can apply scientific knowledge, rather than merely retrieve or assemble information. We evaluate the framework on both seen and unseen splits.
Game of 24 is mathematical game where the agent is given four numbers and must arithmetic operations (addition, subtraction, multiplication, and division) to make the numher 24. For instance, given the input 5, 7, 11,' one possible solution is '(7-3) (11 1-) 24'. Following Yao et al. (2023b), we selected 100 challenging puzzles, specifically those indexed from 901 1,000, and the performance metric is the success rate across these puzzles. As shown in Fig. 7 of the Appendix, we use the chain-of-thought prompting technique, prompting the LLM agents to output intermediate steps followed by the final answer. Each step of the solution is considered an action.
LLM Setup. Our framework requires LLM models to act as agents, generating synthetic task instructions from the environment along with few-shot examples in the prompt context. We also deploy agents to perform these synthetic tasks in the environment, collecting diverse trajectories for further analysis. In this paper, we primarily use the Llama3-70b- 700-instruuc model (Dubey et al., 2024) to synthesize training data for the automatic reward models, as it is open-source, easy to deploy locally, and delivers robust performance. We avoid state-of-the-an commercial models like GPT-4 or Gemini due to their high costs and the complexity of reproducing results caused by frequent model updates, making them less suitable for our research objectives.
To evaluate the performance of various LLM agents, we serve a representative set of LLM APIS locally, balancing model diversity with affordable serving costs. We identify the LLMs by their model family and size. Specifically, these are Llama70B, Llama&B, Mistral7B, and Phi3.8B We note that these open-source model families are frequently updated, and we provide the current model links in the Appendix A.3. All models can be easily set up using the vLLM library (K (Kwon et al., 2023b) and single H100 GPU.
Baselines. We implement our ARMAP framework using different planning algorithms, including Reflexion, oest-of-N, and MCTS, which we denote as ARMAP-R. ARMAP-B, and ARMAP-M, respectively. We limit the maximum number of trajectories Science World and Webshop environments to systematically evaluate the pipeline's effectiveness across different L.MM agent backbones. We also compare the model with baselines that do not use reward model guidance: Sampling and Greedy. For the Game of 24 environment, we follow the setup of previous study (Yao et al., 2023b) and set the maximum number of explored trajectories to 100. For Sampling, we set the model temperature to I and sample action trajectories using ch th-tuughhtt prompting (Wei et al., 2023). For Greedy, we set the temperature 0, generating the action sequence with the highest probability. Further implementation details are provided in the Appendix. We will release all the code, model, and data for easy reproduction upon acceptance.
ARMAP explore 10 in the
4.2 EFFECTIVENESS FOR REWARD PLANNING.
In this section, we investigate the effectiveness of the framework across different language mod- els (Dubey al., 2024; Jiang et al., 2023; Abdin et al., 2024) and various planning algorithms. The results are shown in Table Based on the table, we have the following observations. First, our proposed pipeline is effective, as it consistently outperforms the Sampling and Greedy baselines
7

---

Published as a conference paper at ICLR 2025
Science
Backbone
Algorithms
Webshop
Game24
Average
unseen
Sampling
52.0
53.9
50.6
9.6
38.0
Greedy
50.4
57.2
55.1
6.0
37.5
Llama70B
ARMAP-R
56.5
59.0
56.7
16.0
43.5
ARMAP-B
62.0
57.3
57.0
19.0
46.1
ARMAP-M
66.8
58.2
55.9
24.0
49.3
Sampling
56.4
24.5
20.6
2.0
27.0
Greedy
57.7
29.9
23.8
2.0
28.9
Llama8B
ARMAP-R
58.3
31.2
28.0
6.0
31.3
ARMAP-B
59.3
35.7
28.1
11.0
34.1
ARMAP-M
60.2
32.5
24.9
9.0
32.6
Sampling
17.7
18.4
17.1
1.0
12.2
Greedy
37.2
21.1
19.6
1.0
19.5
Mistral7B
ARMAP-R
54.1
21.7
19.7
2.0
25.6
ARMAP-B
54.4
24.5
21.2
2.0
26.4
ARMAP-M
58.2
30.0
23.4
4.0
29.6
Sampling
34.7
10.0
7.6
2.0
15.2
Greedy
42.4
9.5
6.5
2.1
17.5
Phi3.8B
ARMAP-R
53.3
9.6
7.2
4.0
21.9
ARMAP-B
52.1
20.0
17.0
9.0
26.5
ARMAP-M
53.7
28.3
24.3
10.0
30.0
Table I: Effectiveness of the proposed method different benchmarks. Our ARMAP framework consistently outperforms the baselines across different language models.
across different planning algorithms. Additionally, we observe that the average improvement is more significant weaker models, such Phi (Abdin et al., 2024) and Mistral-7B (Jiang al., 2023), compared to stronger models like Llama3- 1-7BB (Dubey et al., 2024). We believe this is because weaker models explore more low-reward trajectories, providing greater opportunities for the reward model to improve performance.
Among the three planning algorithms, MCTS performs the best average, likely due to its superior mechanisms for identifying higher-reward trajectories and searching less-explored trajectories. We also notice that Reflexion performs the worst on weaker models like Mistral7B and Phi3.8B, We suspect this because Reflexion was designed for
agents and requires the LLM agent to possess strong capabilities for learning from trial and error. Finally, we present qualitative results of different methods in Fig. 3, where it is clear that our ARMAP generates better trajectories than the baselines, aided by the guidance of automatic reward models. Appendix A.5, we analyze several failure cases, offer more detailed insights into the limitations of the current approach, and suggest potential improvements in reward modeling.
4.3
CONTROLLABLE GENERATION.
Another benefit of our ARMAP pipeline is that we can customize our reward targets during inference. allowing generate more controllable action sequences, rather than solely maximizing the predicted rewards. Agent fine-tuning methods (Li et al., 2023; Zeng et al., 2023) find it challenging achieve this goal since agent behaviors typically fixed during inference. We conducted experiments in the Webshop environment to evaluate the impact of customizable reward targets. In addition to the original objective of maximizing the predicted reward R(z, h). we defined two additional optimization targets. First, we aimed to minimize the number of actions in the trajectory NumberOfAction(?) (h). Second, we sought minimize
history, defining the reward target as (xx,A the price of the target product, with customized target of R2z h) PriceOfProduct(h) Table presents the results. By applying a length penalty on the reward target for ARMAP-M. we reduced the average action length from 4.5 to 4 and the average product price from 97.9 to 69 ... while maintaining comparable performance on the default matching reward. Similar performance was observed for ARMAP-B. Additionally, we provide qualitative example in Fig. 4. From this example, we see that our customized reward target successfully guided the LLM agent to purchase products with fewer action steps while still finding the target product.
8

---

Published as a conference paper at ICLR 2025
I'm looking for furniture to make living room and dinning room nice, and price lower than 300 dollars.
Search
Default Policy Model
Back to Search
Back to Search
search(furniture]
search(furniture]
click(back to search]
B09LSBCVOT
B07D54V8HT
search(living room dining room furniture]
clck[B09LSBCVar]
Reading Room Set, Restroom Mirrer,
$2488.95
cick[B09MQFMML8]
BO9MQFMCLS
B08J9WPMJ8
Table Set for 4 Price: Table Set for 6 Price:
click[Buy Now]
Buy Now
click[Buy Now]
$259.99
$853.29
Automatic Reward Model
need blue women coat, and price lower than 80 dollars.
Search
Searchh
search(blue women
Back to Search
search[blue sherpa
Back to Search
coat price 80.00]
sweatshirt 80.00]
click(Bo9LRVCAK2 BOSLAVCAK2
click(BC9LRVC4K3]
808J9W/MJ8
BO9LRVC4K2
BORISWPMIA
poo
click[blue]
Price: $9.19 to $10.96
clickflarge] click[Buy Now]
Price: $9.19 $10.96
click(Buy Now]
@ blue o purple o black
Size: large O medium
Two qualitative results the Webshop task. The figure shows two examples utilizing the advantages of our ARMAP framework and we able to correct In the top example, when the search results do not meet the requirements, our ARMAP method leverages the advantage of the structure backtrack and search again, thereby retrieving the appropriate target item. In contrast, existing methods fail to backtrack when the target item is not found. In the bottom example, by using the ARMAP evaluate different states in the environment, our method is able to select the color that offers a higher reward and better meets the requirements when choosing between size and color, rather than mistakenly selecting the wrong size. These two examples sufficiently demonstrate the advantages of our method compared to traditional approaches.
Figure 3:
made by existing methods.
Algorithms
Action
Price
Reward
Greedy
4.6
102.4
50.4
ARMAP-B
4.7
102.2
62,0
ARMAP-M
4.5
97.9
66.8
CRMMAP-B+Length-PPP
3.9
98.8
60.3
ARMAP-M+Longtthae penalty
4.0
102.1
65.5
ARMAP---PPrie-pnnatyy
5.0
65.5
57.5
ARMAAP-M-Price-pnnally 4.3
69.0
62.4
Table Controllable Trajectory Generation. We show that we can generate controllable trajectories like shorter action lengths and lower prices by customizing reward targets. We use Llama70B as the default API for action prediction.
4.4 ABLATION STUDIES.
We conduct ablation studies to investigate the effectiveness of the framework. Specifically, we aim to answer the following questions: Q1. Can we train policy model with fully supervised learning to handle multi-step tasks from the synthesized trajectory data? Q2. Can large, general language model be used as the reward model to perform guidance without automatic reward learning?
We conducted experiments using the ScienceWorld benchmark, and the results are shown Table 3. When comparing our pipeline the SFT model trained using our reward backbone VILA3B, we observed that although the policy model trained through fully supervised learning performed reasonably well (18.6), it still lagged behind the performance of our planning framework (28.3). This suggests that learning policy model is more challenging than learning a reward model, highlighting the effectiveness of our proposed ARMAP pipeline (answering Q1).

---

Published as a conference paper at ICLR 2025
Instruction: Please select 1 pound, certitied orgenic sea selt shaker in the flavor tripia blend flakes, and price
Sea Semsring deaeees Orpant
See Senterings Tredn
Rasorable 12
Blan Pakes Outces
Barcol
.000 dollars.
Gready
ARMAP
ARMAP L-panalty
Actient: sollen:ccore [ pound contified
Actlent: scorchiorganic sob colt sholoer ipfl bland flebes
Actiont: Socro salt
shasber trip biennd
shoker triple blend fiakes pound price 80 00)
takes 50.00)
00)
Action2:
click/oon stMbro)
Action2: olekg=o7ehazry Whingg
Action2: DUck (B0007MMLMM))
Organic!
Actiond: slick( po ppack of
Action?: soarch (oatitted trglniic...
clickbookk
Actiond: connchCorgaric... Takkes 30.00
clickbaack
Actiond: tr pi blwnnd fialows)
facis pound price 4 90.00)
Actiond: slick Nano)
Action8: catiinn.ciick(oonoo (lMBeee00ee))))) Price
Action6: LIOM)B007MMLLMM
pound 11]
Brie
Action?: clickttr bland lisboas
Actiond: olekkoock to search) Seacce
ActlonB: cllkkBuy Novd /
Failed with Reward 0.0, Steps =
Succeed with Reward 1,0, Steps = Succead with Rewerd 1.0, Steps
Figure 4: typical example of customized reward target for shorter trajectory generation. the left, we show the default greedy decoding generates a long trajectory without finding the target product. the middle, we show our default reward can guide the LLM agent to generate a correct but long trajectory. On the right, show framework with a customized reward target for shorter trajectories, which finds a correct and short trajectory for the target product.
Models
Model Base
ScienceWorld seen
Greedy
Phi3.8B
SFT-Policy
VILA3B
18.6
ARMAP-B w/o
16.0
Llama70B and Phi3.8B
ARMAP-M w/o R
26.5
ARMAP-B ARMAP-M
20.0
VILA3B and Phi3.8B
28.3
Table 3: Ablation study of the proposed framework. Our ARMAP framework is more effective than directly finding a policy model and using the general LLM for reward generation.
Next, we replaced our smaller 3B reward model with much larger language model, Llama3- --70B, and used few-shot prompting to predict the reward the extracted trajectories. We found that this larger model also improved performance compared to the default greedy model, demonstrating the effectiveness of our planning framework. However, still performed worse than our pipeline using automatic reward learning, despite the Llama3- 1-B being about 20 times larger, further showcasing the efficiency and effectiveness of our approach (answering Q2).
We provide additional ablation experiments in the Appendix A.2, including the data quality from various LLMs, reward modeling target and computational efficiency.
5 CONCLUSION
We propose framework, ARMAP, for large language model (LLM) agents to manage tasks that require multi-step decision-making and environmental feedback, such as online shopping or scientific reasoning. This framework allows L.LM-based agents to enhance task planning by autonomously learning reward model from the environment, without the need for human labeling. The method utilizes pre-trained LLM agents to generate diverse action trajectories within an environment, which are then evaluated by separate LLM based on the task's intent. These evaluations help train capabilities. The framework enhances the
reward model that strengthens the agents' decision-makinn performance of LLM agents in addressing complex tasks and mitigates issues related data scarcity and API limitations. Its effectiveness is demonstrated across various benchmarks, representing a significant advancement in the development of AT agents for real-world, multi-step problem-solving.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, and al. Phi-3 technical report: highly capable language model locally on your phone, 2024. URL httt::://axxv..oo/ org /b/////////44. 14219.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22/2.08000 2022.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 1952.
R. Brooks. A robust layered control system for mobile robot. IEEE Journal on Robotics and Automation, 2(1): 23, 1986. doi: 1AL100//RRA.198.1000003...
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, and et al. Language models are few-shot learners, 2020. URL https://aarkiii org/ abs/ 14165.
Avyay Casheekar, Archit Lahiri, Kanishk Rath, Kaushik Sanjay Prabhakar, and Kathiravan Srinivasan. A contemporary review on chatbots, ai-powered virtual conversational agents, chatgpt: Appli- cations, open challenges and future research directions. Computer Science Review, 52: 10062,, 2024.
Zhenfang Chen, Rui Sun, Wenjun Liu, Yining Hong, and Chuang Gan. Genome: Generative neuro-symbolic visual reasoning by growing and reusing modules. In ICLR, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 2017.
Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games. Springer, 2006.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinay Pandey, Abhishek Kadian, Ahmad Al-Dahle, and et al. The Ilama 3 herd of models, 2024. URL httpss:/aarxvv.org/bbsss
21783.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 133344 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Yu Gu, Boyuan Zheng, Boyu Gou, Kai Zhang, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi. Huan Sun, and Yu Su. Is your I1m secretly a world model of the internet? model-based planning for web agents. arXiv preprint
Stdi::21110055555 2024.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small ilms can master math reasoning with self-evolved deep thinking, arXiv preprint arXiv:2501. 255.:2500..4599, 2025.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong. Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv, 2023.
Mary Hegarty. Mechanical reasoning by mental simulation. Trends in cognitive sciences, 2004.

---

Published as a conference paper at ICLR 2025
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt. Will Kay, Mustafa Su- leyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL http:hhttt/////teeeeeeeeee http:////ooceeen....... neur suurips..o pcppcccppprrr.....pppppp/ppp .ddf.
Wenyi Hong, Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023a.
Yining Hong. Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. NeurlPS, 2023b.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.orgg abs/2310. 06825..
Richard M Karp. On the computational complexity of combinatorial problems. Networks, 1975.
Lukas Kirchdorfer, Robert Blümel, Timotheus Kampik, Han Van der Aa, and Heiner Stuckenschmidt. Agentsimulator: An agent-based approach for data-driven business process simulation. In 2024
6th International Conference on Process Mining (ICPM), pp. 97-104. IEEE, 2024.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang. Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401. I 3649, 2024a.
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. arXiv, 2024b.
Minac Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models, 2023a. URL
orggabs/2300 00000..
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023b.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and bruin sciences, 2017.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv, 2023.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Junyan Li, Delin Chen, Tianle Cai, Peihao Chen, Yining Hong, Zhenfang Chen, Yikang Shen, and Chuang Gan. Flexattention for efficient high-resolution vision-language models. ECCV, 2024a.
Junyan Li, Delin Chen, Yining Hong. Zhenfang Chen, Peihao Chen, Yikang Shen, and Chuang Gan. Covim: Composing visual entities and relationships in large language models via communicative decoding. In ICLR, 2024b.
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023.
12

---

Published as a conference paper at ICLR 2025
Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, and Kai- Kai-Wei Chang. Qlass: Boosting language agent inference via q-guided stepwise search. arXiv preprint arXiv:2502.02584 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023a.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang. Yuxiao Dong, and Jie Tang. Agenthench: Evaluating iims agents. arXiv preprint arXiv: 2308.03688 2023b.
Moni Naor. Evaluation may be easier than generation. In Proceedings of the twent--eighth annual ACM symposium Theory of computing, pp. 74-83, 1996.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, and et al. Gpt-4 technical report, 2024.
Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever, Markus Wulfmeier, Nicolas Heess, and Martin Riedmiller. Towards a unified agent with foundation models, 2023. URL https: //wrxxv.org///a/ee/tt//t wtt.r////b////////////// 2tt.//22700000 09668..
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous
evaluation and refinement of digital agents, 2024.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408. 002008001990 07190,71999999 2024.
Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Agent planning with world knowledge model, 2024. URL https:///axiie org/ /405. 14205.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2024.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL https://aaxxv.. http:p/:///axxt.org/abb///////// 166.. 05250.
Krishan Rana, Jesse Haviland, Sourav Garg. Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using scene graphs for scalable task planning. In 7th Annual Conference on Robot Learning, 2023. URL wMpOM00Ss7a.
http://ppprreii......./.. net/forumiid=
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, and et al. A generalist agent, 2022. URL https:///axxxt ht//ab/s/205.0e0 06175.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieser,, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv, 2024.
Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. Agentelinic: multimodal agent benchmark to evaluate ai in simulated clinical environments, 2024.
Archit Sharma, Sedrick Keh, Eric Mitchell, Chelsea Finn, Kushal Arora, and Thomas Kollar. A critical evaluation of ai feedback for aligning large language models. arXiv, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Mohit Shridhar. Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottahii Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10740-10749, 2020.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté. Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://arxiv. org/ abs////s..00./// 03768.
David Silver, Thomas Hubert, Julian Schrittwieser, Toannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv, 2017.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can more effective than scaling model parameters. arXiv preprint arXiv:2408. 03311,, 2024.
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization of LLM agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics. Association for Computational Linguistics, 2024.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv, 2023a.
Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with principle-following reward models. arXiv, 2023b.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. arXiv, 2021.
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than 5th grader?, 2022a. URL https: http///axxvvor//e/ee orriii.org/aaaee http/bbs/2003 2203.07540.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang. Aakanksha Chowdh- cry, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. arXiv preprint 2 2022b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https: http///axvvvv.e wwwp:iii.orr/bb////// 11903.
Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! Ilm agent learning via iterative step-level process refinement. arXiv, 2024.
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang. Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv, 2022.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable
real-world web interaction with grounded language agents, 2023a. URL abs/2207.0122 00070001066
14

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023b. 10601.
URL https://arxiv, org/
Licheng Yu, Patrick Poirson, Shan Yang. Alexander C. Berg, and Tamara Berg. Modeling context in referring expressions, 2016. URL https://arxiv, wwwpariwwii..oiaoboaoeee org/abs/////oooooooooooooo 20222.......
Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. Exact: Teaching ai agents explore with reflective-mets and exploratory learning. arXiv preprint arXiv:2410.02052, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv, 2024.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv, 2023.
Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with Ilama-3 8b. arXiv preprint arXiv:2406.07394, 2024a.
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv, 2023a.
Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B Tenenbaum, and Chuang Gan. Planning with large language models for code generation. ICLR, 2023b.
Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Improving reinforcement learning from human feedback with efficient reward model ensemble. arXiv, 2024b.
Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models commonsense knowledge for large-scale task planning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is generalist web agent, if grounded. In Forty-first International Conference on Machine Learning. 2024.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, al. Webarena: realistic web environment for building autonomous agents. arXiv preprint arXiv:2307. 13854, 2023. URL https: https///wwbbbbbedeeee.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
In this section, we provide supplementary material for the main paper.
A.1 EXPERIMENTS ON ALFWORLD AND AGENTCLINIC.
We extend our experiment on ALFWorld (Shridhar et al., 2021). a classic environment for House- Holding, where the agent must accomplish tasks in physical house-holding environments, like "Put pan on the dining table". Following the setup of AgentBench (Liu et al., 2023b) for LLM evaluation, we test the model on the dev and std split, using the default success rate the evaluation metric. Specifically, we used L.LMMa-3.1-777 samples with our data generation pipeline. Then we train a reward model with these synthesized data. We evaluate our ARMAP framework on ALFWorld using various planning algorithms, including Reflexion and Best-of-N, which we refer to as ARMAP-R and ARMAP-B, respectively. Additionally, we compare our approach with two baseline methods that do not incorporate reward model guidance: Sampling and Greedy. The results are shown below. As shown in Table 4, our model still performs well in this challenging environment, which contains diverse scenes and long-horizon planning tasks.
to generate around 1600 pairs of positive and negative
Models
ALtWorddsstd
ALFWorld-dev
Sampling
0.13
0.14
Greedy
0.18
0.30
ARMAP-R
0.22
0.35
ARMAP-B
0.30
0.45
Table 4: Experimental Results ALFWorld.
We also extended our experiments to ClinicalAgent (Schmidgall et al., 2024), an environment designed for medical decision decision-making tasks. ClinicalAgent evaluates models on their ability to interpret clinical scenarios and make accurate, high-stakes decisions. Results of ClinicalAgent are provided Table 5, further supporting the versatility of ARMAP in domains requiring precise reasoning.
Models
AgentClinicMMedQAA
Sampling 11.89
14.02 Greedy
ARMAP-B 44.33
Table 5: Experiments Results on AgentClinic.
A.2 ABLATION STUDY.
Dependence on Quality of Synthetic Data from Various LLMs. We choose ScienceWorld and conduct experiments to study the effectiveness of different reward models. As shown in Table the left column represents the results of using LLaMA-8B greedy directly and the Best of N results of LLaMA-8B with the reward model trained by the data generated from LLaMA-70BB LLaMA-8B. Mistral-7B, and Phi-3.8B, respectively. Greedy is our bascline result and it can be observed that using the reward model leads to better experimental outcomes. Among all the results, ILaMA-70B achieves the best performance. Compared to the other three models, LLaMA-70P has the largest scale and is naturally the most capable model. LLaMA-8B and Mistral-7B have a similar number of parameters, and in the Sorld task, Mistral-7B performs better than LLaMA-SB. Phi-3.88 8B the smallest of these models, yet it still achieved very good results. Notably, compared to the larger-scale LLaMA-8B and Mistral-7B, Phi-3.8B still scored better. These results indicate that our method exhibits good robustness when faced with LLMs of different scales and capabilities. Even with the smallest model, our method can still achieve good results. From these experimental
16

---

Published as a conference paper at ICLR 2025
Models
SciWorld-seen
SciWorld-unseen
Greedy
29.9
23.8
L0B
35.7
28.1
Llama8B
32.2
24.7
Mistral7B
33.7
26.5
Phi3.8B
34.7
26.9
Table 6: Experiments of training data generated from various LLMs.
outcomes, it is clear that method does not overly rely on the capabilities of language models. In other words, our method is highly efficient and robust.
To further investigate the optimization target of the reward model, conduct experiments to compare the performance of pairwise comparison and binary classification as learning methods for the reward model. Specifically, in the classification setting: each input pair treated as a positive and a negative example. The model trained to predict a score positive examples and 0 for negative examples. The comparative results shown in Table 7. Across all settings, pairwise comparison consistently outperforms binary classification, This confirms that pairwise comparison captures nuanced preferences more effectively than binary classification, leading to better reward modeling and overall task performance.
Reward Modeling Target.
for
Classification Unseen
Comparative Unseen
Backbone
Algorithms
Seen
Seen
ARMAP-R
57.0
55.4
59.0
56.7
LLaMA-70B
ARMAP-B
47.2
43.3
57.3
57.0
ARMAP-R
29.0
24.2
31.2
28.0
LLaMA-8B
ARMAP-B
27.5
22.2
35.7
28.1
ARMAP-R
17.8
18.2
21.7
19.7
Mistral-7B
ARMAP-B
19.1
17.3
24.5
21.1
ARMAP-R
8.6
4.8
9.6
7.2
Phi-3.8B
ARMAP-B
17.7
13.7
20.0
17.0
Table Comparison of the Classification target and Comparison target on ScienceWorld.
Computational Efficiency Analysis. We further study the data demands of the reward modelings. We show the performance of using different amounts of training data. Table 8 and Table 9, we selected ScienceWorld and used ARMAP-B as the experimental setting. In the leftmost column, we listed the different LLMs used our study. In the first row, we introduced VILA-3B, VILA-13B, and LLaVA- 13B, to compare the impact of different sizes and types of reward models the final outcomes. In the last two columns, we trained the reward models using 1/5 and 1/25 the original training dataset size, respectively, assess how varying amounts of training data affect method. (1) As seen, the effectiveness of our method continues to improve with increasing reward model sizes. However, in the experiments with LLaMA-SB and Phi-3.8B, despite using more potent reward models, there was no improvement in results. We believe that in the processes of planning and reasoning, the capability of the policy model still plays dominant role. If the policy model is more robust, and concurrently. if we enhance the capability of the reward model, we can continuously achieve better results. (2) We also observe that the performance of LLLVA-13B
not as good as VILA-13B. We attribute this to VILA being an improved version of LLaVA, and it utilizes an interleaved image-text dataset in its training, which better aids the model in perceiving, understanding. and handling multimodal information. Hence, VILA outperforms LLa VA. (3) From the Table 8 and Table 9, is evident that regardless of whether the data is seen or unseen, increasing the model size improves the final experimental results. If
the results of the VILA-3B model as benchmark and compare it with the two settings, 1/5 data and 1/25 data, it is clear that increasing the training data enhances the outcomes. Conversely, even when using extremely limited data amounts like 1/5
17

---

Published as a conference paper at ICLR 2025
1/25 of the original dataset, still achieve capable model, and the performance does not dramatically decrease.
These results demonstrate that our method can still yield good results in low-resource environment. In other words, our approach does not rely on large volumes of data and the strong capability of large models; it is succinct and efficient, capable of performing well in extremely low-resource settings.
Backbone
VILA-3B
VILA-13B
LL 13B
1/5 Data
1/25 Data
LLaMA-70B
57.3
61.2
44.3
52.1
50.6
LLaMA-8B
35.7
34.3
26.0
31.4
29.3
Mistral-7B
24.5
26.0
19.5
22.6
21.7
Phi-3.8B
20.0
19.5
16.7
17.9
13.9
Table 8: Comparison of reward model selection and data demands on ScienceWorke seen set.
Backbone
VILA-3B
VILA-I3B
LLaVA-13B
1/5 Data
1/25 Data
LLaMA-70B
57.0
60.7
48.2
50.0
47.7
LLaMA-8B
28.1
27.5
22.2
26.8
24.2
Mistral-7B
21.1
22.9
19.2
21.6
19.7
Phi-3.8B
17.0
15.3
13.7
14.2
11.7
Table 9:
Comparison of reward model selection and data demands ScienceWorld unseen set.
Ablation on Visual Input. We also train new reward model without visual information. As shown Table 10, we can see that, different settings, the reward model with visual information performs better than the model without visual information, which shows the importance of visual context in the Webshop task.
Backbone
Algorithms
w/o visual
w/ visual
ARMAP-R
56.1
56.5
LLaMA-70B
ARMAP-B
61.6
62.0
ARMAP-R ARMAP-B
53.6
54.1
Mistral-7B
51.3
54.4
Table 10: Ablation of the visual input.
Overhead in Data Synthesis. We calculate the tokens have used for task instruction generation and trajectory exploration. We summarize these overheads in Table 11. To provide more intuitive comparison, we first calculated the average tokens per sample for these different tasks. We found that although Game of 24 overall consumes the most tokens, the average number of tokens spent per Game of sample is relatively the least. In contrast, Webshop has the fewest total samples but highest average number of tokens spent per sample. Science World falls in between these two. The reason Webshop has a higher average number of tokens compared to Game of 24 is that the environment required for Webshop is more complex, involving more diverse elements and possibilities.
Tasks
Samples
Tokens
Tokens per Sample
ScienceWorts World
4064
2541255
625
Webshop
2436
6645746
2728
Game of 24
37885
12846182
339
Table 11: Tokens of data generation in three different tasks.
Proprietary Models as Training Data Generators and Policy Models. In the main content, we mainly consider using open-source models act training data generators and policy models. In
18

---

Published as a conference paper at ICLR 2025
order to investigate the upper bounds of proposed method, we also conduct some experiments by using powerful proprietary models. However, to serve as the training data generator, closed- source models have several drawbacks, including high costs, limited commercial access, and lack of reproducibility. In contrast, our approach achieves strong results without relying on closed-source models. Given the expense associated with API-based models like GPT-4o for generating training datasets, we have opted not to pursue this method for now.
For API-based proprietary models serving as policy models, the high cost of GPT-4o and API access rate limitations prompted us to focus our experiments primarily on ALFWorld. Specifically, we used GPT-44-2222--8- to sample five trajectories each on ALFWorld's Dev and Std sets, then conducted experiments using our automatic reward model. As shown in Table 12, our reward model is able to help the powerful GPT-4o gain better performance, demonstrating the effectiveness our framework.
GPT-4o
Std
Dev
Sampling
0.74
0.88
Greedy
0,82
0.90
ARMAP-B
0.84
0.95
Table 12: Experiments of using the proprietary model on ALFWorld
A.3 IMPLEMENTATION DETAILS.
Large Pretrain Model Setup. We serve a diverse set of open-source LLM APIs to evaluate the effectiveness of the proposed pipeline. We list all the open-source models and their weights on huggingface in table 13. All these models can be easily setup and reproduced with the VLLM libarary (Kwon et al., 2023b). We prove the effectiveness of our ARMAP framework across different LLM APIs
Acronym
Model description and weight on huggingface websites
Llama70B
--00B--nstruct-A---NN
Llama8B Mistral7B
Instruct
Phi3.8B
VILA3B
Table 13: Agent models, the reward model, and their associated description on huggingface websites.
Environment Setup. We build our environments based on the environment setup the previous works (Liu et al., 2023b; Song et al., 2024; Yao et al., 2023b; Shridhar et al., 2021; Schmidgall et al., 2024). For Webshop and ALFWorld environment, we start these docker environments from AgentBench (Liu et al., 2023b) and implement different planning algorithms, Reflexion, Best-of-N and MCTS on it. Similarly, we build our ScienceWorld, Game of 24 and AgentClinic environments from Song et al. (2024), Yao et al. (2023b) and Schmidgall et al. (2024), respectively.
Planning Algorithm Details. We compare the performance of different planning algorithms by limiting their maximum explored trajectory number. We set the maximum number to be on Webshop and ScieneWorld in consideration of effectiveness and efficiency. We set the maximum number to be 100 on Game of 24 following the setup of Yao et al. (2023b). In Webshop, ScienceWorld, only consider the top 10 available actions suggested by the LLM agent at each state to reduce search space. We also set a trajectory's 's maximal action number length 10 for simplicity.
ALFWorld and AgentClinic benchmarks,
For Reflexion, we set the maximum trial number to be 10 for all tasks. For different tasks and models, set the threshold of Reflexion separately. During the iteration process, if the reward of the current
19

---

Published as a conference paper at ICLR 2025
trail's trajectory exceeds the threshold, the iteration will stop, and the current trail will be taken as the result. If the maximum number trials is reached, the last trial will be taken as the result in Webshop and Game of 24, while the first trial will be taken the result in ScienceWorld.
Data Generation. In total, we generate 2,436, 4,064 and 37,885 pairs of data for Webshop, ScienceWorld and Game of 24, respectively. Sampled synthesized data sample can be seen in Fig. 5, Fig. and Fig. 7. We provide the sampled prompt we use for data generation from Fig. 8 to Fig. 11. Fig. we show an example how we prompt the LLM to generate language instruction for ScrenceWorld. In Fig. 9, we show how we refine the language instruction to refine the instruction goal. In Fig. 10 and Fig. 11, we show the prompt how the L.M agent synthesizes positive and negative trajectories, respectively.
20

---

Published as a conference paper at ICLR 2025
Training Data Example for Webshop
Task Instruction: need long-lasting bluetooth radio speaker units for a soundbar with 4ohm impedance and 6w power, and price lower than 50.00 dollars.
Positive Example:
Thought:
Action:
Current Screenshot:
Observation:
Thought: The option B09STMXYRS matches the required characteristics. should select it.
Action: (09SMT0RSS
Current Screenshot:
Intracdas
bing lertigg
yester
JUSTQUJN 2pos 1.5 Inch Bluetooth Radio Speaker Unit Ohm GW Sound Bar Horn 18 Core 45mm Music Portable Loudspeaker Rubber Edge Compatible with Altavoces Bookshelf Speakers (Color 400mm ohm BW)
52836
Rating NA
Day Now
Observation: Instruction: [SEP] need 2 long-lasting bluetooth radio speaker units for soundbar with 4ohm impedance and 6w power, and price lower than 50.00 dollars [SEP] Back to Search [SEP] < Prev [SEP) JUSTQUUN 2pes 1.5 Inch Bluetooth Radio Speaker Unit 4 Ohm 6W Sound Bar Horn 18 Core 45mm Music Portable Loudspeaker Rubber Edge
40mm 4 ohm 6W) [SEP] Price: $28.36 [SEP] Rating: N.A. [SEP] Description [SEP] Features [SEP] Reviews [SEP) Buy
Compatible with Altavoces Bookshelf Speakers (Color :
Now
Thought: should click on the "Buy Now" button to proceed with the purchase.
Action: click[buy now]
Negative Example:
Thought:
Action:
Current Screenshot:
Observation:
Thought: think I should click on the product need, which is B09SWKXBYS, since it matches the requirements of 4ohm impedance and 6w power and the price lower than 50.00
dollars.
Action: click[ BBOSSKXBKYS
Current Screenshot:
msed lorplaning wooth spenee scwe and over Ean 0200 0oTa
with Impecaveent
JUSTQUJU 2pcs Full Range Inch Speaker 15W DIY Soundbarr Boombox Unit Portable Racio 20w Ohm Speaker HiFi Bluatooth Speakers 55mm Bookshelf Speakers (Color:
2.20
201)
$42
Rating N.
Day Rox
10x
15w
chm
Observation: Instruction: [SEP] need 2 long-lasting bluetooth radio speaker units for soundbar with 4ohm impedance and 6w power, and price lower than 50.00 dollars [SEP] Back to Search [SEP] < Prev [SEP] color [SEP] ohm 10w [SEP] 4 ohm 15w [SEP] 4 ohm 20w [SEP] JUSTQUJUN 2pcs Full Range Inch Speaker 15W DIY Soundbar Boombox
21

---

Published as a conference paper at ICLR 2025
Unit Portable Radio 10W 20w 4 Ohm Speaker HiFi Bluetooth Speakers 55mm Bookshelf Speakers (Color : 4 ohm 20W) [SEP] Price: $42.66 [SEP] Rating: N.A. [SEP] Description [SEP] Features [SEP] Reviews [SEP] Buy Now
Thought: I think I should click on the 'Buy Now" button to purchase the product.
Action: click[buy now]
Figure Training Data Example for Webshop.
Training Data Example for ScienceWorld
Task Instruction: You are helpful assistant to do some scientific experiments in an environment. In the environment, there are several rooms: kitchen, foundry, workshop, bathroom, outside, living room, bedroom, greenhouse, art studio, hallway You should explore the environment and find the items you need to complete the experiment. You can teleport to any room in one step. All containers the environment have already been opened, you can directly get items from the containers. The available actions are: open OBJ: open container close OBJ: close container
waitl: task no action for a step
Task Description: Your task find a non-living thing in the environment. Focus on the jug in the greenhouse. When you are done, move the jug to the green box in the kitchen.
Positive Example:
Action: look around
Observation: : This room is called the hallway. In it, you see:
the agent, substance called air, a picture
You also see:
door to the art studio (that is open), A door to the bedroom (that is open), A door to the greenhouse (that is open), A door the kitchen (that is open), A door the living room (that is open), A door to the workshop (that is open)
Action: teleport the greenhouse
Observation: You teleport to the greenhouse. Action: look around
Observation: This room is called the greenhouse. In it, you see:
the agent, substance called air, adult bee, a adult bee, a adult bee, a adult bee, a bee hive. The bee hive door is open. In the bee hive is: adult bee. flower pot (containing peach tree in the adult stage, soil), a flower pot 4 (containing a substance called water), a flower pot 5 (containing substance called water), flower pot 6 (containing banana tree in the adult stage, soil), a flower pot 7 (containing a apple tree in the adult stage, soil), a jug (containing nothing), shovel, sink, which is turned off. In the sink is: nothing. You also see:
door the hallway (that is open), A door to the outside (that open)
Action: focus on the jug
Observation: You focus on the jug.
Negative Example:
Action: look around
Observation: This room called the hallway. In it, you see:
the agent, substance called air, picture
You also see:
A door to the art studio (that is open), A door the bedroom (that is open), A door to the greenhouse (that is open), A door to the kitchen (that is open), A door to the living room (that is open), A door to the workshop (that is open)
Action: teleport to the greenhouse Observation: You teleport the greenhouse.
22

---

Published as a conference paper at ICLR 2025
Action:
connect painting to kitchen
Observation: terminal on painting is now connected to terminal on kitchen Action: connect agent bathtub
Observation: No known action matches that input.
Figure 6: Training Data Example for ScienceWorkd.
Training Data Example for Game of 24
Task Instruction: You are start agent and generate data for Game24. Game24 requires users to use numbers and basic arithmetic operations (+ /) to obtain 24.
You task generate a new input (4 digital number) for Game 24.
1. each new input number should he in the range of 1 to 13.
People can use numbers and basic arithmetic operations
/) to obtain 24. At each step,
people are only allowed to choose two of the remaining numbers to obtain new number.
Here are the few-shot examples.
3. since there is only four number input and the intermediate steps should only be three.
Input: 12 10 8
Positive Example: Steps: 10
(left: 2 12)
12/2 = 6 (left: 4 6)
6 * 4 24 (left: 24)
Answer: (12 (10 8)) 4 24 24
Negative Example:
Steps: = -2 (left: 8) 8/4 2 (left: 22
2 (left: 4)
Answer: ((10- 12) * (8/4)) 4
Figure 7: Training Data Example for Game of 24.
Instruction Generation Prompt for WcildceWorld
Task Instruction: You are helpful assistant to do some scientific experiments an environment.
In the environment, there several rooms: kitchen, foundry, workshop, bathroom, outside, living room, bedroom, greenhouse, art studio, and hallway.
The available actions are:
open OBJ: open container
...
You will be given dialogue between you (assistant) and human user. You need to generate the task description after understanding the dialogue given to you. In order to help you better generate the task description, will give you an example below.
In this example, human an assistant and talk with user to finish the task step by step using the available actions above. Thought and Action are your reasoning and the actions you take each step. Observation is provided by the human user; is an observation of the surrounding environment after you have executed the action. In the end, you need restore the Task Description based on the
gives Task Description
first. Then you work as
23

---

Published as a conference paper at ICLR 2025
whole dialogue.
Example:
User
Task Description
Assistant
Thought: need to find materials to create green paint, which are likely to found in an art studio.
Action: look around
User
Observation: This room is called the bedroom, In it, you see:
User
Observation: Blue and yellow paint mix produce green paint.
Assistant
Thought:
The task is to create green paint and then focus on it. Now that the green paint is made, should concentrate on the final product.
Action: focus green paint
Task Description: Your task is to use chemistry to create green paint. When you are done, focus on the green paint.
Here the task you need to do:
You should generate the Task Description following the format like examples below and generate it based on the dialogue above without any other words.
Examples:
Your task is to boil ice cream. For compounds without a boiling point, combusting the substance is also acceptable. First, focus the substance. Then, take actions that will cause it to change its state of matter.
Your task to use chemistry to create violet paint. When you are done, focus on the violet paint.
Your task is to find a(n) living thing. First, focus on the thing. Then, move it to the red box in the bathroom.
Task Description:
Figure 8: Instruction Generation Prompt for Science
Instruction Refinement Prompt for ScienceWorld
Task Instruction: You are helpful assistant to do some scientific experiments in an environment.
In the environment, there are several rooms: kitchen, foundry, workshop, bathroom, outside, living room, bedroom, greenhouse, art studio, and hallway. The available actions are: open OBJ: open a container
You will be given task description and corresponding trajectory. The task de- scription concludes what you have done in this trajectory. You need to elaborate this description based this environment by adding more details.
24

---

Published as a conference paper at ICLR 2025
Example:
Task Description: Your task is to grow an apple. You can find seeds in the kitchen, You should focus the grown apple.
Corresponding Trajectory:
look around
This room is called the hallway. In it, you see:
open door to kitchen
The door is already open.
go to kitchen You move to the kitchen.
Refined Task Description: Your task is to grow apple. This will require grow- ing several plants, and them being crosspollinated to produce fruit. Seeds can be found in the kitchen. To complete the task, focus the grown apple.
Here the task description you need refine, and the corresponding trajectory is also provided:
Refined Task Description:
Figure Instruction Refinement Prompt for Science World.
Positive Trajectory Synthesis Prompt for ScienceWorld
Task Instruction: You are helpful assistant to do some scientific experiments in an environment.
In the environment, there are several rooms: kitchen, foundry, workshop, bathroom, outside, living room, bedroom, greenhouse, art studio, and hallway.
The available actions are: open OBJ: open a container
Based this environment, you need to randomly propose Task Description, which concludes what you have done in this environment.
Here are some examples: Your task to use chemistry to create green paint. When you done, focus on the green paint.
Your task determine whether tall plant height dominant or recessive trait in the pea plant. If the trait is dominant, focus the red box. If the trait is recessive, focus on the green box.
Once you obtain the Task Description, you need to navigate through the environment to complete the instruction and generate trajectory.
Example:
Task Description: Your task is to use chemistry to create green paint. When you are done, focus the green paint.
25

---

Published as a conference paper at ICLR 2025
Trajectory:
Thought: need to find materials to create green paint, which are likely to be found in art studio.
Action:
look around
Generated Trajectory:
Figure 10: Positive Trajectories Synthesis Prompt for Science World.
Negative Trajectory Synthesis Prompt for ScienceWorld
Task Instruction: You are helpful assistant to do some scientific experiments an environment,
In the environment, there several rooms: kitchen, foundry, workshop, bathroom, outside, living room, bedroom, greenhouse, art studio, and hallway.
The available actions are:
open OBJ: open container
You will be given task description and a corresponding trajectory. Based on them, you need to generate a negative sample that is similar to the correct trajectory but different from it. The generated trajectory should not meet all requirements of the task description. Moreover, the generated trajectory should satisfy all requirements of the environment.
Example:
Task Description: Your task to focus on the life stages of the apple plant, starting from earliest to latest. The plants are located outside.
Positive Trajectory:
look around
This room called the hallway. In it, you see:
open door outside
The door is already open
Negative Trajectory:
look around
This room called the hallway. In it, you see:
open door to kitchen
The door is already open.
go to kitchen
You move to the kitchen.
Here the task you need to do:
Negative Trajectory:
26

---

Published as a conference paper at ICLR 2025
Figure 11: Negative Trajectories Synthesis Prompt for Science World.
Reward Model Training Details. The detailed hyperparameters we use for reward model during
training and inference are shown in Table 14. We employ identical hyperparameters for reward models of different environments. For Webshop, we use checkpoint of 100 steps in ARMAP-B, and checkpoint of 1200 steps in ARMAP-R and ARMAP-M.
Name
Science
Webshop
Game of 24
lora
64
lora alpha
16
lora dropout
0.0
lora target modules
q_rrj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
epochs
10
3
10
batch size batch size per device
8
4
1
1
gradient accumulation steps
16
16
learning rate warmup ratio checkpoint steps temperature
le-5
2e-5
le-5
0.2
0.1
0.25
160
1100, 1200
1500
0.0
0.0
0.0
Table 14: Detailed hyperparameters used in reward model.
Implementation Details of Ablation baselines. For SFT, use all positive examples from the reward model training as the training data. The training objective is to enable the model to predict the output of the LLM in the positive examples.
For using few-shot prompting to guide the LLMs to predict the reward of historical trajectories, we the following format of the few-shot prompt:
Few-shot Prompt for LLMs Directly Serving ScienceWorld Reward Model
Task Instruction: You are an autonomous intelligent agent tasked with evaluating the trajectories of the past experience. You will be given the history of past experience in which you were placed in environment and given task to complete. These tasks will be accomplished through the use of specific actions. Now you are trying to evaluate the performance
past task. You will be given the objective of the task, the history of interaction including the observations you had and the actions you issued, and the status of the task. Your goal is think about the strategy and provided path to pro- duce a score ranging from 0 to to measure whether the objective of the task has been reached.
Here are 2 examples:
Examplel:
Human: You are helpful assistant to do some scientific experiment in an environ- ment. In the environment, there are several rooms: kitchen, foundry, workshop, bathroom, outside, living room, bedroom, greenhouse, art studio, hallway. You should explore the environment and find the items you need to complete the experiment. You can teleport to any room in one step. All containers in the environment have already been opened, you can directly get items from the containers. The available actions are: open OBJ: open a container, close OBJ: close container, activate OBJ: activate device, deactivate OBJ: deactivate device, connect OBJ to OBJ: connect electrical components, disconnect OBJ: disconnect electrical components, OBJ [on OBJ]: use a device/item, look around: describe the
27

---

Published as a conference paper at ICLR 2025
current room, examine OBJ: describe object in detail, look OBJ: describe container's contents, read OBJ: read a note or book, move OBJ to OBJ: move an object to a container, pick up OBJ: move an object to the inventory, pour OBJ into OBJ: pour a liquid into container, mix OBJ: chemically mix a container, teleport to LOC: teleport to specific room, focus on OBJ: signal intent on a task object, wait: task no action for 10 steps, waitl: task no action for step
Task Description: Your task is to find a(n) non-living thing. First, focus on the thing. Then, move it to the purple box in the living room. Please only output one Thought and one Action. The available actions are: open OBJ: open a container, close OBJ: close a container, activate OBJ: activate device, deactivate OBJ: deactivate device, connect OBJ to OBJ: connect electrical components, disconnect OBJ: disconnect electrical components, use OBJ [on OBJ]: use device/item, look around: describe the current room, examine OBJ: describe an object in detail, look at OBJ: describe a container's 's contents, read OBJ: read a note or book, move OBJ OBJ: move an object to a container, pick up OBJ: move an object to the inventory, pour OBJ into OBJ: pour a liquid into a container, mix OBJ: chemically mix a container, teleport to LOC: teleport to specific room, focus on OBJ: signal intent on a task object, wait: task no action for 10 steps, waitl: task no action for step. Please strictly follow the format of the actions.
Assistant: Thought: I need to find non-living thing, which could be an object in any room, so I should start by looking around the current room. Action: look around
Observation: This room is called the bathroom. In it, you see: the agent, substance called air, a bathtub, which is turned off. In the bathtub is: nothing. a drawing, a glass cup (containing nothing), sink, which is turned off. In the sink is: nothing, toilet. In the toilet is: A drain, which is open, a substance called water. You also see: A door to the kitchen (that is open). Please only output one Thought and one Action. The available actions are: open OBJ: open a container, close OBJ: close a container, activate OBJ: activate a device, deactivate OBJ: deactivate a device, connect OBJ to OBJ: connect electrical components, disconnect OBJ: disconnect electrical components, use OBJ [on OBJ]: use a device/item, look around: describe the current room, examine OBJ: describe an object in detail, look at OBJ: describe a container's 's contents, read OBJ: read a note or book, move OBJ to OBJ: move an object to a container, pick up OBJ: move an object to the inventory, pour OBJ into OBJ: pour liquid into a container, mix OBJ: chemically mix a container, teleport to LOC: teleport to a specific room, focus on OBJ: signal intent on a task object, wait: task no action for 10 steps, waitl: task no action for step. Please strictly follow the format of the actions. The glass cup in the bathroom is a non- non-living thing, so should focus on it. Action: focus on glass cup.
Human:
Assistant: Thought:
Assistant: Thought: I've found the purple box, and now need to move the glass cup into it.
Action: move glass cup to purple box.
Human: Observation: No known action matches that input.
Human: Please evaluate whether you complete the 'Task Description' or not.
Following your definitions, the task completion score is <0.750>
Example2: ...
Now is your turn:
Figure 12: Few-shot Prompt for LLMs Directly Serving as Science World Reward Model.
A.4 MORE QUALITATIVE RESULTS.
We show more qualitative results of our ARMAP Fig. 13 and Fig. 14. Based on the examples, we can see that the automatic reward models in our pipeline can distinguish good trajectories from bad ones and guide LLMs to generate better trajectories to finish the tasks.
28

---

Published as a conference paper at ICLR 2025
Qualitative Visualization for ScienceWorld
Task Description: Task Description: Your task is to find a(n) non non- living thing. First, focus on the thing. Then, move it to the orange box in the living room.
Trajectory A:
Observation: You teleport to the living room.
Thought: Now that I'm in the living room, should look around to find a non-living thing that I can move to the orange box. Action: look around
Observation: This room called the living room. In it, you see:
the agent, substance called air, chair. On the chair is: nothing. couch. On the couch is: white pillow. finger painting a orange box (containing nothing) a table. On the table is: nothing.
You also see: door to the hallway (that is open).
Thought: see several objects in the living room, but need to find non-living thing that I can move to the orange box. Action: examine white pillow
ARMAP Aborted
Trajectory B:
Observation: This room is called the workshop. In it, you see:
the agent, a substance called air, a table. On the table is: a battery, a blue wire, a orange light bulb, which is off, red light bulb, which is off, red wire, switch, which is off, yellow light bulb, which off, a yellow wire, a ultra low temperature freezer. The ultra low temperature freezer door is open. In the ultra low temperature freezer is: nothing. You also see:
door the hallway (that is open)
Thought: see several objects in the workshop that could be considered non-living things, light bulbs. should focus on one of these objects to complete the task. Action: focus on battery
such as the battery, blue wire,
Observation: You focus on the battery. Thought: Now that I've focused on the battery, need to move it to the orange box in the living room. Action: teleport living room
Observation: You teleport to the living room. Thought: Now that I'm the living room, should look around to find the orange box and move the battery into it. Action: look around
Observation: This room called the living room. In it, you see:
the agent, substance called air, chair. On the chair is: nothing. couch. On the couch is: white pillow. finger painting a orange box (containing nothing) a table. On the table is: nothing.
You also see: A door to the hallway (that is open)
Thought: I've found the orange box in the living room, and can move the battery into it. Action: move battery to orange box ARMAP Selected
Figure 13: Qualitative Visualization for WorlnceWorld. Based on the comparison of the two trajectories, is evident that the final action of trajectory B brings the resulting state closer the task objective. Therefore, our model identifies trajectory B as the superior option.
29

---

Published as a conference paper at ICLR 2025
Qualitative Visualization for Game of 24
Input A: 2.5
Input B: 2.6.
Trajectory A1:
Trajectory B1:
Steps: * 2 (left: 5 11 16)
Steps: = (left: 2
11 16 (left: 16 16)
6+6 12 (left: 2 12)
16+ 32 (not 24, not valid)
2 12 = 24 (left: 24)
11-5= 6 (left: 2 8)
Answer: 2 (6 (9-3)) = 24
8/2=4 (left: 6)
ARMAP Selected
4 (left: 24)
Answer: (8/2) (11 5) 24
ARMAP Selected
Trajectory A2:
Trajectory B2:
Steps:
Steps:
8-5=3 (left: 2 11)
9 6 (left: 2 6)
11-3=8 8 (left: 2 2)
2-2 = 4 (left: 4 6)
2 16 (left: 16)
* 4= (left: 24)
16+ 24 (left: 24)
Answer: (6 (9 3)) = 24
Answer: ((88 5) 8) 2) 8
ARMAP Aborted
ARMAP Aborted
Figure 14: Qualitative Visualization for Game 24. Trajectory A and Trajectory correspond to input A and input B respectively. Results show that our ARMAP can accurately pick out the correct trajectory.
A.5 FAILURE CASE ANALYSIS
In this section, we investigate the common failure cases of our framework, aiming to provide data points and insights for future research.
The most common error occurs when there are multiple restrictions in the instruction, the reward model overlooks some of these key conditions. A representative example is illustrated in Fig. 15, the model focuses on price and size but ignores the details about 'Fluoride" hidden in the product description.
Another common failure mode occurs when commonsense knowledge is involved. As demonstrated Fig. 16, the agent was tasked with buying blackout shade but failed choose both the color and the size. While, in everyday life, size is generally more important, the reward model prioritized color instead. In Fig. 17, the reward model cannot assess the lifespan of dragonflies and chipmunks because it lacks the necessary biological knowledge.
The analysis of failure modes highlights the significant potential of our framework. To improve its performance, we propose two possible strategies for improvements in reward modeling: (a) Constructing Data with Focus on Complex and Detailed Conditions: enhancing the dataset to include scenarios with higher complexity and better handle intricate situations and edge cases. (b) Intervening Reward Scoring with External Knowledge: incorporating external knowledge by combining prompted Large Language Model with the trained reward model. This approach allows the LLM's generalized knowledge to calibrate the reward scores in controllable manner, improving the overall accuracy and robustness of the reward model.
Discussion.
nuanced conditions will help the framework
30

---

Published as a conference paper at ICLR 2025
Instruction: need fluoride free tosth for fresh breath. will need pack ounce size, and price lower than 50.00 dollars.
Action1: searchfl free toothpaste 3.5 oz pack of price
Action1: search[toothpaste fresh breath fluoride free pack 3.5 oz
50.00]
B0SOR45OL
50.00]
Action2: click(b0712164xx
$12.00 59
Action2: Click(BB
$31.27
Spriniene Natural Fluoride Free Toothpaste for Cleen and Strong Teeth
Sprinlene Natural Toothpaste for Cavity Protection Teeth
sunce tpackor
0. ousnc (pock
Action3: click[pack 4]
$12.00
Action3: click[pack 4]
Price: $23 27
Blez:
Action4: click(Buy Now]
bibur 41
Action4: click(Buy Now]
(peck of
GT Reward 1.0
GT Reward 0.75
ARMAP Aborted
0o
ARMAP Selected
Figure 15: Failure Example from Webshop. The reward model ignores certain key conditions in the task instruction.
Instruction:
want find white blackout shades that 66 inches width and 66 inches height. they need to be easy to install, and price lower than 90.00 dollars
Action1: search[white blackout
Action1: search(white blackout
shades 66 66 easy install price
shades 66x66 easy install price
90.00]
BounorcWWP
90.00]
BO6GFTCHOH
BOSRG-44R49
Action2: cIIcX(B997XXCWP7)
Pric:
$7a
Action2: [
$51.99
Prixe
fime
Oom
X Missing color!
Color:
X Missing size!
oftcck e White
Action3: click[66"w
Bow
Price:
$70.40
Action3: click(white]
509 de
Action4: click[Buy Now]
Action4: click[Buyckcckcco Now]
oftick wwnne
GT Reward = 0.8
GT Reward 0.4
ARMAP Aborted
O0
ARMAP Selected
Figure 16: Failure Example from Webshop. The reward model misjudged the importance of different conditions, such the size and color in this case.
31

---

Published as a conference paper at ICLR 2025
Instruction: Your task is to find the animal with the longest life span, then the shortest life span. First, focus on the animal with the longest life span. Then, focus on the animal with the shortest life span. The animals are in the 'outside' location.
Action1: look around
Action1:: look around
Observation1: This room is called the art studio. In it, you the agent, substance called large cupboard.
Observationt: This room is called the art studio. In it, you the agent, substance called large cupboard.
Action2: teleport to outside
Action2: teleport to outside
Observation2: You teleport to the outside.
Observation2: You teleport to the outside.
Action3: look around
Action3: look around
Observation3: Here you
axc, baby chipmunk,
Observation3: Here you see: axe, baby chipmunk.
a baby dragonfly... ... a parrot egg
a baby dragonfly a parrot egg
Action4:
focus parrot (Thought: parrot of bird,
Action4:
focus parrot (Toought
type of bird,
birds ame known to live for fermany
The life span of parro can be quite
The life parrot
long, often exceeding 30 years in some species.)
long, often exceeding 30 years in some spocies.)
Observation4: You focus on the parrot egg.
S
Observation4:
You focus on the parrot oggd
Action5: focus on baby dragonfly (houghh: Oragonflees are
ActionS:
focus on baby chipmunk (Thoughe: need to focus
known to have a relatively short life span, typically ranging from few weeks
the animal with the shortest life span, which is the baby chipmunk.)
few montrs.)
Observation5: focus on the baby chipmunk.
ObservationS: You focus the baby dragonfly.
GT Reward = 1.0
GT Reward 0.83
OO
ARMAP Aborted
00
ARMAP Selected
Figure 17:
Failure Example from ScienceWorld. Reward models have problems when common- sense knowledge is involved, such as the inability to determine the shortest life span.
32


Reviews:
[{"review_id": "3TSzQ9vK8f", "reviewer": "Reviewer_G3nc", "summary": "The paper proposes ARMAP, a novel framework that enhances the task-solving abilities of large language model (LLM)-based agents in interactive, multi-step environments. The authors tackle key challenges associated with data scarcity and API restrictions, presenting a method that automates reward model learning from LLM agents’ interactions within an environment, thus eliminating the need for human annotations or commercial LLM-based evaluation. The reward model can then guide planning algorithms (e.g., Monte Carlo Tree Search and Reflexion) to improve LLM agents’ performance in tasks requiring iterative decision-making, such as e-commerce navigation and simple scientific experiments.", "questions": "Some suggestions for improvement:\n\nWhy do we need pairwise comparisons - this works in foundation model post-training, but why not use success/failure reward model training and using that as areward or value function?\n\nCan you extend the experimental scope to include more diverse or high-stakes decision-making environments, such as ALFRED, BEHAVIOUR or HABITAT to illustrate ARMAP’s performance on tasks requiring more advanced capability.\n\nComputational Efficiency Analysis: Including an analysis of the framework's data demands and comparisons with reward learning approaches would be beneficial, especially if extending the applicability of ARMAP to realistic low-resource settings.\n\nDetailed Error Analysis: A more granular analysis of failure cases in each environment, particularly for tasks that involve complex dependencies or decision making, would provide deeper insights into the limitations of the current approach and inform possible improvements in reward modeling.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Innovative Reward Modeling Approach: The ARMAP framework leverages LLMs to generate diverse action trajectories, then synthesizes task goals and feedback to train a reward model. This automation of reward modeling is a strong innovation, addressing critical limitations in agent-based tasks by reducing reliance on costly and often proprietary data.\n\nFramework Flexibility: The framework’s compatibility with multiple planning algorithms (MCTS, Reflexion, Best-of-N) demonstrates flexibility and potential for broader application. The performance boost across different LLMs (Llama, Phi, and Mistral) also underscores the generalizability of the ARMAP model.\n\nEffectiveness in Customization: ARMAP’s ability to modify reward targets for controllable behavior generation (e.g., minimizing action length or cost) is a valuable capability for task-specific tuning, as demonstrated in the Webshop experiments.", "weaknesses": "Limited Scope of Tested Environments: Although the ARMAP framework was evaluated in multiple environments, these remain relatively constrained in task diversity (e.g., online shopping, elementary science tasks). Further exploration into environments with more complex multi-modal interactions or requiring intricate goal alignment would provide stronger evidence of the framework’s versatility.\n\nPotential Overhead in Data Synthesis: While the automated reward modeling is valuable, the reliance on in-context LLMs for both task generation and trajectory synthesis could introduce computational overhead. It would be useful to discuss the cost-benefit analysis of this approach, particularly in environments requiring higher levels of interaction fidelity.\n\nDependence on LLM Quality: ARMAP’s effectiveness is inherently tied to the quality of the LLMs generating the synthetic data. While the framework was evaluated on open-source models, a more explicit discussion of performance across varying LLM qualities or limitations when using smaller LLMs would provide more insight into its applicability in resource-constrained scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0dSDX7eW6i", "reviewer": "Reviewer_XSp2", "summary": "The paper proposes a framework named ARMAP, aimed at enhancing the task-solving capabilities of LLM-based agents in challenging environments that necessitate multi-step decision-making. While traditional LLMs perform well in text-based tasks, they face challenges with interactive, goal-oriented tasks due to limited access to large-scale decision-making data. ARMAP tackles these issues by developing an automated reward model that assesses action trajectories without requiring human annotations.\n\nThe framework comprises three main components:\n1. Data Generation: An LLM agent interacts with the environment, producing diverse action trajectories that include both successful and unsuccessful task completion attempts. These trajectories, encompassing task intents, positive outcomes, and negative outcomes, are utilized to train the reward model.\n2. Reward Model: A specialized model evaluates the effectiveness of each trajectory in fulfilling a task, thereby guiding the LLM agents in their planning.\n3. Planning Algorithms: By integrating the reward model with planning methods like Monte Carlo Tree Search (MCTS) and Reflexion, the agent can optimize its actions to follow high-reward paths.\n\nExperiments depict ARMAP’s efficacy across various benchmarks, demonstrating improved planning performance for different LLM agents. The approach offers advantages in flexibility and practicality, as it reduces reliance on human labels and expensive, closed LLMs, thereby facilitating the development of more autonomous and efficient AI agents capable of managing real-world tasks.", "questions": "Synthetic Data Quality: How do you ensure the quality and diversity of the synthetic trajectories generated by LLMs? Have you observed any limitations when these synthetic trajectories don’t align closely with real-world decision-making patterns?\n\nComputational Cost in Real-Time Applications: Given the computational demands of planning algorithms like MCTS, how would ARMAP perform in applications requiring real-time decision-making? Are there strategies for reducing overhead while retaining performance?\n\nReward Model Generalization: How well does the reward model generalize to tasks and environments different from those it was trained on? Have you tested ARMAP in domains requiring more complex, domain-specific knowledge, such as legal or medical contexts?\n\nScalability and Practical Deployment: What are the main challenges you foresee in scaling ARMAP for broader deployment in real-world applications? Are there specific areas (e.g., hardware requirements, integration with other models) that need further development?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Automated Reward Modeling: It presents an innovative method for autonomously learning reward models without the need for human-annotated data, addressing issues related to data scarcity and dependence on costly closed-source LLMs. This makes the framework scalable and practical for real-world applications.\n\nEnhanced Decision-Making for LLM Agents: By offering a reward-based evaluation system, ARMAP significantly boosts the ability of LLM agents to perform complex, multi-step tasks that require sequential planning, an area where standard LLMs often struggle.\n\nEfficiency and Cost-Effectiveness: By eliminating the need to fine-tune LLMs and avoiding reliance on proprietary LLM APIs, ARMAP provides a cost-effective solution that could make high-performing AI agents more accessible for widespread use.", "weaknesses": "Limited Applicability in Highly Dynamic Environments: While the framework performs well in simulated environments with fixed rules, such as online shopping simulations and controlled benchmarks, its effectiveness in rapidly changing, unpredictable real-world environments is uncertain. The model may struggle with scenarios that require quick adaptation to new patterns not present in the training data.\n\nComputational Overhead with Complex Planning: The integration of planning algorithms like MCTS, while effective, can introduce significant computational costs, especially when exploring multiple trajectories. This may limit ARMAP’s efficiency in resource-constrained settings or for tasks requiring real-time responses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z6mNhyzgu5", "reviewer": "Reviewer_Rvsi", "summary": "ARMAP presents a novel framework for autonomous agents by leveraging reward modeling and planning. It trains a reward model on contrastive trajectories, enabling effective decision-making in complex environments through LLM-as-agents. Unlike input-optimized prompting-based approaches, ARMAP scores steps within task trajectories, focusing on task completion. The ablation study supports the framework’s effectiveness and adaptability.", "questions": "Although the automatic reward model training is a good idea, there are few concerns after going through the paper and demand clarity of choice:\n1. Writing and Formatting:\n    * In Figure 1, the title \"Tree Planning\" should use lowercase \"(c)\" instead of capital \"(C).\"\n2. Reward Model Specifics:\n    * Could authors clarify the size of the reward model used in this study?\n    * In Line 100, authors mention challenges in developing a reward model (RM). Could they provide a few specific examples of these challenges for clarity?\n    * What neural architecture was selected for the reward model in this framework? Is this inspired from any previous works?\n3. Dataset Selection:\n    * Some established decision-making agent datasets, such as AlfWorld, BabyAGI, or PDDL, are not included. These embodied agent datasets offer complex, long trajectories that could be valuable to the study. Could authors comment on their absence or suitability?\n4. Multimodal Feedback:\n    * Line 150 refers to multimodal feedback. Could you specify which modalities other than text were used in predicting the next action?\n5. Reward Model Type:\n    * In Line 161, you state a focus on developing the reward model. Is this a classification model with a defined set of output classes, or is it a regression model?\n6. Observation Clarification:\n    * In Line 225, the phrase “...corresponding environment observations...” could benefit from refinement, as there’s typically one extra observation at the start. Could this section be adjusted to clarify the distinction?\n7. Trajectory Generation and Instruction Use:\n    * In Figure 2, authors mention using “initial language instructions in the environment” to generate trajectories, but it’s unclear if any LLM was employed to identify keywords. For instance, in “I am looking for jeans with 40w x 34l size, and price lower than 200 dollars,” did the framework use LLM predictions to determine \"Jeans\" as the keyword for search?\n8. Impact of Visual Inputs:\n    * What role do visual inputs play in the reward model’s training? Have authors conducted any ablation studies that use only text from trajectories to measure their impact? It would be helpful to know if the visual inputs significantly influence the final model performance. I find this missing.\n\nThese points would enhance the clarity and depth of the paper, particularly around architectural choices and empirical coverage. I am looking forward to the rebuttal during the discussion phase.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Originality: The automatic reward model and data generation approach presented is novel, allowing the framework to guide task completion within complex decision-making environments effectively.\n\nQuality: ARMAP stands out by using a reward model to evaluate and guide navigation steps in agentic environments, enhancing decision-making processes and setting a solid foundation for handling intricate tasks autonomously.\n\nClarity: The paper is well-written, with a clear flow that effectively communicates the core concepts and approach. While a few notational details could be clarified, the overall presentation is strong and accessible.\n\nSignificance: The framework's value is demonstrated through LLM-agent task performance, highlighting flexibility in controllable task generation and practical application via a reward model, which reduces reliance on large LLMs or human labeling.", "weaknesses": "Specificity in Reward Model Design: The paper lacks detailed information on the size and neural architecture of the reward model. Additionally, challenges in reward model development are not clearly defined. More depth and specific examples are needed to clarify these choices and support the framework's claims.\n\nLimited Dataset Scope: The study could benefit from evaluating on a broader set of complex, long-trajectory decision-making agent datasets. Including established datasets such as AlfWorld or BabyAGI, which could strengthen the empirical evaluation and demonstrate robustness across diverse environments.\n\nInsufficient Detail on Multimodal and Visual Input Integration: While the paper mentions multimodal feedback and visual inputs, it lacks clarity of their impact on reward model training. An ablation study that isolates the effect of visual inputs compared to text-based inputs could better illustrate their importance and further validate the framework’s design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)