Discriminator-Guided Embodied Planning for LLM Agent
Authors: Haofu Qian
Chenjia Bai
Jiatao Zhang
Fei Wu
Wei Song
Xuelong Li
Venue: ICLR 2025 Poster (2025)
Keywords: LLM Agent
Embodied Planning
Discriminator
Critic-Regularized Optimization
Primary area: reinforcement learning
TL;DR: A novel framework that generalizes demonstrations to establish critic-regularized grounding and optimization in the long-term planning of LLMs

Abstract: Large Language Models (LLMs) have showcased remarkable reasoning capabilities in various domains, yet face challenges in complex embodied tasks due to the need for a coherent long-term policy and context-sensitive environmental understanding. Previous work performed LLM refinement relying on outcome-supervised feedback, which can be costly and ineffective. In this work, we introduce a novel framework, Discriminator-Guided Action Optimization (DGAP), for facilitating the optimization of LLM action plans via step-wise signals. Specifically, we employ a limited set of demonstrations to enable the discriminator to learn a score function, which assesses the alignment between LLM-generated actions and the underlying optimal ones at every step. Based on the discriminator, LLMs are prompted to generate actions that maximize the score, utilizing historical action-score pair trajectories as guidance. Under mild conditions, DGAP resembles critic-regularized optimization and has been demonstrated to achieve a stronger policy than the LLM planner. In experiments across different LLMs (GPT-4, Llama3-70B) in ScienceWorld and VirtualHome, our method achieves superior performance and better efficiency than previous methods.

Published as a conference paper at ICLR 2025
DISCRIMINATOR-GUDDE EMBODIED PLANNING FOR LLM AGENT
Haofu Qian', Chenjia B=i Jiatao Zhang Fei Wu', Wei Song 1.3 Xuelong Li? 'Zhejiang University Institute of Artificial Intelligence (TeleAI), China Telecom, 3 "Yuhang Humanoid Robot Industry Innovation Center, Hangzhou, China,
*Shenzhen Research Institute of Northwestem Polytechnical University qianhfêz www.iuuuuuuuueuoooooooo..oo edu. en, faic www.echiintelcc.oooooo..oo cn
ABSTRACT
Large Language Models (LLMs) have showcased remarkable reasoning capabil- ities in various domains, yet face challenges in complex embodied tasks due coherent long-term policy, context-sensitive environmental understanding. Pre- vious work performed L.MM refinement relying on nutcome-supervised feedhack, which can be costly and ineffective. In this work, we introduce a novel framework, Discriminator- Guided Action OPtimization (DGAP) for facilitating optimization employ limited set
of LLM action plans via step-wise signals. Specifically, of demonstrations to enable the discriminator in learning a score function, which assesses the alignment between LLM--genrattr action and the underlying opti- mal one at every step. Based on the discriminator, LLM is prompted to generate actions maximize the score utilizing historical action-score pairs trajectory guidance. Under mild conditions, DGAP resembles the critic-regularized opti- mization and is demonstrated achieve stronger policy than the LLM planner. In experiments across different LLMs (GPT-4, Llama3-70B) in Science World and VirtualHome, our method obtains superior performance and better efficiency than previous methods.
INTRODUCTION
The effectiveness of large language models (LLMs) task planning hinges on their ability to gen- erate coherent, executable plans in dynamic, open-ended environments (Song al. 2023 (Suzuun (ct al. 2022). However, embodied scenarios pose greater challenges, requiring long-term planning intricate contexts with high-dimensional state spaces and tangible interactions. Specifically, involves coordinating long action sequences while managing intricate, dynamic environments, ini- tial inaccuracies compounding over steps, causing significant deviations from the plan and risking
mission failure (Ross et al. 2011 Luo et al. 2024).
Facing these difficulties, in-context learning methods (Dong et al. 2022 Abernethy al. 2023
Akyürek et al.] 2023). tree-of-thoughh((o1) methods Yao et al. 2023a Feng et al. 2023 Zhou al. (023a) and demonstration-baase methods (Lin al. 2023 Rita et al. 2024) have accom- plished partial progress. However, a key issue is that they generally receive feedback signals at the trajectory level, which non-proactive and limits their effectiveness and generalization in dy-
namic embodied scenarios (Chen et al. 2024b] Liu et al.) [2244] Shi et al. (024). In particular, in-context learning methods introduce closed-loop feedback(Song et al. 2003 Wu al.) (202)) for
failed results at completion via inner monologue (Madaan et al.) 2023 Huang et al.) 2022a Yao
al., 2023b) or physical feedback (Shinn et al. 2023] Mandi et al. 2023). ToT methods Yao
al. 2023a Feng et al., [223 Zhou al.] 2023a) generate multiple trajectories to represent several reasoning pathways, and perform trajectory-Ievel switching optimization, which incurs high explo-
ration costs (Zhang al. 2024a). Demonstration-be based approaches require extensive, high-quality
trajectories in diverse scenarios to obtain a generalized policy (Lin et al.) 2023 Rita et al.] 2024
(see Fig. [
To address these limitations, we consider an alternative way for grounding and generalization of embodied planning by leveraging the knowledge from limited demonstrations construct step- level guidance, which is subsequently integrated into in-context learning to boost planning. A key
"Corresponding Author

---

Published as a conference paper at ICLR 2025
Input
Inpot
Input
0.9
Guidance
Imitation Learning
Score Solution
Reflection
output
output
Input
output
output
In-context Learning
Tree of Thought
Demonstration-base
DGAP
Expert
Underlying expert
Generation with Optimization
Figure 1: Comparing methods of LLMs to conduct embodied planning. DGAP leverages step-wise alignment signals to guide LLMs in the planning process.
challenge applying the limited information from few-shot demonstrations in diverse scenarios. As described before, embodied agents that directly imitate these demonstrations often accumulate errors due to their restricted scope (Ross et al.) 2011 Luo et al. 2024). In contrast, humans possess dis- tinct capabilities for internal generalization and alignment (Barsalou 2008 Rizzolatti & Craighero 2004). allowing them to comprehend quickly task objectives and environment dynamics from few- shot demonstrations (Schaal 1999 Malaviya al. 2022). Motivated by this, we allow the agent to acquire small number of demonstrations and further leverage them to provide step-wise signals to embodied planning. Specifically,
the demonstrations includes task objective and dynamics con- straints that are lacking in LLMs, while it struggles to handle across diverse scenarios: and in contrast, (ii) LLMs that contain vast amount of commonsense knowledge enabling long-term reasoning on broad set of situations without suffering from OOD) problems, while L.LMs lack domain knowledge and expert guidance for grounding. As result, our work aims to combine the benefits of LLMs with the precise insights from demonstrations to provide few-shot generalization and task-specific grounding simultancously.
(OOD) situations
In this paper, we propose novel framework named Discriminator-Guidee Action Optimization (DGAP) to perform efficient grounding and generalization for the LLM planner at every step. In- stead of imitating expert actions, learn discriminator form score function within the framework of sentence transformers, which measures the align- ment between LLM actions and underlying expert actions the step level. By using meta-prompts that contain previously generated actions with score guidance Yao et al.] (2023a), the LLM planner performs closed-loop optimizer for new actions with high scores through its understanding of existing discriminative action-score pairs. This approach effectively treats the LLM as an optimizer Yang al. 2024), which inherently involves an optimizing process, bypassing the need phys- ical or outcome feedback required by previous methods for refinement. Under mild conditions, we show that DGAP resembles the critic-regularized optimization RL and provable to achieve stronger policy than the LLM planner.
small number of demonstrations to
Our contributions are summarized as follows. (i) We propose novel framework that combines the long-term reasoning of LLMs and task-specific grounding under guidance from small number of demonstrations. (ii) We propose simple discriminator learned with a small number of demon- strations serve step-level score function, which helps the LLM planner generate high-score actions via implicit optimization. (iii) We build theoretical connections between DGAP and critic- regularized optimization in RL, which shows planner under mild conditions. (iv) We conduct extensive experiments challenging Science (Wang et al. 2022) and VirtualHome (Puig et al. 2018) benchmarks combined with different L.M planners (GPT-4 (OpenAl 2023), Llama3-70B (Meta 2024)), and the result shows DGAP obtains superior performance and better efficiency than previous methods.
method obtains stronger policy than the LLM
PRELIMINARIES
The embodied tasks be formalized Markov Decision Process (MDP) defined by tuple M = SC, O, A, P.r, T") In the MDP, I ~ C is the language description a task specifying a high-level goal. For example, Science task description be "your task is to boil lead".
2

---

Published as a conference paper at ICLR 2025
Task: task to grow an orange. First, growing several plants, then... Observation: This room is the bathroom. In it, you see: the agent bathtub... focsaa focus, rule...
Scienceworld
Additional: Action-Score pairs, message Failed message,
DGAP
Planning
Grounding
Virtualhome
Disorimiratto
Tip1:
Review A Action-Scare Scores
Way1: Threshold Handling
Tip2: Subgoal Focus
Way2: Closed-loop Replan
Tip3:
Access High Score
Action: move soil to pot
~>oore:7
Excution Sequence:
Action Sequence: move pot pour jug into pot move seed to sink.
Action: pour jug into pot
1.move soil to pot(Score: 7).
->Soree: 8
2.pour jug into pot 8) (ftx)3. move seed
Action: move seed to Sink=Score:3
(Score: 8)
Figure 2: Framework of DGAP: (i) Acquiring domain-specific knowledge through discriminator's regressive training. (ii) Optimizing action generation via historical action-score pairs.
At time step the state 8t S contains the description of environment information including the agent observations Oz ~ O (i.e., environment feedback of previous action or information queried) and environment states e, E (i.e., details about all visible objects). The action ar (a)s)) is generated by following valid policy т S + A.. For example, an LLM planner π "al""(a,sss) is valid policy that generates an action a, based on the state st. most embodied planning tasks, a valid action should follow some supported action templates such as "use X on Y, examine Y" For example, valid action a; in this task can be "use thermometer on liquid tin," and the initial action ao is always "look around" for showing initial environment information. Each step involves the agent executing the action to interact with the environment, obtaining the next state 8t-1 and some reward r,. For an RL-based agent, the rewards will be used for policy learning via policy gradient algorithms. As for an LLM policy, the policy generates actions based on commonsense knowledge, the provided prompts, and (optionally) the closed-loop feedback.
In DGAP, we additionally acquire a small number of demonstrations, which include the opti- mal action sequences generated by oracle planner oracle. We denote the expert dataset as where M denotes the number of episodes. An imitation policy be 1 )1
B (7, (ah. .i...., a,) learned via Behavior Cloning (BC) in the expert data, by maximizing the log- wel--iikehoodd of expert tions as n = max, Es (T [a] hx)], where ht = ai-1, ... oi-n contains previous actions up to 10. T can perform poorly in real-world interactions due to limited state coverage of demonstrations.
METHOD
In this section, we present the details of our proposed framework, DGAP, as depicted in Fig. 2 This framework capitalizes the domain knowledge embedded in both expert and handcrafted datasets to assign scores to responses from the LLM. These scores then strategically guide LLMS in planning towards specified objectives. Specifically, it consists of three parts: (i) Acquiring domain-specific knowledge through discriminator's regressive training from augmented expert data with score labels in $3.1 (ii) Utilizing the pre-trained discriminator to optimize action generation via historical action- score pairs and ground actions when facing anomalous scores in $3. (iii) Qualitative analysis of DGAP and regulc-regularizze optimization.
3.1 THE DISCRIMINATOR AS SCORER
As previously mentioned, reflection based methods MMddan et al.] 2023 Shinn et al. 2023) and search-based methods (Yao et al. 2023a Chen et al. 2024a] Zhou et al.) 2023a supervised by out- come present inefficient to some extent. Demonstration based methods require large datasets for their scalability in embodied tasks (Rita al. 2024 Sun & van der Schaar 2024). Recognizing the potential complementarity of them, we investigate solution that combines their strengths by em- ploying discriminator. Specifically, the discriminator integrates external knowledge from demon-
3

---

Published as a conference paper at ICLR 2025
strations, subsequently provides score measuring the alignment between the LLM's response and the underlying expert choice within the current context. The process is defined as:
(1) where D& denotes the offline discriminator with parameters Φi I, refers to task objective and environmental information as stated in Sec. 2 ht is a summary of the past ten actions, (z represents the action generated by LLM based (between O and 10), where a higher score indicates greater alignment with the expert policy. A
Dø (1, he, (xll)((,)))) Q:
task goal and state at timestep t. Q is the score
detailed description of the discriminator is in Appendix
Previous research has highlighted the generalization capacity and adaptability of demonstrations in planning for various embodied tasks (Mu et al.) 2023 Lin et al., (023). Unlike existing methods that directly utilize fixed demonstrations to transfer to new scenarios, our approach converts the demon- strations' information into numeric values, enabling measurable step-level feedback. Additionally, We enhance these values through augmentation on limited demonstration samples to improve their generalization across embodied scenarios. This numerie representation used simplify integra- tion with task planning, offering more efficient, scalable solution with dense feedback.
Data Overview The discriminator is designed to numerically differentiate actions. Intuitively, em- bedding regression serves as an effective method for this purpose. A predominant factor contributing to this is the high similarity between expert action embeddings and those of generated actions, which makes it difficult to distinguish and be represented as a score with generalization. To tackle this is- sue, previous studies propose data distribution adjustments such as an unbalanced combination of expert data, sub-optimal data (Xu et al. 2022), and data augmentation to provide generization (Jha jet al.] 2020 Zhang et al., 2024b) utilizing a customized fine-tumed LM (Tan et al.] 2024). In light of them, we adopt a data collection strategy that involves tailored modification expert data, com- bined with comparable random data and a substantial volume of generated data via fine-tuned LM, as illustrated in Fig. And the specific implementing details are stated in Appendix. C
Expert Data (score 10): This parts are composed of oracle trajectories from ScienceWorld (Lin et al. 2023) and VirtualHome (Puig et al., 2018) official datasets, from which we exact and he instructions. The correlated instruction-action pairs adhere to policy zoracle and are assigned score of 10. Together, they constitute dataset B..
Random Data (score 0): We collect a dataset with negative pairs where the instruction is paired with a ground truth action from demonstrations that is the least semantically related, where both the predicate and the object of the action are changed and collectively form the dataset B,..
Offline Data (score within [0, 10]): Accomplishing regression tasks using highly polarized sam- ples presents considerable difficulty. To enrich the diversity of both instructions and actions, we fine-tune a language model (LM) via imitation learning (IL) provide domain knowledge, using the instruction-action pairs in expert data, as detailed in Appendix B Subsequently, we utilize a fine-tuned model to generate action candidates azee through beam search across a range of instruc- tions, thereby forming entirely new pairs. Here be denotes the action generated by the fine-tuned L.M. We employ a pre-trained sentence embedding model extract candidates' features and fur- ther evaluate semantic similarity with ground-truth actions agoracle. This process is described as Sim(aghe, (ggrrcte)) ). In unseen scenarios without ground-truth, we mildly treat the first candidate generated by the LM as 10 for the distribution of fine-tuned small models aligns more closely with the training data domain (Panigrahi et al., 2023). The scores for the subsequent candidates are determined by multiplying their cosine similarity the first candidate by 10. These data hereby make up dataset B2 and constitute a significant proportion of the overall dataset B..
The reason we transfer the similarity naturally ranges between [0. 1] integrals between [0, 10] is that LLMs more effectively process integers than decimals (Gruver et al., 2024). Thus, during the discriminator's 's training, we scaled the scores to span in J0. 10J and subsequently rounded them to integers for the subsequent stage of LLM reasoning to better optimize LLMs' interpretative effec- tiveness.
Model Overview Previous research has demonstrated the significant capability of sentence trans- formers in performing text classification (Cohan et al. (211)). Similarly. We also formulate our dis- criminator's 's training regression task and employ another sentence transformer-based network
4

---

Published as a conference paper at ICLR 2025
B
Imput
Score
Bo
41
Instruction Samplet)
10
B (L.H Q. 10
Flon- 75
(CC.
A.
Unstruction Samplez)
Sintsmplee2, 1ample11"10
Deam Search Sample
Insttrccionn
Sample3)
Süntsample) semglet)) 11
Raic
L2 loss
Instrrctioo,, Sampled)
Süngampled, sample1)
B, (L. Q:0)
Augmented Data F.
RoBERTa
Figure Illustration of dataset construction and discriminator training
backbone, utilizzng the data in Bo, which aims establish the mapping from instruction-actioo pairs to scores. Specifically, we employ the RoBERTa (Liu et al.] (019)) model architecture com- plemented by a linear head get precise score, thereby ensuring robust integration of paired data into our evaluative framework. We explicitly apply L2 loss in training as shown in equation Details are Appendix
L = arg min BLC.N.A)
Σ (olli,hy,a)) Qillz -
-
The introduction discriminator offers intuitive and readily obtainable feedback which supplants accessing feedback through exploration in the environment, and enhances the planning process by directly concentrating on the action-score information.
3.2 LLM OPTIMIZATION WITH DISCRIMINATOR
In this section, outline how to combine the pre-trained discrim- inator with in-context learning to boost planning: (i) Prompts with equip the LLM with the foresight to discern whether each action contributes to the successful completion of the task. (ii) Given the LLM's inherent role as a sampler (Zhao et al.) [023a] Hopkins et al. 2023), closed-loop feedback is introduced to en- sure the optimization process. When the action score at any step falls below a predefined threshold, the LLM planner is required to adjust its policy in response to the suboptimal performance ob- served in the prior iteration.
Illustration
Property
Initial-
Mass- covering
genera
tion
Score- regular ized
Alignment
Prompt with scores Unlike the prevalent use of Outcome- Supervised Prompts, which assess only the final outcome of so- lutions (Hu et al. 2023; Zhou et al, 2023a [Shinn et al. 2023]
Figure 4: Regularizing the LLM policy distribu- tion through score-based replanning
Yao al. 2023b Wang al. 2023c), approach implements Process-Supervised Prompt (PSP) via scores obtained at each step.
Through this way, LLMs are encouraged model and optimize task completion by evaluating inter- mediate steps, enhancing its decision-making in complex, long-term tasks (Hao et al. 2024 Xiong al. 2024). Specifically, LLMs are allowed pairs, dynamically adjusting its next action to maximize future rewards. By focusing the planning on numerical feedback, the approach is similar to reinforcement learning (RL), where agents refine their policies through incremental action-based rewards, as to identify key decision points, evaluate potential future paths, and rebalance its plan according to the stepwise feedback. There are more details in Corollary 3.2 The structured feedback mechanism ensures that each action aligns tions more accurately with long-term objectives while reducing the risk of cumulative errors (Sun & van der Schaar 2024 Rita al. 2024 challenge aforementioned in emboied planning.
the entire trajectory of executed action-score
Building on this theoretical foundation, we integrate step-based scores into the prompt structure. Several studies have demonstrated that LLMs' numerical sensitivity and utilization capabilities can be unveiled through appropriate prompting (Yang et al.] 2024; Liu et al. 2003). Drawing inspi- ration from these findings, have integrated the following context into prompts to heighten
5

---

Published as a conference paper at ICLR 2025
the model's focus and faithfulness scores and further leverage LLMs' capacity generate and optimize existing solutions. For a detailed prompt template, please refer to Appendix D.3
D.3
Previous actions, scores and observations are as follows... You are required to maximize high-score actions cumulatively while adhere to the task Identify the intrinsic relationship between the action-score pairs
Refinement for the failed plan As an inherent role as a sampler (Zhao et al. 2023a Hopkins et al., 2023), the LLM is required to function as optimizer. Considering the diffculty of gener- ating action by score maximizing implicitly and the hindrance caused by high variance in action- score pairs to the LLM, a closed-loop refinement process is introduced. If the score falls below the threshold т, the LLM is asked to replan based on the unsatisfactory action-score pair until Da(h, Ne, dr mhmm T, as illustrated in Fig. 7 This ensures improvements over mim by score, with minimal changes to its policy through the closed-loop replanning process. It also decreases the interaction rounds of agents since the action a, has been evaluated and refined via discrimina- tor feedback before execution. Specifically, we adopt a threshold of 5 for ScienceWone and 6 for VirtualHome, based on their respective training data distributions.
3.3 QUALITATIVE ANALYSIS
In the following, we give qualitative analysis to connect the proposed framework and the critic- regularized optimization problem. Since the score function (s a) measures the similarity be- tween LLM actions and expert actions, it forms an implicit reward function for the LLM agent as Ts = =D a). DGAP, since the planner is prompted to generate an action that maximizes the score function, our method implicitly maximizes the reward considered in the RL framework. Mean- while, since the output of DGAP largely relies on the commonsense knowledge of the initial LLM planner (i.c., n n!ll)), the resulting policy still lies closely to the x ællm, As result, method can be formalized as constrained optimization problem that aims to learn an improved policy dgup over the initial Im by maximizing rewards, as
π dgap arg max E Ex
E.ro(sn an) BDI.. (aee(el))) "Im (ar [s]]s:)]
(3)
29
where the states are sampled from a state distribution dimm (5) induced by the LLM policy, the actions are sampled by following the LLM policy, and B is a balance factor. The cumulative return is defined as Ro(st,a)) at) ET=1 re (8( without discount factor. Then the optimization objective becomes
Edgap_ = EBrddii aampli [Ro(s, a)] - 3DKL [To(a]s) [m]
(4)
In DGAP, since Rø(s, a) comes from learned discriminator, it can be considered as the critic in an RL framework. We remark that such an objective is slightly different from DGAP where the LLM planner is prompted to generate actions that maximize the single-step return rather than the cumulative return since an episode-level discriminator can be more difficult to train. Neverthe- less, for embodied planning tasks in World and VirtualHome, successful multi-step plan requires single-step optimality in each planning step. The objective in equation M resembles a critic- regularized RL objective (Peng et al. 2019). Then the following Lemma gives the solution for me that maximizes this objective.
Lemma 3.1. The optimal policy that solves the constrained optimizaiioonn problem in Ldgap is
~ (als) = π -!'!m (als) exp (Rg(s,a)/3)/22
(5)
where 7()) is a normalized factor to make क (a]s) a valid policy, iee,, fo (als))d 1.
The proof is given in Appendix A Then we have a direct corollary, which is as follows:
Corollary 3.2. The updated policy ($(as)) improves over (all) as Q (s, a) 2 q= (s.a). a).
The proof is given in Appendix A In Corollary 3.2 the Q-function is defined as the expected return Q" (x, (Ro(R a)). As result, the policy n (a)ss is provable to obtain higher expected

---

Published as a conference paper at ICLR 2025
return than o IIlm in the data coverage of the LLM policy. In practice, Z(s) is a partition function that can be hard to estimate, so we have
π dgap X π - (als) exp (Ro(s, a)/3).
(6)
Thus, the resulting policy π dagap combines the benefits of the LLM planner and score function. Specifically, LLM is basic policy that gives candidates actions with high probability, and then the score function forms a filter to choose actions from the candidates with the highest scores. Such a process is simplified DGAP by using LLM as an optimizer, finally obtaining better policies in expected returns. The parameter / is a tuning factor that balances the effect of the LLM planner and score functions. In a special case, when B + co, we have T Ri π llm.
Connection to RLHE Our method is also closely related to Reinforcement Learning from Human Feedback (RIHF) (Ouyang et al. 2022 Bai al. 2022, Touvron al., 2023), which follows similar optimization objective as in equation 3 that optimizes some reward function with constraints to the supervised fine-tuning (SFT) model. However, there exist several major differences. (i) The rewards in DGAP are learned from demonstrations and sampled offline data, while in RLHF are learned from human preference data that can be more expensive to collect. (ii) The reward function in DGAP is trained by a simple regression objective, while RLHF requires BT-model for explicit reward learning or DPO-style optimization for implicit reward learning (Lambert et al. 2024). (iii) We consider LLM itself as optimizer via interactive prompting, while RLHF requires explicit optimization via RL (Ahmadian et al., 2024) or DPO (Rafailov et al. 2024 Bai et al. 2025) by updating the parameters of LLMs. Thus, our method is desirable for APl-based strong LLM models, while RLHF often requires an open-sourced LLM model for reward learning and RL optimization.
4 RELATED WORK
Embodied Planning with LLMs LLMs have exhibited notable reasoning abilities in solving var- ious tasks through in-context examples and prompting techniques like chain-of-thoughh (Wei al.
2022
Vemprala et al., 2023) and tree-of- -fe-oofthouugt (Yao et al. 2023a Zhou et al. 2023a Feng et al.,
2023 However, for embodied tasks, the in-context learning often fails as the embodied knowledge
lacking or even conflicts with that in LLMs. The existing methods introduce closed-loop feedback such as self-reflection mechanisms for self-evaluation and re-plan based on failure analysis (Madaan eet al. 2023, Huang et al. 2022a Yao et al. 2023b Chen et al. 2024a). and external feedback for reflection (Shinn et al.j 2023; Mandi et al.] 2023 Zhou et al. 2023b). making them considerably costly and inefficient in querying interactions. In contrast, we learn score function from demon- strations, eliminating the reliance on self-evaluation and external feedback. Several methods have recently considered LLMs RL agents that interact with the environment collect transitions with external rewards, and perform parameter tuning by RL algorithms (Yao et al. [2024] Zhai et al.) 2024 In contrast, our method is designed for API-based LLMs without parameter tuning. Similar to use, SwiftSage (Lin et al., 2023) adopts demonstrations in LLM planning while relying on imi- tation learning to generate actions. In contrast, we train a score function from demonstrations with augmented data, which leads to a new solution to combine the benefits of LLMs and demonstrations.
LLM for Decision Making. Beyond planning, LLMs can also play other important roles in em- bodied decision-making. (i) LLMs as a reward designer (Ma et al. 2023 Xie et al., 2023 Yu et al.) 2023). The code-writing ability of LLMs can be used generate reward codes according to the robot and task scripts. The reward function is used to train an RL policy, and the feedback from the environment can be used to perform evolutionary optimization for code generation. (ii) LLMs as a world model (Zhao et al. 2023b;; Hao et al. 2023] Murthy et al.) (023). The world model is important for predicting future states and simulating long-term outcomes of actions. I.L. can serve as world model that benefits based RL and LLM planning. (iii) L.LMs foundation policy. LLMs can serve as a policy for imitation learning (Li et al.) 2024, Brohan et al. 2023), and the policy is fine-tuned with embodied data from real-world tasks. (iv) LLM as codes generator. LLM can directly generate robot code for execution (Liang et al. 2023 Mu et al. 2024) or generate value maps to combine with model-predicee control methods (Huang et al. (223). (v) Environment gen- erator. LLMs can generate environments and tasks in a simulator via a closed-loop process (Wang
et al. 2023a b), which can subsequently generate training data for policy learning.
7

---

Published as a conference paper at ICLR 2025
Task Type
*enn
TDT
SFT
Reflexion
S-GPT4
D-GPT4
S-LJama3
D-LIama3
1-1(L)
107.70
0.71
15.00
4.22
97.04
100,00
40.33
82.67
1-2(L)
78.60
0,44
24.40
10.61
87.04
92.75
79.00
91.50
1-3(L)
88.90
3.88
32.20
7.78
72.78
74.00
59.33
82.67
1 1-4(L)
75.20
0,55
57.45
0.92
100.00
100.00
84.00
90.66
2-1(M)
21.40
6,16
9.45
5.92
99.17
100.00
76,00
78.67
2-2(M)
35.20
6.43
6.75
28.59
88.17
80.17
58.00
46.67
2-3(L)
65.00
19.87
5.75
22.37
95.73
88.33
76.00
76.00
3-1(S)
13.60
40.55
70.00
100.00
88.67
91.50
76.00
78.67
3-2(M)
20.80
14.26
48.33
17.45
55.33
58.00
100.00
100.00
3-3(M)
25.60
10.16
59.50
72.54
71.90
78.57
100.00
100.00
3-4(M)
29.00
21.65
69.00
70.22
77.86
88.14
100.00
100.00
4-1(S)
14.60
41.93
100.00
64.93
100.00
100.00
100.00
100.00
4-2(S)
8.80
55.76
100.00
87.27
100.00
100.00
100.00
100.00
4-3(S)
12.60
27.82
94.45
16.42
91.67
100.00
72.33
76.29
4-4(S)
14.60
47.15
100.00
100,00
100.00
100,00
100.00
100.00
5-1(L)
69.50
6.89
13.45
7.33
74.59
73.14
58.00
78.00
5-2(L)
79.60
11.86
44.67
13.00
93.93
90.57
35.67
57.33
6-1(M)
33.60
15.10
26.25
70.35
49.40
57.40
100.00
78.67
6-2(S)
15.10
15.70
53.33
70.67
100.00
100,00
100.00
100.00
6-3(M)
23.00
5.25
8.00
15.77
91.48
92.43
84,67
68.00
7-1(S)
7.00
30.00
11.19
100.00
95.00
100.00
100.00
85.71
7-2(S)
7.00
8.43
83.33
67.50
85.00
85.71
84.67
92.00
7-3(S)
8.00
8.34
100.00
50.00
93.33
92.71
80,00
100.00
8-1(M)
40.00
3.86
77.87
2.58
89.00
100,00
52,00
100.00
8-2(S)
16.30
8.00
33.00
8.00
68.50
38.50
61.67
45.00
9-1(L)
97.00
2.53
8.00
50.63
75.00
75.00
50.00
57.14
9-2(L)
84.90
14.66
73.33
100.00
70.00
83.33
66.67
100.00
9-3(L)
123.10
9.12
73.33
70.62
60.00
71.43
77.67
88.67
10-1(L)
130.10
1.51
53.33
50.90
92.30
87.71
43.00
53,00
10-2(L)
132.10
1.29
17.00
23.69
77.60
78.00
78.00
84,00
Short Medium
11.76
28.37
78.68
71.47
92.22
90.84
87.47
87.77
28.58
10.36
32.90
35.43
77.79
81.84
83,83
84.01
Long
94.30
6.11
32.55
30.17
83.00
84.52
62.31
78.47
Overall
49.26
14.66
49.22
45.34
84.68
85.91
76.43
82.96
Table 1: TASK PERFORMANCE ACROSS BASELINES SCIENCEWORLD.
5 EXPERIMENTS
To evaluate the effectiveness of DGAP and other baseline methods in complex embodied reasoning tasks, we employ ScienceWorld (Wang al. 2002) and VirtualHome (Puig al., (018)) bench- mark. They both encompass a collection of open scenarios and diverse objects for manipulation to accomplish embodied tasks.
5.1 SCIENCEWORLD
Experimental Setup We conducted our evaluation on Science World, virtual textual environment designed for complex science tasks that are structured with 30 different types of science experiments across 10 topics, featuring diverse locations like an art studio, kitchen, and outdoor area. Over 200 types interactive objects and 25 action templates are included.
Our evaluation employed series of test variations that presented unique combinations of objects and scenarios. For example, while our expert data included experiments such as freeze water, the evaluation extended to scenarios requiring the freeze mercury.
Compared Methods We benchmark DGAP methodology against three kinds of approaches: (i) Behavior Cloning-Only: The Text Decision Transformer (TDT) leverages behavior cloning and incorporates reward-to-ge go input, which enables the model predict actions designed to max- imize future expected rewards (Chen et al. 202T). (ii) Planning via Reflecefett Techniques such Reflexion (Shinn et al. 2023 integrate self-reasoning mechanism within the planning pro-
8

---

Published as a conference paper at ICLR 2025
18
22
26
13
Figure 5: Visualizing trajectories of DGAP, SWIFTSAGE and ORACLE, X: time steps, Y: scores. Task identifiers are positioned at the bottom right of each figure, whose detailed information can be found in Fig. 10
cess to enhance reasoning capabilities. (iii) Demonstration Method: STF is based on imitation learning on expert data. SwiftSage (Lin al. 2023) amalgamates rapid thinking with demonstra- tions as our method and methodical reasoning, establishing itself as the state-of-thh-ar baseline in World, making our primary focus of compare.
For our implementations, We used around around half number of demonstrations as ToT, SFT, and SwiftSage, with 10 to trajectories per task. For Reflexion, we provided the three most relevant trajectories in the context each time, ensuring coverage of expert trajectories across tasks. We utilize Llama3-70B and GPT-4 as the foundational Large Language Models. Specifically, S-GPT4 repre- sents the SwiftSage method utilizing GPT-4, while D-GPT4 denotes the DGAP strategy integrated with GPT-4. Similarly, S-Llama3 corresponds to the SwiftSage approach adapted Llama3-70B, and D-Llama3 signifies the DGAP method deployed with Llama3-70B. TDT and Reflexion utilize GPT-4.
Results Our findings are outlined in Tab. 5.1 which elucidates the performance across thirty dis- tinct task types. The details of tasks can be referred in Appendix. Analysis of the results yields several observations: (i) DGAP outperforms SwiftSage in most tasks, suggesting that the external suggestion of alignment with expert data is more effective than limited internal environmental feed- back. (ii) In tasks classified as short, our method shows no substantial superiority over SwiftSage. We suspect this comes from the limited pairs of short task sequences being less effective compared to the plentiful pairs in long trajectories, while short tasks primarily rely on detailed textual informa- tion elicitate LLMs' reasoning prowess. (iii) In addition to the success rate, DGAP demonstrates notable enhancement in efficiency, achieving higher scores in fewer steps as validated by external expert assessments. This increased efficiency is visually represented in Fig. and Appendix [0] (iv) The inferential prowess of various LLMs exhibits discrepancies, with each excelling in distinct areas. For example, GPT-4 shows outstanding performance in tasks 1-1, 1-3, and 1-4, which involve changing the state of objects. Conversely, in more complex scenarios such tasks 3-3, 3-4, and 3-5 that are related to circuits, Llama3 surpasses GPT-4, underscoring the diverse strengths of different models.
5.2 VIRTUALHOME
Experimental Setup VirtualHome is an interactive platform to simulate complex household ac- tivities via programs and train agents perform complete them. also includes Knowledge Base, Providing instructions for a diverse combination of activities, such as put one pancake in slove and switch on stove,put two milk on kitchentable.
We perform experimental evaluations on three distinct settings as delineated in (i et al. 2022): In-Distribution, NovelTasks, and NovelScenes. They differ markedly in task complexity, which we assess using the number of action steps required for each task. Specifically, the average action steps required for In-Distribution are 13.4, for NovelTasks 25.61, and for NovelScenes 27. To ensure robustness and reliability of our findings, we replicate experiments on 60 tasks from each setting three times, thereby gathering comprehensive results. We employ EXEC. and SR. to evaluate the feasibility of the generated plans. where EXEC. measures whether the generated plan can be exccuted in the given environment, and SR. measures the fulfillment of task-specific goal conditions.
We evaluate DGAP against three kinds of approaches: (i) Planning based: L.Mss employ method that directly generates planning results through in-context learning, such as
Compared Methods

---

Published as a conference paper at ICLR 2025
In- Distribution
NovelScenes
NovelTasks
EXLC
SR.
EXEC
SK.
EXBC.
SR.
ProgPrompt
87.33 = 2.02
82.33 + 1.76
38.67 = 1.45
32.33
+: 1.45
49.67 3.18
49.00
+ 3.21
Inner Monologue
79.67 3.38
79.33 1 3.18
54.33 1 1.76
53.33
1.76
47.33
1.67
46.00
1.15
Tree Planner
89.33 + 0.17
41.67
3.20
90.33
0.32
52.33
2.03
DGAP-Llama3
90.67 0.86
84.33 + 2.12
63.00 = 1.68
56.33
4 1.12
78.33
1.03
68.00
2.03
DGAP-GPT4
93.33 1.76
88.00
1 2.45
71.67 1 1.15
62.67
1 1.33
73.67
1.15
72.17
1. 3.18
DGAP-IntemVI.2-8B
84.33 1.15
68.67 + 2.30
57.06 + 2.11
45.33
+ 1.20
52.25
1.00
41.17 + 2.80
Table 2: OVERALL PERFORMANCE DGAP AND BASELINES ACROSS VIRTUALHOME
ProgPrompt (Singh et al. 2023). (ii) Reasoning based: In this approach, LLMs are rendered with a specialized prompting mechanism to enhance their inference when tackling complex tasks, such as Inner Monologue (Huang et al. 2022b). (iii) Search based: This approach reframes the inference process into plan sampling and tree construction, thereby establishing a comprehensive and efficient pathway for task execution (Hu al. 2023). (iii) VLM experiments: We have supplemented our experiments with Vision Language Models (VLMs) in the VirtualHome benchmark. Specifically, utilized Intern VL2-8B to generate the key object states within scenes, replacing the information that was previously obtained directly from the environment graph. Additionally, due the limitations of the VLMs' field of vision, which only allows for the retrieval of objects within the current scene, we adjusted the action step length for each query. Instead of generating full sequence for subgoal at once, the generation now stops when the next action is [walk] (indicating a need to move to a different location) and then begin another in the new scene.
Results The "nn-Distribution" task category is ill-suited to the Tree Planner (Hu et al. 2023) mech- anism, designed to adapt to novel scene and task combinations. So, the experiments with Tree Plan- ner in the In-Distribution setting are ignored. As shown in Tab. 8 the primary results emphasize several specific insights: (i) Across nearly all evaluated metrics and settings, DGAP consistently outperforms competing methodologics. This underscores the superior guidance approach offers over environmental feedback, thereby significantly boosting success rates. (ii) In the majority of settings, DGAP demonstrates a minimal standard deviation. This indicates that our framework aug- ments the embodied capabilities of LLMs and substantially improves their stability and robustness. (iii) Though the Tree Planner exhibits superior performance in EXBC. Due to the action tree being grounded and optimized, our method maintains a lead in success rates. This advantage stems from our continuous action evaluation loop, which refines the strategic foresight of LLMs by leveraging step-wise action-score pairs and real-time contextual inputs. (iv) The results indicate relatively mild impact of using VLMs on action exccutability (EXEC) and a more significant influence on task success rate (SR). This suggests EXEC may rely more heavily on reasoning models rather than perception models, while SR appears to be more sensitive the accuracy of environmental infor- mation. Additionally, we observe that among the three task categories, In-Distribution tasks less affected by VLM-generated information, while the other two categories experience a greater impact. This suggests that familiar tasks exhibit certain level of robustness under varying types of environmental descriptions.
Empirical evidence from experiments two benchmarks corroborates that the DGAP method not only refines the performance of LLMs but also reduces the necessary steps and queries, thereby enhancing the efficiency and effectiveness of the embodied task's planning process.
CONCLUSION
This paper presents the Discriminator-Guudded Action Optimization (DGAP) framework, addressing the challenges of complex embodied tasks which demand extensive, executable planning in dynamic scenerios. By utilizing a few demonstrations, the DGAP framework establishes discriminator with a scoring function as real-time feedback, guiding LLMs to closely align with expert actions. Ex- perimental results demonstrate that DGAP outperforms other baseline methods in benchmarks such as Science World and VirtualHome, showcasing superior performance and higher efficiency. The limitations of this work lie in its suboptimal performance on short-sequence tasks and the additional effort required to prepare the data. In the future, We will also investigate lightweight frameworks and adopt a generalized approach to extend their applicability to a broader range of LLM tasks.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY
To ensure reproducibility, the code for our experiments is available at htap::///gtthii htm.tccbomm Hauf fQi an an/DGAP Detailed information models, data processing steps and experiments can be found there.
ACKNOWLEDGEMENTS
This work is supported by National Key Research and Development Program of China (Grant No. No.20244FF002000000 National Natural Science Foundation China(Grant No. U21A20488) and National Natural Science Foundation of China (Grant No.62306242).
REFERENCES
Jacob Abemettyy, Alekh Agarwal, Teodor V Marinov, and Manfred K Warmuth. A mechanism for sample-efficient in-context learning for sparse retrieval tasks, arXiv preprint arXiv:2305. /7040, 2023.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint 2rX 2024.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou What learning algorithm is in-context learning? investigations with linear models. In International Conference on Learning Representations, 2023.
Chenjia Bai, Yang Zhang, Shuang Qiu, Qiaosheng Zhang, Kang Xu, and Xuelong Li. On- line preference alignment for language models via count-based exploration. arXiv preprint arXiv:2501. 12735, 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05800 2022.
Lawrence Barsalou. Grounded cognition. Annu. Rev. Psychol., 59(1):617-645 2008
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, and et al. RT-2: vision- language-action models transfer web knowledge robotic con- trol. CoRR, abs/2307. 15818, 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15004-15099 2021.
Sijia Chen, Baochun Li, and Di Niu. Boosting of thoughts: Trial-and-croor problem solving with
large language models. In The Tweifth International Conference on Learning Representations, 2024a.
htt.ctt::///penrevieee net. Fome:
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang. Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao, Agent-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403. 12881, 2024b.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instrcctionffinttuna lan- guage models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Arman Cohan, Iz Beltagy, Daniel King. Bhavana Dalvi, and Daniel S Weld. Pretrained language models for sequential sentence classification. arXiv preprint arXiv: / 11999990405 2019.
11

---

Published as a conference paper at ICLR 2025
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint TrXiv:2301.002344 2022.
Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero- like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179 2023.
Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Wilson. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 2024.
Shibo Hao, Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint 2499992.00.. 2023.
Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, et al. Llm reasoners: New evaluation, library, and analysis of step reasoning with large language models. arXiv preprint arXivv2404.05221, 2024.
Aspen K Hopkins, Alex Renda, and Michael Carbin, Can Ilms generate random numbers? evaluat- ing Ilm sampling in controlled domains. In ICML 2023 Workshop: Sampling and Optimization in Discrete Space, 2023.
Mengkang Hu, Yao Mu, Xinmiao Yu, Mingyu Ding, Shiguang Wu, Wenqi Shao, Qiguang Chen, Bin Wang. Yu Qiao, and Ping Luo. Tree-planner, Efficient close-loop task planning with large language models. arXiv preprint arXiv:2310.085822 2023.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and brian ichter. Inner monologue: Embodied reasoning through planning with language models. In Annual Conference Robot Learning, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.056603 2022b.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Annual Conference Robot Learning, 2023.
Rohan Jha, Charles Lovering, and Ellie Pavlick. Does data augmentation improve generalization in nip? arXiv preprint arXiv:2004. 15012, 2020.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision- making. Advances in Neural Information Processing Systems, 35:31 08:31199-31211 31212, 2022.
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, and Tao Kong. Vision-language foundation models as effective robot imitators. In The Tweifth International Conference on Learning Repre- sentations, 2024. URL httph:ttt/./. /ttp::://ppeeeeee http///eeeeeeeeeee//ee/eeeeeeeee n fow.mmid=lFy
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation, pp. 9493-9500. IEEE, 2023.
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabroll, Yejin Choi, and Xiang Ren. Swiftsage: generative agent with fast and slow thinking for complex interactive tasks. In Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2025
An Liu, Zonghan Yang, Zhenhe Zhang, Qingyuan Hu, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. Panda: Preference adaptation for enhancing domain- abilities of llms. arXiv preprint arXhv:2402.12835, 2024.
Shengeai Liu, Caishun Chen, Xinghua Qu, Ke Tang, and Yew-Soon Ong. Large language models as evolutionary optimizers. arXiv preprini arXiv:2310. 19046, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeitlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprini arXiv: /907. 1692, 2019.
Jianlan Luo, Perry Dong, Yuexiang Zhai, Yi Ma. and Sergey Levine. Rlif: Interactive imitation learning reinforcement learning. In The Twelfth International Conference Learning Repre- sentations, 2024. URL http::: 2ot.
net
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. CoRR, abs/2310 10. 12931, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with feedbake. arXiv preprint arXiv:2303. 17651, 2023.
Maya Malaviya, Ilia Sucholutsky, Kerem Oktar, and Thomas Griffiths. Can humans do less-than- one-shot learning? In 44th Annual Meeting of the Cognitive Science Society: Cognitive Diversity, CogSci 2022, 2022.
Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large language models. CoRR, abs/2307 04738, 2023.
Meta. Meta llama 3, hutps//llama.mttt.comllaaaa 2024.
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding. Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. EmbodiedGPT: Vision-language pre-training via embodied chain of thought. In Neural Information Processing Systems, 2023.
Yao Mu, Junting Chen, Qinglong Zhang, Shoufa Chen, Qiaojun Yu, Chongjian Ge, Runjian Chen, Zhixuan Liang, Mengkang Hu, Chaofan Tao, Peize Sun, Haibao Yu, Chao Yang, Wenqi Shao, Wenhai Wang. Jifeng Dai, Yu Qiao, Mingyu Ding, and Ping Luo. Robocodex: Multimodal code generation for robotic behavior synthesis. ArXiv, abs/2402. 16117, 2024. URL https://app. semanticschol wttpschholaar.oooooouuussseee wwwg//orrpssuaaa 267 79836333333
Rithesh Murthy, Shelby Heinecke, Juan Carlos Nicbles, Zhiwei Liu, Le Xue, Weiran Yao, Yihao Feng, Zeyuan Chen, Akash Gokul, Devansh Arpit, et al. Rex: Rapid exploration and exploitation for ai agents. arXiv preprint wxXiv:2307.08966 2023.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill local- ization in fine-tuned language models. International Conference on Machine Learning, pp. 27011-27033, PMLR, 2023.
Xue Bin Peng. Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage regression: Simple and scalable off-policy reinforcement learning. CoRR, abs/1910.00177, 2019.
Xavier Puig, Kevin Ra. Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8494-8502, 2018.
13

---

Published as a conference paper at ICLR 2025
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. Countering reward over-optimization in Ilm with cntnonssrationnguiddee reinforcement learning. arXiv preprint arXiv:2404. /9409, 2024.
Giacomo Rizzolatti and Laila Craighero. The mirror-neuron system. Annu. Rev. Neurosci., 27(1): 169-192, 2004.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, (6):233-244 242, 1999.
Yu-Zhe Shi, Haofei Hou, Zhangqian Bi, Fanxu Meng. Xiang Wei, Lecheng Ruan, and Qining Wang. Autodsl: Automated domain-specific language design for structural representation of procedures with constraints. arXiv preprint arXiv:2446.1223: 2024.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Neural formation Processing Systems, 2023.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523-121 11530. IEEE, 2023.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. LIm-planner: Few-shot grounded planning for embodied agents with large language models. In IEEE/CVF International Conference Computer Vision (ICCV), 2023.
Hao Sun and Mihaela van der Schaar. Inverse-rligmment: Inverse reinforcement learning from demonstrations for I1m alignment. arXiv preprint arXiv:2405.15624 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether thain-of-thouuht can solve them. arXiv preprint arXiv:2210.0920 2022.
Bowen Tan, Yun Zhu, Lijuan Liu, Hongyi Wang, Yonghao Zhuang, Jindong Chen, Eric Xing, and Zhiting Hu. Redcoast: A lightweight tool automate distributed training of lims any gpu/tpus. arXiv preprint arXiv:2310.1633 16355, 2023.
Bowen Tan, Yun Zhu, Lijuan Liu, Eric Xing, Zhiting Hu, and Jindong Chen. Cappy: Outperforming and boosting large multi-task Ims with a small scorer. Advances in Neural Information Processing Systems, 36, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res, 2:20, 2023.
Linui Wang. Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. CoRR, abs/2310.01361. 2023a.
14

---

Published as a conference paper at ICLR 2025
Ruoyao Wang, Peter Jansen, Mlec-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11279-112998 2022.
Yufei Wang. Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. CoRR, abs/2311.01453 2023b.
Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents, arXiv preprint arXiv:2302.011 5600, 2023c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
Yike Wu, Jiatao Zhang, Nan Hu, Lanling Tang, Guilin Qi, Jun Shao, Jie Ren, and Wei Song. Mldt: Multi-level decomposition for complex long-horizon robotic task planning with open- large language model. In International Conference on Database Systems for Advanced Applications, pp. 251-267. Springer, 2024.
Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models. arXiv preprint arXiv:2307.01848, 2023.
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang. and Tao Yu. Text2reward: Automated dense reward function generation for reinforcement learning. CoRR, abs/2309. 1489, 2023.
Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! Ilm agent learning via iterative step-level process refinement. arXiv preprint arXiv:2446.1111 2024.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discrininator-weigheed offline imitation learning from suboptimal demonstrations. In Proceedings of the 39th International Conference on Machine Learning, pp. 24725-24742, 2022. URL
http:///ooocccieeee/ .mlr press/
2162/xu2213 html
Chengrun Yang. Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations, 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Neural Information Processing Systems, 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learming Representations, 2023b.
Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh R N, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phil L Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. Retroformer: Retrospective large language agents with policy gradient op- timization. In The Twelfth International Conference on Learning Representations, 2024. URL wwwppppeeeeeeeeeeeeeeeeeeeeeeeee htttss:/epeereviiet
https: /
/ forum httts//oormm11ckO00. 1d= 11cbb
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang Huei Lee, Montse Gonzalez Are- nas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. CoRR, 0bs/23006006647, 2023.
Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong. Yifei Zhou, Alane Suhr, Sain- ing Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large dision-language models as decision-making agents via reinforcement learning, 2024. URL https: // /aaxivivvv...///vv/./av////vvavv wwwp:xv..vr.........o... abs/ 2405. 10292
15

---

Published as a conference paper at ICLR 2025
Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Lim self- training via process reward guided tree search. arXiv preprint arXiv:2406.03816, 2024a.
Yang Zhang, Shixin Yang, Chenjia Bai, Fei Wu, Xiu Li, Zhen Wang, and Xuelong Li. Towards efficient IIm grounding for embodied multi-agent collaboration. arXiv preprint arXiv:2405.14311 2024b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong, preprint arXiv:2303. 182223,303.1823,, 2023a.
survey of large language models. arXiv
Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36, 2023b.
Andy Zhou, Kai Yan, Michal Seapeettokh-Rothmmae Haohan Wang, and Yu-Xiong Wang. Lan- guage agent tree search unifies reasoning acting and planning language models. arXiv preprint 1rXii:2310.04400 2023a.
Zhehua Zhou, Jiayang Song, Kunpeng Yao, Zhan Shu, and Lei Ma. Isr-llm: Iterative self- refined large language model for long-horizon sequential task planning. arXiv preprint 133244,308.3374, 2023b.
16

---

Published as a conference paper at ICLR 2025
A THEORETICAL PROOF
A.1 PROOF OF LEMMA 3.1
Proof. We recall the critic-regularized problem as follows.
s.t. / no(als)da (7)
dga
E Brndi Um anegli Ra a)] 3DKL - [o(a|s)|[]
In constrained optimization problem, 3 can be considered Lagrange multiplier that controls the KL-divergence between the learned policy and the basic LLM policy. Then we convert into Lagrangian form by introducing factor a,
((T0,3,0) [dim(s)) / 7 (als) a) dads 3 dinn (xL [=o(-s)]] ( ds
{.a as [1-[m(assddd ds.
(8)
with 3 and Vs S} corresponding to the Lagrange multipliers. Differentiating L(mo, B, with respect to [() results in
OC (T(as))
dm) (s) a) 3dim (8) log No (a|s) Bdimm (8) log क Thm (a|s) - -Bel (8) as.. (9)
Setting zero and solving for Ta(als) gives
1 Ro Ro(8, a) log Iim (als)) -1
Os B di
log up B
(10)
Then we have
To(a|s) = -|ا|ا (als) exp (+ exp
(-ax
(-)
a di
(1I)
Since f. (als) 1, the second exponential term the partition function Z(s) that normalizes the conditional action distribution, as
Ca 1 +1)
/ x 4l (i (a'ls) (dlss)exp exp (1 () da'.
Z(s) = exp
(12)
3
dim
Then the optimal policy is given by
n Ul (als) exp ( R)((saa)
(als) = Z(s)
(13)
which concludes our proof.
A.2 PROOF OF COROLLARY 3.2
Proof. We remark that the objective function Ldgap also be formulated constrained opti- mization problem. Considering a tabular case with finite state and actions, we have
z arg max E, Egndy Th
Σ
o Iim (s, af (s, (s. s.t. DKL(To(-Iss) (m]{s] S c, Vs, (14)
where we use (T a) to denote E, (s o)). is easy to check that equation 14 has the same solution equation by relaxing the hard KI. constraint into soft constraint with coefficient B. From equation 14, we have
Σ (8 (s) 1im (s,a) 2 Σ n Um (s, (s,aa)""" (s, (s,4), Vs.
(15)
17

---

Published as a conference paper at ICLR 2025
Then have
mm (s, = E a+) + Σ x (04118441)9 (5141, 02+1) 0,
E
r(0t, ue) + Σ n0(a 1l((t+il++t))@ (01+1,
8, =1
E
r(s,, ae) + Σ To(a++als(41) ((((+1, + Σ (2422224442)) ($1+2,
E
(st,, ae) + Σ T (((++118(41))
((8441,
Th (044242))122200 (8142,
(16)
Repeat this process expand the equation, we have
T-
Q Q' (8,a) S Bug
Σ
(1(5 8
(8,
(17)
0=
which concludes our proof.
B LM FINE-TUNING
We selected FLAN-T5-large as the base model due to its fine-tuning on datasets phrased as instruc- tions, which improves its ability to follow and respond to instructions. According to the research detailed in (Chung et al. 2024), the model architecture includes:
Number of layers: 24
Number of attention heads: 16
Hidden layer size: 1024
Feedforward Network Size: 4096
Activation Funetion: ReLU (Rectified Linear Unit)
Total parameters: 770 million
Further conduct imitation learning FLAN-TS-Large using expert data in format of (1, h4) groadde(t.))) The data primarily encompasses three components: 1.a description of the task, 2.a record of historical actions, and 3.the actions that ought be executed. The specifies delineated below.
GcienceWorldh
("input": "Your task is to measure the melting point of lead, which is located around the kitchen. 10. look around.". "Action": "go to hallway")
("input": "Your task is to measure the melting point of lead, which is located around the kitchen. 10. look around. 9. go hallway." ". "Action": "open door to kitchen")
("input": "Your task is to measure the melting point of lead, which is located around the kitchen. 10. look around. 9. go hallway. 8, open door kitchen.". ", "Action": "go to kitchen"]
VirtualHome:
("input": "Task: Pay bills. [Walk] <home -ofineefffie>>>>--oe (1)", "Action": "[Walk] filling_cabbnet> (1)"}
("input": "Task: Pay bills. [Walk] <home <home_office> (1), [Walk] <filing_cabinet> (1)", "Ac-
tion": "[Find] <bills> (I)")
("input": "Task: Pay bills. [Walk] <home saaeeee_offfe>>eeee (1), [Walk] <ablinggcabine>> (1), [Find]
<bills> (1)", "Action": "[Grab] <bills> (1)")
Then we refined FLAN-T5-large with the dataset previously outlined, including 15k samples from ScienceWorld and 34k from VirtualHome. For the training, we employed the Adam optimizer with epsilon value of le-06, learning rate of 1e-4, and batch size of 32. We conducted training epochs comprising 25000 steps total. We employ four A100 GPUs for conducting this task, consuming eight hours.
18

---

Published as a conference paper at ICLR 2025
C DISCRIMINATOR TRAINING
Data Preparation For training the discriminator, we collected diverse dataset consisting of pos- itive, negative and augmented samples, illustrated 3.1 Our motivation to assess the generaliza- tion necessitated the judicious use of expert data, thus preventing its overuse, particularly for tasks like Measuring Boiling Point, Testing Conductivity, and Growing, each initially comprising approx- imately 10k samples. Thus in ScienceWorld, to mitigate the dependency on expert information, we reduced the expert samples for each task to 1.5k, resulting in total expert dataset of 45k. In par- allel, we sampled 45k negative random samples. Finally, we generated 400k augmented samples, thereby creating a comprchensive pool of 500k samples. In VirtualHome, we utilized the entirety of available expert samples, given the infrequency of task repetition and total count of only 34k. We constructed a dataset of 400k instances a manner similar to that employed for ScienceWorld The specific format of the data is h4, a4) q, represented as follows:
Science Expld-Expert:
("input": "Your task is to measure the melting point of lead, which is located around the kitchen. look around. Action: go to hallway", "Score": "10"
("input": "Your lask is to measure the melting point of lead, which is located around the kitchen. 10. look around. 9. go to hallway. Action: open door to kitchen", "Score": "10" } CorineeeWrrd-Raaaa
("input": "Your task is to measure the melting point of lead, which is located around the kitchen. 10. look around. Action: look at art studio", "Score": "0"
("imput": "Your task is to measure the melting point of lead, which is located around the
kitchen. 10. look around. 9. go hallway. Action: put down orange", "Score": "0"
Science World- Worldd-Augmened:: ("input": "Your task is measure the melting point of lead, which is located around the kitchen. 10. look around. Action: look at hallway", "Score": "9.03" ("input": "Your task is measure the melting point of lead, which is located around the kitchen. 10. look around. 9. look at hallway. Action: open door to outside", "Score": "6.13"
("input": "Your task is to measure the melting point of lead, which is located around the kitchen. 10. look around. 9. look at hallway. 8. open door to outside. Action: teleport to kitchen". "Score": "8.87"
HattaalHmm--Epperrt
("input": "Task: Pay bills. [Walk] <home_offices (I). Action: [Walk] ccaling_
(I)","Score"" "10")
("input": "Task: Pay bills. [Walk] <home -ofmeeoffic>>>>>> (1), [Walk] sfilinn_cabinel (1), Action:
[Find] <bills> (1)", "Score": "10")
VirtualHome-Randeee Random:
("input": "Task: Pay bills. [Walk] <home _aanee_fffce>>>>> (1). Action: [open] <microwave>
"S""""""""""""
("input": "Task: Pay bills. [Walk] conumeofffce>>> (1), [Walk] <filing (1), Action:
[grab] <chicken> (1)", "Score": "0")
("input": "Task: Pay bills. [Walk] <home <homm__offce>>>> (1). Action: [walk] <kitchencabinet>
(I)", "Score": "8.32")
<home_offices office> (I), [walk] <kitchencabinet> (1). Action: [grab] <bills> (1)", "Score": "7.42"}
("input": "Task: Pay bills. [Walk] <home
("input": "Task: Pay bills. [Walk] chome_officer (1), [walk] <kitchencabinet> (1), [grab]
<bills> (1), Action: [walk] <livingroom>(1)), "Score": "6.99")
10 0 Sim (azbe, agoracte) ) 10 if data in B,
data in B.
if data in B,
(18)
19

---

Published as a conference paper at ICLR 2025
Model Architecture We employ RoBERTa complemented by a linear head, which facilitates di- rect output for regression tasks. This configuration leverages RoBERTa's 's robust contextual embed- ding capabiliiiies while the linear head(768,1) simplifies the mapping from embedded space to target labels, the model architecture is listed:
Number of Layers: 12 layers
Hidden Size: 768
Number of Attention Heads: 12
Feedforward Network Size: 3072
Activation Function: GELU
Total parameters: 125 million
Training Procedure We assembled datasets containing 500k instances for Science World and 400k instances for VirtualHome seperately, each formatted as (instruction, action, score). The model was initialized with RoBERTa parameters and optimized using the Adam optimizer a learning rate of le-5, warmup rate of 0.1, and batch size of 32. For detailed training specifications, please refer to (Tan et al. 2023). During training, the inputs comprising instructions and actions are given into the RoBERTa model to obtain the last hidden state. This state is then processed through linear head to compute scores, which are compared against labels to determine the 1.2 loss. Subsequently, this loss is used to update the model parameters. We employ four A100 GPUs for conducting this
task, consuming around forty hours.
az) (1, ac+1)
qr
9c+1
Figure 6: Illustration of applying discriminator to LLM
Discriminator Application Fig. 6 depicts the functioning of a discriminator within a LLM inter- acting framework. Initially, the discriminator receives input tuples composed of a task decription history actions ht and response action of LLM oę. It evaluates these inputs to generate score Ct. which assesses the relevance the expert response of the input tuple.
As the interaction progresses, the discriminator's ' inputs are advanced to the subsequent state, en- capsulated by the tuple (1, h dx+1). Here, h4+1 incorporates the previously generated action a and based on the new context the LLM subsequently derives the new action a++1- In response, the discriminator calculates subsequent score, 9++1, for this new input.
The blue and orange dashed lines in the diagram represent information transmission at different time steps, highlighting the iterative and conditional role of the discriminator in evaluating successive actions in the sequence.
Actions generated by the Large Language Model (LLM) are not executed immediately but instead stored in an action buffer. These actions are subsequently scored by discriminator prior to execu- tion. If the score below 4, replanning process called Score-based Search triggered, wherein the discriminator evaluates and selects the highest-scoring action from the set of valid actions for execution, as shown in Fig. 7 Conversely, if the score exceeds 8, the action is highlighted in the subsequent interaction round by including it as 'Noted: history action-scort pair' in the prompt, ensuring its prominence. This procedure ensures that the discriminator not only accesses LLM in- ferences but also grounds actions when necessary prevent anomalies.
20

---

Published as a conference paper at ICLR 2025
Step
open cupboard pick up pot open blast fimaace
8
Step 2
Step 3
teleport to foundry
Fix
-[Instruct]:
Improve plans..
[[aakk Context]
[[Observation]
-[Action-Score
Discriminator
Pairs]
|Related Actions]
Figure 7: Illustration of score-basee search
D STATISTICS AND DETAILS IN PLANNING
D.1 EXPERIMENT STATICS
Table provides an overview of the 30 distinct tasks within the ScienceWorld benchmark, catego- rized by task type, topic, and associated attributes. For the evaluation phase, optimized time efficiency by selecting only the first 10 variations for tasks that originally had more than 10 test variations, resulting in total of 270 task variations. This approach ensured fair and cost-effective comparisons across agents. Jalso includes trajectory statistics, where *Len represents the av- erage length of the oracle agent's trajectories, offering insight into task complexity across different task types.
D.2 BASELINES
Science World Baselines We benchmark DGAP methodology against three kinds of approaches: (i) Behavior Cloning-Only: The Text Decision Transformer (TDT) leverages behavior cloning and incorporates reward-to-go as input, which enables the model predict actions designed to max- imize futuree expected rewards (Chen et al. 2021). (ii) Planning via Reff-Reflection Techniques such as ReAct (Yao et al. 2023b) and Retlexion (Shnn al., 2023) integrate self-reasoning mechanism within the planning process to enhance reasoning capabilities. (iii) Demonstration-
Driven Method: SwiftSage (Lin al. 2023) amalgamates rapid thinking with demonstrations as our method and methodical reasoning, establishing itself as the state-of-the-art baseline in Science- utilize Llama3-70B and GPT-4 as the foundational Large Language Models. Specifically, S-GPT4 represents the Swift- Sage method utilizing GPT-4, while D-GPT4 denotes the DGAP strategy integrated with GPT-4. Similarly, S-Llama3 corresponds to the SwiftSage approach adapted Llama3-70B, and D-Llama3 signifies the DGAP method deployed with Llama3-70B. TDT ReAct and Reflexion utilize GPT-4.
World, making it our primary focus of compare. For implementations,
VirtualHome Baselines We evaluate DGAP against three kinds of approaches: (i) Planning based: LLMs employ a method that directly generates planning results through in-context learn- ing, such as ProgPrompt (Singh al. 2023). (ii) Reasoning based: In this approach, LLMs are rendered with specialized prompting mechanism to enhance their inference when tackling complex tasks, such Inner Monologue (Huang et al. 2022b). (iii) Search based: This approach reframes the inference process into plan sampling and tree construction, thereby establishing a comprehensive and efficient pathway for task execution (Hu al. 2023
21

---

Published as a conference paper at ICLR 2025
You are giving exocutable advico and instructions in science experiment..
let's do science exper ments with sequence of ections... enviranment, few
Example task: Your task to use chernistry to create green paint...
Example Your use chemistry to create
paint...
hallway) Action: telegort to studio(Score:10) (in
-iin hallwayy Action: TELEPORT to art studiofScore:100 ->> You to the studio.
studio) Action:
containing tlue paint in
studio into [ug/Score:10)
-[in art studio) Action: POUR cup contalningg blue paint in
art studio into jug(Score:10)
studio) Actions
contalning yellow paint
(ug[Score:10)
pour
the wood cup into the jug.
studio) art studio into (ug[Score:10) You pour the contents of wood cup the jug.
cup contalning yellow
new science experiment.
There are few room
Action History:
experiment.
-1. You move the studio studio.(Score:10) (Score:10)
-2. art studio, pour red paint in cup containing red paint
previous actions,
observations follows:
into
nothing table(Score:5)
hallway) You move the art studio, (Score:0) studio) In studio, red
-3, art studio, mix paint wood cup containing red (Sorn[(C
containing red paint inco cup containing nothing in table(Score:5) (in art studio) art studio, mix paint wood cup containing red paint(Score:5)
-4, in art studio, Scire:)
studio, Score:)
(in art studio) In art studio, wait[Scoress: studio)
This room called the kitchen in you see..
wait(Score:5)
nothing.
counter. On the counter is: a bowl
need to obtain next steps with scores as high as possible.
cupboard The supboard
Response to Questions in Stage 1.
Task Goal:
You maximize your score complete efficiently and sorrectly by high score action.
This room called the kitchen. In it you
Please consider the environment and action information provided above to obtain the next actions to achieve higher scores.:
2. Please identify the intrinsic relationship between action-score peirs and decide the progress corresponding to the successful experiment
The ONLY allowed action types
3. Please review the task description and the previous observations and then answer the following questions
-WAT() :watt for something be done, for example, an object to bolled
Please review description and observation then subgoal
(rao))) directly go to a room such as TELEPORT[kttchen
Question:
1. List required materials...
Please think carefully and be concise use the below format to organize the response. Action 1: [...]]
Identify five subgoals focusing specifled items. 3. Indicate achieved subgoals and outiine immediate next
Action 2:
Figure 8: Prompt Template in ScienceWorld
22

---

Published as a conference paper at ICLR 2025
Task Num
Type
Topic
Name
"Lens
1
1-1
Matter
Changes of State (Boiling)
107.7
2
1-4
Matter
Changes of State (Any)
75.2
3
6-1
Chemistry
Mixing (generic)
33.6
4
6-2
Chemistry
Mixing paints (secondary colours)
15.1
5
6-3
Chemistry
Mixing paints (tertiary colours)
23
6
4-4
Classification
Find an animal
14.6
7
4-1
Classification
Find a living thing
14.6
8
4-2
Classification
Find a non-living thing
8.8
9
4-3
Classification
Find plant
12.6
10
1-3
Matter
Changes of State (Freezing)
88.9
11
5-2
Biology
Grow fruit
79.6
12
5-1
Biology
Grow a plant
69.5
13
8-2
Biology
Identify life stages (animal)
16.3
14
8-1
Biology
Identify life stages (plant)
40
15
9-1
Forces
Inclined Planes (determine angle)
97
16
9-2
Forces
Friction (known surfaces)
84.9
17
9.3
Forces
Friction (unknown surfaces)
123.1
18
7-1
Biology
Identify longest-lived animal
7
19
7-3
Biology
Identify longest-lived animal
8
20
7-2
Biology
Identify shortest-lived animal
7
21
2-2
Measurement
Measuring Boiling Point (known) Measuring Boiling Point (unknown)
35.2
22
2-3
Measurement
65
23
1-2
Matter
Changes of State (Melting)
78.6
24
10-1
Biology
Mendelian Genetics (known plants)
130.1
25
10-2
Biology
Mendelian Genetics (unknown plants)
132.
26
3-1
Electricity
Create circuit
13.6
27
3-2
Electricity
Renewable vs Non-renewable Energy
20.8
28
3-3
Electricity
Test Conductivity (known)
25.6
29
3-4
Electricity
Test Conductivity (unknown)
29
30
2-1
Measurement
Use Thermometer
21.4
Short <"Len S 20)
Total: 10
11.76
Medium (20 <*L.n < 50)
Total: 8
28.58
Long ('Ln 50)
Total: 12
94.30
Overall
Total: 30
49.26
Table 3: The table presents detailed information the 30 distinct tasks in the Science World bench- mark, categorized by task number, type, topic, and name. *Len refers to the average length of the oracle agent's trajectories, based on which tasks are grouped into short, medium, and long cate- gories, indicating the complexity and duration of each task. Additionally, the breakdown by task type (e.g., Matter, Biology, Chemistry) highlights the diversity of domains covered in the bench- mark.
D.3
PROMPT DETAILS
Our prompt design was inspired by the two-stage framework of SwiftSage (Lin et al. 2023). and further incorporate DGAP information at various places in the context(marked in red).
Our prompt was influenced by ProgPrompt (Singh et al. 2023) and Pree-Planner (Hu et al. 2023 and incorporates score-related information into the context(marked in red).
E ADDITIONAL. RESULTS AND ANALYSIS
Ablation Study In ScienceWWoldd the tasks 1-15 are based on the DGAP-Llama3, whereas tasks 16-30 are according DGAP--PT4, both excluding content related scores from the context
23

---

Published as a conference paper at ICLR 2025
from actions import walk <obj>, grab <obj?, sw tchon <obj>, switchoff <obj>... The total task goal put chicken_onn kitchentabl (iteenttabl((::13111 The completed task goal:
Rule:
# remeber the key object INSIDE kitchencabinet
The completed Action:
[walk] <livingroom> (262) (Score:8)
[walk] <pancake> (342) (Score:8)
[grab] <pancake> (342) (Score:9)
There are some examples:
The task goal: put cupcake in stove and switch stove find (nntnccaaaii(((d1111))))))
(Score:10)
open[ (kikchenaaiiiee((11111)) (Score:10)
find (ccpcccceeddd3344)) (Score:10)
grab (cupcake(id:33))) (Score:10)
Question:
#The currnet task goal: find stove(id:1150): #Remember obtain actions with scores as high as possible Def task():.....
Figure 9: Prompt Template in VirtualHome
10
12
14
15
16
17
18
22
24
26
27
28
30
Figure 10: Visualizing trajectories of DGAP, SWIFTSAGE and ORACLE, the x-axis represents time steps, ranging from 0 to T", while the y-axis denotes scores, which vary from 0 to 100. Each graph illustrates the trajectories corresponding to different tasks in test variation. Task identifiers are posi- tioned the bottom right of each figure, whose detailed information be found in Tab. 3
24

---

Published as a conference paper at ICLR 2025
Task
D-LIama3
w/o Score
Task
D-GPT4
New Data
82.67
77.33
16
83.33
78.57
2
90.66
76.00
17
71.43
85.71
3
78.67
66.50
18
100.00
100.00
100.00
100.00
19
92.71
92.71
5 68.00
100.00
20
85.71
78.57
6 100.00
85.71
21
80.17
62.86
7
100.00
87.50
22
88.33
50.14
8 100.00
100.00
23
92.75
80.67
9
76.29
88.14
24
87.71
100.00
10
82.67
65.33
25
78.00
46.14
11
57.33
45.86
26
91.50
83.00
12
78.00
73.43
27
58.00
50.50
13
45.00
40.00
28
78.57
77.80
14
100.00
100.00
29
88.14
87.43
15
57.14
71.43
30
100.00
61.43
Table 4: Ablation study of DGAP in ScienceWorkd
In In-Diitribution
NovelScenes
NovelTasks
EXEC.
SR.
EXEC.
SR.
EXEC.
SR.
ProgPrompt
87,33 = 2.02
82.33 + 1.76
38.67 + 1.45
32.33 = 1.45
49.67 3.18
49.00 = 3.21
Inner Monologue
79.67 + 3.38
79.33 + 3.18
54.33 4: 1.76
53.33 = 1.76
47.33 + 1.67
46.00 = 1.15
Tree Planner
89.33 + 0.17
41.67 = 3.20
90.33 + 0.32
52.33 = 2.03
DGAP-Llama3
90.67 = 0.86
84.33 + 2.12
63,00
+ 1.68
56.33 1.12
78.33 - 1.03
68.00 2,03
DGAP-GPT4
93.33 = 1.76
88.00
+ 2.45
71.67
= 1.15
62.67 1.33
73.67
1.15
72.17 = 3.18
D-GPT4 w/o Score
92.33 + 2.19
86.00 1 2.65
70.67
2.67
60.67 0.63
70.33
1 3.06
70.00 = 3.21
Table 5: ABLATION STUDY OF DGAP AND BASELINES
and scoring thresholds mechanism for handling the exception. The results reveal that tasks 5, 9, 15, 17, and 24 exhibited better performance after the removal of score-related content, spanning both short and long task types. This phenomena likely stems from a significant deviation between the augmented data used for training the discriminator and the actual data encountered in real- world interactions in these tasks. It is noteworthy that tasks 6-3, 4-3, 9-1, 9-3, and 10-1 exhibit no significant performance improvements when evaluated against the SwiftSage baseline.
In VirtualHome environment, we selected DGAP-GPT4 for an ablation study due to its significantly superior performance compared to DGAP-Llama3. The results indicate that, following the removal of score-related context and the absence of scoring thresholds mechanism, both the success rate and the execution rate decreased by to 2 percent. This demonstrates that DSAP is effective not only in science tasks but also in houschold tasks, where it provides substantial guidance.
E. ALFRED EXPERIMENTS
Methods
Seen
Unseen
SR
GC
SR
GC
LLMPlanner
0.121
0.267
0.162
0.402
LoTaBench
0.255
0.442
0.241
0.398
Prompter
0.494
0.560
0.423
0.537
5Turoo
0.121
0.267
0.162
0.402
DGAP-40
0.462
0.507
0.441
0.545
DGAP-40-v
0.323
0.404
0.381
0.414
DGAP-IntemWL2--e VL2-8B
0.236
0.291
0.246
0.391
Table 6: Performance comparison across methods on ALFRED seen and unseen tasks.
25

---

shares Published as a conference paper at ICLR 2025
Analysis of Methods and Results
compares several methods on Seen and Unseen environments using Success Rate (SR) and Goal Completion (GC) metrics. LLMPlanner and LoTaBench rely only LLMs and expert data for in-context learning. Prompter integrates semantic map information and combines low- level parsing with LLMs. DGAP-3,5Turbo uses 3pt-3.5-turbo--e S-turbo-0122 25 as the backbone model. DGAP-40 is based on Azure's ' GPT-4o model. P-4ov shares the same GPT-4o backbone but incorporates updated environmental maps during the planning phase. DGAP-InteraVL.2 the same settings as DGAP-40-v, including updated environmental maps.
DGAP-40 only textual environment information and one-fifth of the expert data for discrim- inator's training, yet achieves competitive results. In Seen environments, DGAP achieves strong performance (SR: 0.462, GC: 0.507), slightly behind Prompter. However, Unseen environments, DGAP outperforms Prompter in GC (0.545 vs. 0,537), demonstrating better generalization. On the other hand, DGAP-40-v, which incorporates RGB images into the planning process, shows decline in performance. It's likely attributed to two main factors: (I) the prompts and discriminator have not been fully optimized to utilize visual information. (2) the RGB images may introduce ambiguous or redundant information rather than render location information. These observations suggest that the integration visual data requires more refined strategies to fully unlock its potential benefits.
Weaknesses of DGAP:Despite its strengths, there two notable limitations that affect its perfor- mance in certain scenarios:
The ground truth action sequences occasionally lack logical co- herence or completeness. For example, in the "Turn on the bedroom lamp" task, the ground truth includes extraneous steps such as "Pick up CellPhone" before toggling the lamp. In the "Put plate in cabinet" task, additional steps like "Place plate in the fridge on the top shelf" are required. Task "Put bowl with a pencil in it on the desk," " where the ground truth includes unnecessary inter- mediate step to "Put the bowl on the Shelf." There are various such cases which introduce noise and reduce alignment between the generated actions and expected outcomes for LLM planning,
1. Inconsistent Ground Truth:
2. Ambiguity in Object Location:
For tasks involving similar objects, selecting one randomly among them can often suffice to meet the task's objective. However, scenarios requiring precise identification of specific storage locations (e.g., determining the correct cabinet where an object is stored), DGAP faces challenges due the absence of explicit spatial or semantic distinctions in the textual input. This limitation impacts its ability to perform accurate object placement, leading to reduced performance on tasks that rely on spatial reasoning and clear differentiation between similar storage units.
E.2 VH-LONG-TASK
Object Extraction
Goal Construction
Complexity Verification
Grabable Object
Goal Construction
Task Goal
put plate on kitchentable
put <object on kitchertable
2. put parcake in fridge
plate
pancake
3. put chicken
put plate
Step Calcullation
Environment Graph
put <object> dre
Min Stap
put pancake fridge
Step Threshald
Goal Object
put <object>
Yes
fridge
put chicken stove
Long Tasks
Figure 11: The framework of task constructing
26

---

Published as a conference paper at ICLR 2025
Extended Task Settings and Dataset Construction To better reflect the nigh-dimensional and continuous tasks commonly encountered real-world scenarios, which are typically multi-goal goal in nature. we extended the task settings (Wu et al. 2024) in the VirtualHome environment to include self-constructed long tasks. By concatenating multiple task objectives, we created tasks exceeding 60 steps, providing
complex and realistic evaluation of long-horizon planning. The statistics of these extended tasks are summarized in the table below.
Dataset
Goals Number
Action Step
Objects Number
Objects Variety
In-Distribution 3.40
26.58
5.32
3.57
NovelScenes
3,39
26.27
5.32
3.56
NovelTasks
3.99
27.02
4.97
3.40
LangTasks Tasks
9.74
77.01
15.79
8.50
Table 7: Dataset statistics for various tasks in VirtualHome.
Extreme- LONG
NovelScenes
NovelTasks
EXEC.
SR.
EXEC
SR.
EXEC.
SR.
ProgPrompt
42.95 +:2 1.22
16.03 + 1.28
38.67 = 1.45
32.33
1.45
49.67 + 3.18
49.00 + 3.21
Inner Monologue
48.45 + 1.05
16.16 + 1.35
54.33 = 1.76
53.33
- 1.76
47.33 + 1.67
46.00 + 1.15
Tree Planner
85.76 + 1.40
19.07
+ 1.79
89.33
= 0.17
41.67
3.20
90.33 + 0.32
52.33
+ 2.03
DGAP-GPT40
78.23 + 1.15
67.25
+ 2.10
71.67 = 1.15
62.67
1.33
73.67 + 1.15
72.17 + 3.18
Table 8: OVERALL PERFORMANCE DGAP AND BASELINES ACROSS VH-LONG-TASK
Experimental results show that method significantly outperforms the bascline in success rate, highlighting its superiority in handling high-complexity, long-horizon tasks and demonstrating the generalization capability of the discriminator.
E.3 DISCRIMINATOR EVALUATION
Methods
Out-of- Distriibitiiittonn Acc
In In-Distribution
PER
OPer
Var
Acc
Var
4x
9.17
0.11
9.28
0.08
85.91
2x
9.16
0.11
9.28
0.08
85.91
0.4x
8.77
0.15
9.19
0.12
83,25
basis
8.08
0.36
8.16
0.36
82.26
83.53
basiswithforce basiswithforcebio
8.06
0.38
8.15
0.26
82.31
84.51
8.91
0.20
9.01
0.17
84.98
86.68
Ix(ours)
9.08
0.15
9.21
0.11
85.91
Table 9: Performance comparison of discriminator across data with different volumn and categories, with including PER and OPer metrics.
E.4 SEARCI-BASED METHOPS EVALUATION
We utilized the Vanna framework construct the RAG pipeline, incorporating the bge-large model as the embedding backbone. The content matching process was based on combination of task instructions and the initial environmental observations. We subsequently present its recall perfor- mance and evaluate its effectiveness within the context of SwiftSage's deliberative reasoning, highly effective in-context learning framework for ScienceWorld. The results are shown in Table
10
E.5 QUALITATIVE ANALYSIS
Feedback Mechanism S-GPT4 failed in Task8 due to its strict adherence to the rule prohibiting focus counterparts when conditions not fulfilled, exposing the limitations of relying solely
27

---

Published as a conference paper at ICLR 2025
Task Type
Len
S-GPT4
D-GPT4
Recall (%)
Performance
I-)(L)
107.70
97.04
100.00
91.8
90.89
I-2(L)
78.60
87.04
92.75
89.2
89,20
1-3(L)
88.90
72.78
74.00
87.2
57.20
1-4(L)
75.20
100.00
100.00
90.6
89,41
2-1(M)
21.40
99.17
100.00
94.8
93.99
2-2(M)
35.20
88.17
80.17
88.9
82.90
2-3(L)
65.00
95.73
88.33
86.4
86.40
3-1(S)
13.60
88.67
91.50
97.8
100.00
3-2(M)
20.80
55.33
58.00
89.6
49.60
3-3(M)
25.60
71,90
78.57
90.0
68,44
3-4(M)
29.00
77.86
88.14
91.5
92.66
4-1(S)
14.60
100.00
100.00
99.4
100.00
4-2(S)
8.80
100.00
100.00
98.2
100.00
4-3(S)
12.60
91.67
100.00
96.2
100.00
4-4(S)
14.60
100.00
100.00
98.5
100.00
5-1(L)
69.50
74.59
73.14
85.2
55.20
5-2(L)
79.60
93.93
90.57
90.8
61.58
6-1(M)
33.60
49.40
57.40
88.5
58.50
6-2(S)
15.10
100.00
100.00
98.5
100.00
6-3(M)
23.00
91.48
92.43
91.4
90.78
7-1(S)
7.00
95.00
100.00
97.1
100.00
7-2(S)
7.00
85.00
85.71
92.4
93.63
7-3(S)
8.00
93.33
92.71
95.8
97.96
8-1(M)
40.00
89.00
100.00
94.4
96.28
8-2(S)
16.30
68.50
38.50
93.3
91.76
9-1(1)
97.00
75.00
75.00
85.9
35,90
9-2(L)
84.90
70.00
83,33
86.1
69,10
9-3(L)
123.10
60.00
71.43
85.8
61.80
10-1(L)
130.10
92.30
87.71
92.4
82.74
10-2(L)
132.10
77.60
78.00
85.9
72.90
Short
11.76
92.22
90.84
97.3
98.12
Medium
28.58
77.79
81.84
90.5
79.14
Long 94.30
83.00
84.52
88.3
71.03
Overall 49.26
84.68
85.91
90.2
82.29
Table 10: Task performance comparison for S-GPT4, D-GPT4 with RAG
on the LLM, though the action is reasonable to some extent. In contrast, DGAP-GPTH succeeded by leveraging discriminator feedback and re-planning through the LLM, which selected alternative equally reasonable actions to adaptively execute comprehensive sequence. This included iden- tifying, transporting, and arranging multiple items to fulfill all task requirements, emphasizing the importance of integrating robust feedback mechanisms and adaptive planning for complex, multi- step tasks, as shown in Figure T2
Enhanced Attention The discriminator's of additional score labels enhances attention rel- evant examples and prior actions, improving environmental interaction understanding and directly boosting planning quality without relying on re-planning. This is evident in Task25. As shown in Figure 13 where DGAP-generated actions are efficiently aligned with task objectives, focusing on the electric buzzer early and following logical path to complete the circuit with minimal redun- dancy. In contrast, the other sequence exhibits inefficiencies, including unnecessary connections and delays, highlighting the limitations of planning without adequate score attention. This comparison underscores the discriminator's s role in enabling precise and efficient task execution.
28

---

Published as a conference paper at ICLR 2025
Your task to find a(n) plant. First, focus on the thing. Then, move it to the yellow box in the bedroom
-1. You move to the Sreenhouse(Scoree.
-2. focus on flower pot (Score: 5) >replan!
pick up flower pot I(Score:9) You move the flower pot the inventory.
Disorimiinato
You move to the bedroom. (Score:7)
In bedroom, move flower pot yellow box(Score:7) ->> You move the flower pot to the yellow box.
DGAP-GPT4
greenhouse, pick up flower pot 6(Score:7) You move the flower
pot 6 to the inventory.
SwittSage-GPT4
6. Repeat the above times.
In bedroom, move flower pot to ycllow box(Score:5)>ropla)
1. look around
7. Repeat the above times.
2. teleport grennhouse
3. focus on flower pot Failure
18. You move to the bedroom (Score::)
19. In bedroom, move flower to yellow box(Score:8) -.>> You move the flower pot to yellow box
Figure 12: The visualization of the task8
Your task is turn on the electric buzzer. First, focus on the electric buzzer. Then, ereate electrical circuit that powers it on.
1. look around
SwiftSage-GPT4
1. look around
DGAP-GPT4
2. teleport workshop
2. teleport to workshop
3. focus on electric buzzer, which is necessury to proceed with
3. focus on electric buzzer to ensure all subsequent actions
task turning
are dirceted towards activating
4. focus electric buzzer
on electric
5. connect anode of battery to terminal of black wire
5. battery anode black wire terminal
6. connect battery anode to black wire terminal
connect batlery calhode to orunge wire terminal
7. connect terminal 2 of black wire to anode of electric buzzer
7. connect battery eathode to terminal in yellow wire
8. connect black wire tenminal to anode electric buzzer
8. connect black wire terminal to cathode in electric
9. connect battery cathode to red wire terminal
buzzer
10. connect terminal
wire cathode of electric buzzer
9. connect orange wire terminnl 2 to anode in electric buzzer
11. connect wire terminol cathode electric buzzer Wait
10. connect terminal yellow wire to anode in buzzer
Failure, circuit(color mismutch)
Figure 13: The visualization of the task25
29


Reviews:
[{"review_id": "rnw6J4DHbL", "reviewer": "Reviewer_Em5T", "summary": "The authors proposed an embodied agent framework to optimize LLM action plans at each step. They achieve so by employing a trained discriminator that learns a score function of alignment between LLM predicted actions and the optimal action, the author. They demonstrated the effectiveness of this policy by conducting experiments on benchmarks in ScienceWorld and VirtualHome, and outperformed previous methods.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "-The proposed framework is effective that it achieves superior results in the benchmarks studied.\n\n-The experiments and ablation studies in this study is thorough and the baselines compared to are extensive.\n\n-The presentation of this paper is good, the paragraphs are well written and easy to follow", "weaknesses": "-The novelty of this proposed framework is unclear to me. What is the fundamental differences of the motivation and the framework between this work and previous language grounding work (e.g. Saycan)?\n\n-The generalizability of the proposed method. If I understand correctly, the method designed does not involve fine-tuning a LLM rather it use a discriminator to capture the domain knowledge of the embodied agent. How do you see this discriminator generalize to more complicated scenarios as the granularity of visual/physical environment\nis far beyond than the text modality can capture. \n\n-The choice of benchmark. I respect your choice of experiment benchmarks. However, there are many embodied agent planning frameworks on benchmarks like ALFRED. If you cannot include them as baselines, please discuss the differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2dAyB6Gm6j", "reviewer": "Reviewer_SJiW", "summary": "The paper introduces a novel framework named Discriminator-Guided Action Optimization (DGAP), which aims to improve the performance of Large Language Models (LLMs) in embodied planning tasks. By leveraging a small set of demonstrations to train a discriminator, DGAP facilitates the optimization of LLM-generated action plans through step-wise signals, leading to better alignment with optimal actions. The proposed method is tested on challenging benchmarks like ScienceWorld and VirtualHome, demonstrating superior performance and efficiency compared to existing methods.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The integration of a discriminator to guide LLMs in generating higher-quality action plans is a creative and promising approach.\n2. The paper establishes a connection between DGAP and critic-regularized optimization in reinforcement learning, providing a solid theoretical foundation for the method.\n3. Extensive experiments are conducted on well-known benchmarks, showcasing the practical benefits of DGAP over prior methods.\n4. The paper is well-written, with clear explanations and logical organization, making it easy to follow the technical details and experimental results.", "weaknesses": "1. While the paper demonstrates the effectiveness of DGAP in specific environments, it would be beneficial to explore its scalability to more complex and diverse scenarios. Future work could investigate how the method performs in real-world settings with higher-dimensional state spaces.\n2. The reliance on a limited set of demonstrations might limit the generalizability of the model. It would be valuable to analyze how the performance changes with varying amounts of demonstration data and whether the method can adapt to unseen tasks effectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BD7T4iq35p", "reviewer": "Reviewer_ESa8", "summary": "The paper proposes a framework named Discriminator-Guided Action Optimization (DGAP), which combines the long-term reasoning capabilities of large language models (LLMs) with task-specific grounding under guidance. The authors introduce a simple discriminator learned with a limited number of demonstrations and. validate the effectiveness of this approach through experiments. The paper also provides a detailed theoretical explanation of the relationship between this method and critic-regularized optimization in RL.", "questions": "1. Could the authors consider experimenting with datasets like ALFRED? Due to its widespread use, ALFRED includes both LLM-based and RL-based methods, providing a valuable basis for further comparison and analysis.\n2. According to my understanding, the authors implicitly store task-specific knowledge in the discriminator through training. It would be beneficial to compare this approach with explicitly stored, search-based methods to better demonstrate the discriminator's advantages.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper employs a score function to quantitatively assess the planning effectiveness of the LLM, making it more reasonable compared to previous works.\n2. The paper provides a detailed theoretical derivation to model the problem and establish its relationship with critic-regularized optimization.\n3. The paper compares various types of planners, including reasoning-based and search-based approaches.", "weaknesses": "1. The experiments in the paper use the ScienceWorld and VirtualHouse simulators. However, it lacks the inclusion of widely used simulators for embodied planning tasks, such as ALFRED (https://arxiv.org/abs/1912.01734).\n2. The proposed method does not achieve optimal performance on short-sequence tasks; its advantages are primarily evident in long-sequence tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yl9o8AHdDR", "reviewer": "Reviewer_f2Sa", "summary": "This paper proposes an LLM action plan optimization framework based on discriminators to enhance the policy performance in long-term tasks with high generalization ability, where a small number of demonstrations is required to guide the optimization process. This framework first learns a score function via the discriminator with limited set of demonstrations, where the LLM is prompt to generate actions to maximize the score for optimization. The experimental results demonstrate the effectiveness of the proposed method.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The topic of LLM-based embodied planning is of great interests in embodied AI especially for general-purpose robots.\n+ The theoretical formulation to link the proposed method and critic-regularized optimization in RL brought some insights on embodied planning optimization\n+ The performance shows the method outperforms the baselines by a large margin.", "weaknesses": "- As introduced in the Introduction Section, the proposed framework aims to bring the long-term reasoning ability from LLMs with task grounding without harming the generalization ability. The discriminator is trained on limited collected data compared with the pre-trained LLMs, and I am not sure whether the discriminator can be generalized well. More proofs are required.\n\n- The groundtruth score for offline data in data collection is evaluated by a sentence embedding model. The quality of the sentence embedding model might significantly influence the performance of the groundtruth, where I doubt the noise of groundtruth will affect the model performance.\n\n- The writing needs to be improved. The proposed method contains a lot of techniques. I think the most important contribution, which I guess might be the discriminator training and usage, should be clearly emphasized.\n\n- Some qualitative results such as the planning results should be visualized to give some more intuition of the benefits brought by the proposed method.\n\n- More analysis to the experimental results especially the performance differences between short and long sequences should be discussed, as the generalization ability on different tasks (short and long-horizon) is both important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)