Discriminator-Guided Embodied Planning for LLM Agent

Authors: Haofu Qian, Chenjia Bai, Jiatao Zhang, Fei Wu, Wei Song, Xuelong Li
Primary area: reinforcement learning
Keywords: LLM Agent, Embodied Planning, Discriminator, Critic-Regularized Optimization

Abstract:
Large Language Models (LLMs) have showcased remarkable reasoning capabilities in various domains, yet face challenges in complex embodied tasks due to the need for a coherent long-term policy and context-sensitive environmental understanding. Previous work performed LLM refinement relying on outcome-supervised feedback, which can be costly and ineffective. In this work, we introduce a novel framework, Discriminator-Guided Action Optimization (DGAP), for facilitating the optimization of LLM action plans via step-wise signals. Specifically, we employ a limited set of demonstrations to enable the discriminator to learn a score function, which assesses the alignment between LLM-generated actions and the underlying optimal ones at every step. Based on the discriminator, LLMs are prompted to generate actions that maximize the score, utilizing historical action-score pair trajectories as guidance. Under mild conditions, DGAP resembles critic-regularized optimization and has been demonstrated to achieve a stronger policy than the LLM planner. In experiments across different LLMs (GPT-4, Llama3-70B) in ScienceWorld and VirtualHome, our method achieves superior performance and better efficiency than previous methods.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729671844902

thread_type:
official_review

<thread_object>
creator: reviewer_f2sa

<content>
summary:
This paper proposes an LLM action plan optimization framework based on discriminators to enhance the policy performance in long-term tasks with high generalization ability, where a small number of demonstrations is required to guide the optimization process. This framework first learns a score function via the discriminator with limited set of demonstrations, where the LLM is prompt to generate actions to maximize the score for optimization. The experimental results demonstrate the effectiveness of the proposed method.

strengths:
+ The topic of LLM-based embodied planning is of great interests in embodied AI especially for general-purpose robots.
+ The theoretical formulation to link the proposed method and critic-regularized optimization in RL brought some insights on embodied planning optimization
+ The performance shows the method outperforms the baselines by a large margin.

weaknesses:
- As introduced in the Introduction Section, the proposed framework aims to bring the long-term reasoning ability from LLMs with task grounding without harming the generalization ability. The discriminator is trained on limited collected data compared with the pre-trained LLMs, and I am not sure whether the discriminator can be generalized well. More proofs are required.

- The groundtruth score for offline data in data collection is evaluated by a sentence embedding model. The quality of the sentence embedding model might significantly influence the performance of the groundtruth, where I doubt the noise of groundtruth will affect the model performance.

- The writing needs to be improved. The proposed method contains a lot of techniques. I think the most important contribution, which I guess might be the discriminator training and usage, should be clearly emphasized.

- Some qualitative results such as the planning results should be visualized to give some more intuition of the benefits brought by the proposed method.

- More analysis to the experimental results especially the performance differences between short and long sequences should be discussed, as the generalization ability on different tasks (short and long-horizon) is both important.

questions:
See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605583410

thread_type:
official_review

<thread_object>
creator: reviewer_esa8

<content>
summary:
The paper proposes a framework named Discriminator-Guided Action Optimization (DGAP), which combines the long-term reasoning capabilities of large language models (LLMs) with task-specific grounding under guidance. The authors introduce a simple discriminator learned with a limited number of demonstrations and. validate the effectiveness of this approach through experiments. The paper also provides a detailed theoretical explanation of the relationship between this method and critic-regularized optimization in RL.

strengths:
1. The paper employs a score function to quantitatively assess the planning effectiveness of the LLM, making it more reasonable compared to previous works.
2. The paper provides a detailed theoretical derivation to model the problem and establish its relationship with critic-regularized optimization.
3. The paper compares various types of planners, including reasoning-based and search-based approaches.

weaknesses:
1. The experiments in the paper use the ScienceWorld and VirtualHouse simulators. However, it lacks the inclusion of widely used simulators for embodied planning tasks, such as ALFRED (https://arxiv.org/abs/1912.01734).
2. The proposed method does not achieve optimal performance on short-sequence tasks; its advantages are primarily evident in long-sequence tasks.

questions:
1. Could the authors consider experimenting with datasets like ALFRED? Due to its widespread use, ALFRED includes both LLM-based and RL-based methods, providing a valuable basis for further comparison and analysis.
2. According to my understanding, the authors implicitly store task-specific knowledge in the discriminator through training. It would be beneficial to compare this approach with explicitly stored, search-based methods to better demonstrate the discriminator's advantages.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638814645

thread_type:
official_review

<thread_object>
creator: reviewer_sjiw

<content>
summary:
The paper introduces a novel framework named Discriminator-Guided Action Optimization (DGAP), which aims to improve the performance of Large Language Models (LLMs) in embodied planning tasks. By leveraging a small set of demonstrations to train a discriminator, DGAP facilitates the optimization of LLM-generated action plans through step-wise signals, leading to better alignment with optimal actions. The proposed method is tested on challenging benchmarks like ScienceWorld and VirtualHome, demonstrating superior performance and efficiency compared to existing methods.

strengths:
1. The integration of a discriminator to guide LLMs in generating higher-quality action plans is a creative and promising approach.
2. The paper establishes a connection between DGAP and critic-regularized optimization in reinforcement learning, providing a solid theoretical foundation for the method.
3. Extensive experiments are conducted on well-known benchmarks, showcasing the practical benefits of DGAP over prior methods.
4. The paper is well-written, with clear explanations and logical organization, making it easy to follow the technical details and experimental results.

weaknesses:
1. While the paper demonstrates the effectiveness of DGAP in specific environments, it would be beneficial to explore its scalability to more complex and diverse scenarios. Future work could investigate how the method performs in real-world settings with higher-dimensional state spaces.
2. The reliance on a limited set of demonstrations might limit the generalizability of the model. It would be valuable to analyze how the performance changes with varying amounts of demonstration data and whether the method can adapt to unseen tasks effectively.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730937232843

thread_type:
official_review

<thread_object>
creator: reviewer_em5t

<content>
summary:
The authors proposed an embodied agent framework to optimize LLM action plans at each step. They achieve so by employing a trained discriminator that learns a score function of alignment between LLM predicted actions and the optimal action, the author. They demonstrated the effectiveness of this policy by conducting experiments on benchmarks in ScienceWorld and VirtualHome, and outperformed previous methods.

strengths:
-The proposed framework is effective that it achieves superior results in the benchmarks studied.

-The experiments and ablation studies in this study is thorough and the baselines compared to are extensive.

-The presentation of this paper is good, the paragraphs are well written and easy to follow

weaknesses:
-The novelty of this proposed framework is unclear to me. What is the fundamental differences of the motivation and the framework between this work and previous language grounding work (e.g. Saycan)?

-The generalizability of the proposed method. If I understand correctly, the method designed does not involve fine-tuning a LLM rather it use a discriminator to capture the domain knowledge of the embodied agent. How do you see this discriminator generalize to more complicated scenarios as the granularity of visual/physical environment
is far beyond than the text modality can capture. 

-The choice of benchmark. I respect your choice of experiment benchmarks. However, there are many embodied agent planning frameworks on benchmarks like ALFRED. If you cannot include them as baselines, please discuss the differences.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732310055026

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all the reviewers for their valuable feedback and insightful comments. In response to the reviewers' suggestions, we have conducted several complementary experiments to address their concerns. The results of these additional experiments are summarized below:

1. \[Adding new **evaluation analysis** of the discriminator]
2. \[Adding new experiments in **ALFRED**]
3. \[Constrcting and evaluating **extreme long tasks** in virtualhome]
4. \[Adding a new baseline of **search-based methods** with full demonstration data]
5. \[Ongoing: A new multimodal discriminator training]

According to the result, our method demonstrates better (or at least comparable) performance to the competitors in all the above settings. We also clarify some questions the reviewers mentioned, including but not limited to:


1. \[The novelty and difference of our method]: DGAP leverages discriminator feedback and critic optimization to improve task generalization and efficiency, offering process-level scoring and adaptability with limited data, setting it apart from methods like SayCan[9], LLMPlanner[10], and ReAd[1].
2. \[Generalization to Visual/Physical Domains]: The modular design of the discriminator facilitates its extension to multimodal tasks. Initial results on ALFRED indicate promising adaptability and competitive generalization in unseen scenarios.
3. \[Performance difference between short and long tasks]: While DGAP performs consistently across tasks of varying lengths, challenges remain in addressing complex temporal constraints, such as handling "wait" actions, which require more refined data construction.
4. \[Groundtruth Noise]: Despite noise in augmented data, DGAP demonstrates resilience, effectively utilizing score signals to achieve generalization.
5. \[Qualitative Analysis]: DGAP effectively enhances planning by leveraging process-level scoring to focus on past actions and environmental interactions, while also demonstrating the ability to discern conditions to re-planning.

We deeply appreciate the anonymous reviewers for their insightful feedback and constructive suggestions, which have substantially improved the quality of our research. Also, we hope our response can be carefully considered and welcome any useful suggestions.


**Reference**

[1] Zhang, Yang, et al. "Towards efficient llm grounding for embodied multi-agent collaboration." arXiv preprint arXiv:2405.14314 (2024).

[2] Liu, Songming, et al. "RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation." arXiv preprint arXiv:2410.07864 (2024).

[3] Cheang, Chi-Lam, et al. "Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation." arXiv preprint arXiv:2410.06158 (2024).

[4] Wu, Hongtao, et al. "Unleashing large-scale video generative pre-training for visual robot manipulation." arXiv preprint arXiv:2312.13139 (2023).

[5] Cubuk, Ekin D., et al. "Autoaugment: Learning augmentation strategies from data." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[6] Wu, Sen, et al. "On the generalization effects of linear transformations in data augmentation." International Conference on Machine Learning. PMLR, 2020.

[7] Raileanu, Roberta, et al. "Automatic data augmentation for generalization in reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 5402-5415.

[8] Pinneri, Cristina, et al. "Equivariant Data Augmentation for Generalization in Offline Reinforcement Learning." arXiv preprint arXiv:2309.07578 (2023).

[9] Brohan, Anthony, et al. "Do as i can, not as i say: Grounding language in robotic affordances." Conference on robot learning. PMLR, 2023.

[10] Song, Chan Hee, et al. "Llm-planner: Few-shot grounded planning for embodied agents with large language models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[11] Inoue, Yuki, and Hiroki Ohashi. "Prompter: Utilizing large language model prompting for a data efficient embodied instruction following." arXiv preprint arXiv:2211.03267 (2022).

[12] Choi, Jae-Woo, et al. "Lota-bench: Benchmarking language-oriented task planners for embodied agents." arXiv preprint arXiv:2402.08178 (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes a Discriminator-Guided Action Optimization (DGAP) to enhance embodied planning. All reviewers consistently recommended accepting this work. AC agrees that this work is interesting and deserves to be published on ICLR 2O25. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes in the final version.