Grounding Robot Policies with Visuomotor Language Guidance

Authors: Arthur Bucker, Pablo Agustin Ortega-Kral, Jonathan Francis, Jean Oh
Primary area: applications to robotics, autonomy, planning
Keywords: Foundation Models for Robotics, Policy-adaptation, Self-guidance

Abstract:
Recent advances in the fields of natural language processing and computer vision have shown great potential in understanding the underlying dynamics of the world from large-scale internet data. However, translating this knowledge into robotic systems remains an open challenge, given the scarcity of human-robot interactions and the lack of large-scale datasets of real-world robotic data. Previous robot learning approaches such as behavior cloning and reinforcement learning have shown great capabilities in learning robotic skills from human demonstrations or from scratch in specific environments. However, these approaches often require task-specific demonstrations or designing complex simulation environments, which limits the development of generalizable and robust policies for new settings. Aiming to address these limitations, we propose an agent-based framework for grounding robot policies to the current context, considering the constraints of a current robot and its environment using visuomotor-grounded language guidance. The proposed framework is composed of a set of conversational agents designed for specific roles—namely, high-level advisor, visual grounding, monitoring, and robotic agents. Given a base policy, the agents collectively generate guidance at run time to shift the action distribution of the base policy towards more desirable future states. We demonstrate that our approach can effectively guide manipulation policies to achieve significantly higher success rates both in simulation and in real-world experiments without the need for additional human demonstrations or extensive exploration. Project videos at https://sites.google.com/view/motorcortex/home.

Decision: reject
Reviewer scores: 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729658708786

thread_type:
official_review

<thread_object>
creator: reviewer_eydk

<content>
summary:
This paper proposes an agent-based framework for grounding robotic policies in the current context by taking into account the constraints of both the robot and its environment, using visuomotor-grounded language guidance. The framework consists of a set of VLM-driven agents, each designed for specific roles, with the ultimate goal of improving existing policies.

strengths:
+The paper proposed an agentic robot policy grounding framework that can self improve by generating the action-scoring guidance functions to update the action distribution of a base policy in an online manner. 
+Their experiments, conducted in both simulated and real-world robot environments, demonstrate significant performance improvements in adapting existing policies, as well as showcasing zero-shot capabilities.

weaknesses:
- The main contribution of this work appears to be the guidance procedure, which stems from its ability to detect failures. Much of the pipeline focuses on object search using Vision-Language Models (VLM) to guide or refine the action distribution. However, the question arises as to how this approach compares to methods like AHA and RACER, which are instruction-tuned VLMs for failure detection and reasoning. In particular, AHA demonstrates that failure reasoning can improve base policies through a similar guidance mechanism.

- The paper claims that their approach is generalizable across various base policy classes, tasks, and environments. However, in the simulation experiments, evaluations were only conducted on a test set within the same domain distribution as the training data. No evidence was provided to demonstrate generalizability to different tasks or environmental perturbations.

- The authors also suggest that their approach can be used as a zero-shot policy to solve tasks without demonstrations. However, they did not provide any comparisons with other zero-shot methods, such as Manipulate-Anything, VoxPoser, or Code-as-Policies, which would strengthen their claim.

- In Figure 2, the example shows the need for granular object search due to the cluttered environment and large scene. However, this level of detail seems unnecessary for the real-world experiments presented in the paper. It would be beneficial to include more real-world experiments in complex, cluttered environments, where granular object search would be more impactful and better showcase the framework's strengths.

-Duan, Jiafei, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. "AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation." arXiv preprint arXiv:2410.00371 (2024).

-Dai, Yinpei, Jayjun Lee, Nima Fazeli, and Joyce Chai. "RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning." arXiv preprint arXiv:2409.14674 (2024).

-Duan, Jiafei, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. "Manipulate-anything: Automating real-world robots using vision-language models." arXiv preprint arXiv:2406.18915 (2024).

-Huang, Wenlong, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. "Voxposer: Composable 3d value maps for robotic manipulation with language models." arXiv preprint arXiv:2307.05973 (2023).

-Liang, Jacky, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. "Code as policies: Language model programs for embodied control." In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023.

questions:
- The paper could benefit from a clearer explanation of the system through the figures. Providing more detailed descriptions, especially in figures like Figure 2, would help readers better understand the design principles and flow of the system.

- Additionally, the definitions of "1% and 10% guidance" are unclear and need further clarification. Explaining how these percentages are determined and applied in the system would provide valuable context for understanding their role in guiding the base policy.

-Why is ACT3D with 1% guidance outperforms the 10% guidance variant, it seems counterintuitive.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730261052938

thread_type:
official_review

<thread_object>
creator: reviewer_itkp

<content>
summary:
The authors propose a method to guide robot policy with the guidance of several Vision Language Models (VLM). Specifically, the authors use several VLMs to compute scores of actions during execution and use the score to improve/correct the policy. The authors evaluate their approach on the RLBench and real world tasks and show improved performance.

strengths:
1. The high-level idea of this paper is interesting. Extracting VLM's knowledge to guide low level behavior is an interesting direction.
2. In the experiments, the authors evaluate their method on several realistic tasks and demonstrate the effectiveness of applying VLM guidance.

weaknesses:
Major Weaknesses: Clarity of Presentation

The main weakness of this paper is the lack of clarity in presenting its concepts and methods. Many key ideas are not explained in sufficient detail, which makes it difficult to follow the proposed approach.

1. In Figure 2, numerous arrows connect concepts like Robotic Agent, Advisory Agent, and Grounding Agent, yet these elements are absent from Algorithm 1. Without these agents and connections represented in the algorithm, understanding the overall inference procedure becomes challenging. A clearer, more detailed explanation of the full inference process would be beneficial.

2. In Algorithm 1, it is unclear how to implement the "Generate_Guidance" and "Infer future state" steps in Line 9.

3. It appears that the authors sample action $a$ from a product distribution $\pi (a|o) * G(a|o, ...)$. This approach is challenging, particularly with complex probabilistic distributions. What are the parameterizations of $\pi$ and $G$? In experiments, it looks like $\pi(a|o)$ is primarily a diffusion-based policy. How is the computed score then used to guide diffusion sampling? Additionally, is $G_{\pi_t}$ or $G(a|o, ...)$ represented as a vector?

4. How is "iterative improvement" achieved, as referenced in Line 300 and Figure 4?

5. Figures 3 and 5 are difficult to interpret. The lack of clarity around score representation (as mentioned in point 3) may contribute to the difficulty in understanding these figures.

questions:
See the weaknesses part. I would like to raise my score if the authors can resolve the clarity issues by more details and convince me the effectiveness of their approach.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730727247761

thread_type:
official_review

<thread_object>
creator: reviewer_kdxu

<content>
summary:
The paper proposes g-MotorCortex, an agentic robot policy grounding framework that can self improve by generating the action-scoring guidance functions to update the action distribution of a base policy in an online manner. The paper presents experiments in both simulated and real-world robot settings and shows significant performance improvements on both adapting existing policies and learning new skills from scratch.
The proposed method is shown to be robust against cluttered and unseen environments, thanks to multi-granular object search, which enables flexible visuomotor grounding.
The authors also open source the code, models, and system prompts.

strengths:
The paper is clearly written and well structured. It is easy to read and to follow through the contributions and components of the proposed method.
The experimental results are well presented as well and span across different axes of evaluations, which make it easier to appreciate the proposed method and contributions.
The results presented are interesting and significant to the field, and well demonstrate a nice integration of VLMs with robot control.

weaknesses:
The experimental results show performance of three algorithms: Act3D, 3D Diffuser Actor, and a RandomPolicy. The reasons behind this choice is not clear and it would be helpful to understand what's behind this, and whether other types of algorithms would also be suitable and how their performance is expected to be.
The real robot evaluations is interesting, although the task defined is not interactive at a physical level, being only a reaching task. The results are interesting and valuable, however real-robot challenges arise often when interaction with environments exist.

questions:
- Can you clarify the choice of algorithms chosen? Are results available from other algorithms too? Are there specific reasons behind this choice?
- What are the limitations of the learning from scratch? Can the system effectively incorporate preferences/demonstrations?
- Can you elaborate on why the self-improvement experiments plateau?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733209685611

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response and Rebuttal Summary

comment:
We thank the reviewers for the valuable feedback and discussion. For clarity, we include a summary of the changes made to the revision.

- **Clarity and Presentation:** We revised the paper to improve clarity, including updated figures (e.g., **Figure 2**) to better illustrate the guidance procedure and agent interactions. **Algorithm 1** now aligns more clearly with the described processes, and additional implementation details are provided in **Appendix A.1**. These changes better elucidate how our policy grounding framework can improve existing policies (see **Table 1**) and guide them to success even in out-of-distribution cases (**Figure 4**). Furthermore, it clarifies how the group of self-orchestrated agents leverages the multi-granular object search proposed to provide meaningful guidance even in cluttered environments (**Figure 3** and **Section 3.3**).

- **Baseline Comparisons:** We addressed concerns regarding the baselines and provided justifications for the chosen algorithms (Act3D, 3D Diffuser Actor, and RandomPolicy) as diverse and representative use cases for our framework. While other methods like REFLECT and dedicated zero-shot approaches were acknowledged, we clarified their limited relevance to the overall scope of our approach. Our response helps better situate our framework among prior works, showcasing two major key differences: flexible and actionable policy correction, as well as zero-shot capabilities tailored to the base policies’ action space (**Figure 4**). 

-  **Iterative Improvement Clarification:** We highlight that the framework incrementally enhances task success rates, even in challenging scenarios with unreliable (untrained) base policies, by leveraging feedback from the perceived robot’s behavior. In comments to reviewers, we further clarified our system's limitations originating in its off-the-shelf components and elucidated the reason behind learning plateaus in our iterative improvement experiments.

We once again thank the reviewers for their insightful comments. We believe that our responses addressed in a meaningful way the raised concerns and clarified the strengths of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a framework leveraging Visuomotor Language Models (VLMs) to guide robot manipulation policies. The proposed system demonstrates promising performance in adapting policies and enabling zero-shot task execution. While reviewers recognized the novelty and potential impact of this work, significant concerns were raised regarding clarity, presentation, and experimental evaluation.

Strengths:

- The paper addresses a compelling challenge at the intersection of robotics and foundation models, presenting a novel framework for dynamic policy guidance.

- Experiments demonstrate promising results in both simulation and real-world environments.

- The modular, agent-based design enhances system flexibility and usability across diverse tasks and policies.

Weaknesses:

- Reviewers consistently pointed to unclear explanations of key concepts (e.g., guidance generation, inference procedures). Updated figures and algorithmic clarifications partially addressed these concerns.

- The citation and references to prior work on was seen as insufficient, particularly given the claims of generalizability and zero-shot learning. I would like to encourage the authors to have a thorough review of recent works on robotic control and planning methods using VLMs, including but not limited to the papers mentioned by the reviewers.

- Real-world experiments are limited to simpler tasks, lacking tests in more cluttered and challenging settings where the method's strengths would be better highlighted.

Although I agree that the paper could be significantly improved with a more comprehensive discussion of prior work on VLM-based robotic control, I would like to point out that Reviewer EYdK's request for comparison to concurrent work such as AHA and RACER is unnecessary and unfair. Notably, AHA is a concurrent submission to ICLR 2025 ([Submission 506](https://openreview.net/forum?id=JVkdSi7Ekg)) and was first arXived on October 1st, 2024, making such a request impractical and against conference guidelines. Therefore, this request was not considered in the final decision on the paper.

While the paper's novelty and potential are apparent, the concerns regarding clarity and insufficient discussion on prior work weaken its overall contribution in its current form. Given these considerations, I recommend rejection for ICLR 2025, with encouragement to improve clarity, expand experimental validation, and engage more comprehensively with related work in future submissions.