VLP: Vision-Language Preference Learning for Embodied Manipulation

Authors: Runze Liu, Chenjia Bai, Jiafei Lyu, Shengjie Sun, Yali Du, Xiu Li
Primary area: reinforcement learning
Keywords: reinforcement learning, preference-based reinforcement learning, vision-language alignment, offline reinforcement learning

Abstract:
Reward engineering is one of the key challenges in Reinforcement Learning (RL). Preference-based RL effectively addresses this issue by learning from human feedback. However, it is both time-consuming and expensive to collect human preference labels. In this paper, we propose a novel Vision-Language Preference learning framework, named VLP, which learns a vision-language preference model to provide preference feedback for embodied manipulation tasks. To achieve this, we define three types of language-conditioned preferences and construct a vision-language preference dataset, which contains versatile implicit preference orders without human annotations. The preference model learns to extract language-related features, and then serves as a preference annotator in various downstream tasks. The policy can be learned according to the annotated preferences via reward learning or direct policy optimization. Extensive empirical results on simulated embodied manipulation tasks demonstrate that our method provides accurate preferences and generalizes to unseen tasks and unseen language, outperforming the baselines by a large margin. The code and videos of our method are available on the website: https://VLPref.github.io.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730450294793

thread_type:
official_review

<thread_object>
creator: reviewer_r4zt

<content>
summary:
This paper introduces VLP, a vision-language preference learning framework, where the preference model is designed to generalize across unseen tasks. The preference model is based on an open-sourced CLIP model augmented with a trainable component. The model is trained using different types of preferences, categorized as ITP, ILP, and IVP. The experiments on the Meta-World benchmark support the paper's claims.

strengths:
- The division of video-language preference types is novel and thoughtfully structured. ITP reflects traditional preferences, while IVP appears to enhance the model’s instruction-following capabilities. ILP seems to serve as a regularizer, adding robustness to the model. This categorization is well-conceived.
- The writing is clear, and the graphical illustrations effectively convey the content, enhancing overall readability.
- Both theoretical analysis and empirical findings support the framework.

weaknesses:
- A primary concern is the rationale behind the train-test task split in Meta-World. While the experimental results favor the proposed framework, it is unclear if the task split was specifically selected for favorable outcomes. Using Meta-World's ML45 benchmark, which provides a pre-defined split for comparability across works, could enhance the reproducibility and rigor of the results. Clarifying this point would strengthen the paper, and I would be inclined to raise my score if this concern is addressed, as the rest of the experimental design is robust.

questions:
- The authors claim novelty in the architecture (line 071), yet it is not immediately clear what sets it apart, and this assertion seems somewhat overstated. Could the authors clarify the specific architectural innovations that distinguish this approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730552329235

thread_type:
official_review

<thread_object>
creator: reviewer_nsgj

<content>
summary:
This paper introduces VLP (Vision-Language Preference learning), a framework for learning general preference feedback for embodied manipulation tasks. The key contribution is a vision-language preference model that provides feedback by aligning video and language modalities. The paper presents extensive empirical evaluation, demonstrating strong performance and generalization capabilities to unseen tasks and language instructions.

strengths:
1. The paper presents an effective framework that combines vision-language alignment with preference learning for robotic manipulation tasks. The experimental results show consistent improvements over VLM-based approaches across multiple tasks and demonstrate good generalization performance.
2. The paper is well-structured and easy to follow, presenting its ideas clearly.

weaknesses:
1. The evaluation is limited to relatively simple Meta-World tasks, without testing on more complex task domains (e.g., MANISKILL2 [1] and MyoSuite [2]).
2. The paper lacks comparison with human preference labels, which would validate the quality of the generated preferences against human intent.
3. The theoretical analysis assumes access to all possible segments, weakening its practical implications.
4. (minor) The paper does not report the performance of scripted policies, which would help establish an upper bound for task performance and validate the quality of collected expert demonstrations.

[1] Gu, Jiayuan, et al. "Maniskill2: A unified benchmark for generalizable manipulation skills." *arXiv preprint arXiv:2302.04659* (2023).

[2] Caggiano, Vittorio, et al. "MyoSuite--A contact-rich simulation suite for musculoskeletal motor control." *arXiv preprint arXiv:2205.13600* (2022).

questions:
1. How does the computational cost of training VLP compare to other approaches like R3M or VIP?
2. How sensitive is the model to the quality and diversity of language instructions? Is there a significant performance drop when using instructions generated by a less capable model than GPT-4V?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722647994

thread_type:
official_review

<thread_object>
creator: reviewer_lwso

<content>
summary:
This paper proposes a novel video-based, vision-language-interleaved preference learning method for robotic control, named VLP. It defines three types of language-conditioned preferences: ITP, ILP, and IVP. The authors introduce a novel vision-language preference alignment framework that includes a learnable cross-modal transformer model to fuse video tokens and language tokens. They constructed a vision-language preference dataset with clear intra-task preference relations, MTVLP, containing 4.8K videos. Experimental results demonstrate the superiority of VLP compared to other RLHF methods with scripted labels or other vision-language rewards. Additionally, empirical evidence suggests that ILP and IVP, alongside the traditional ITP, contribute to improved performance, and that the 4.8K videos are both necessary and sufficient to achieve over 97% ITP accuracy.

strengths:
* This paper presents strong empirical evidence and extensive experiments supporting the proposed approach. 
* The novel cross-modal architecture effectively fuses video and language through learnable parameters to compute preferences. 
* Furthermore, the introduction of language-conditioned preferences, namely Intra-Task Preference (ITP), Inter-Language Preference (ILP), and Inter-Video Preference (IVP), is a notable contribution that enhances the model's adaptability across different scenarios.

weaknesses:
* The theoretical claim seems to lack clear logical reasoning to justify the assertion that "the proposed preference model can be considered as parameterized negative regret that approximates the true negative regret of the whole segment". Although Eq. (10) and Eq. (11) have similar shapes, that does not mean that one approximates the other.
* I'm concerned that the simplicity of ILP and IVP definitions may limit VLP's generalizability. The preference labels defined in Table 1 overlook potential similarities between videos or language instructions across different tasks: They can assign a negative signal even if two videos from different tasks are similar (or in the case where the video and language from different tasks are semantically related) This approach may only work effectively within a carefully selected task distribution, potentially weakening the paper's claims of generalizability.

questions:
* Comparing this work with RoboCLIP (Sontakke et al., 2023) may provide valuable context. Baselines in this paper lack video input, so VLP’s advantage might come from its temporal reasoning. While VLP is compute-efficient, RoboCLIP is zero-shot. Demonstrating the cross-modal architecture’s distinct benefits would strengthen the claims.
* Regarding the second weakness: (1) How are video pairs in MTVLP constructed for ITP, ILP, and IVP regarding optimality levels? Are all combinations (e.g., expert, medium, random) considered? (2) Could you provide more examples of how medium-optimality is defined across the 50 tasks? Do any tasks share similar initial subtasks?
* Writing clarification suggestions: (1) It would be great if Table 1 is accompanied by v_i^j and l^k notations. (2) In several places, “language” is used to mean "language instructions," which might cause confusion. For instance, "unseen language" might imply a different spoken language rather than new instructions in English.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730728120502

thread_type:
official_review

<thread_object>
creator: reviewer_arlj

<content>
summary:
This work we propose a vision-language preference (VLP) learning that uses a vision-language model to provide preference feedback. It defines three types of language-conditioned preferences and contributes a vision-language preference dataset. The framework is evaluated on the Meta-World Benchmark.

strengths:
1. This work proposes three forms of language-conditioned preferences: ITP, ILP and IVP. 

2. This work proposes a framework vision-language preference learning with theoretic analysis of its behavior.

3. Experiments are well organized to answer four key questions. 

4. Experiments show that the proposed VLP leads to better performance than other state-of-the-art baselines.

weaknesses:
1. In the experiments, only a single benchmark, Meta-World, is used. 
- This is limited to show the generality of the proposed preference learning framework.
- Related works in section 4 have tested on several different environments. 

2. Only five tasks in the Meta-work are evaluated among 50 tasks.  
- This set of test tasks is not so challenging compared to 45 training tasks. 

3. Why are RL-VLM-F and CriticGPT not compared?  

4. The dataset construction itself may not be a notable contribution. 
- The trajectory sampling pipeline is rather simple, so its diversity may be unclear. 
- The dataset size is not big. 

5. Figures can be improved. 
- Fig. 2 is somewhat standard and conveys only limited value for the novelty. 
- In Fig.3, the attentions are not clearly seen.

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733208460803

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
We are sincerely grateful to all reviewers for their careful evaluation and constructive feedback on our paper. We appreciate reviewers recognizing the contributions of our work, including the novelty of the vision-language preference [ARLJ, LWSo, R4Zt], the writing of the paper [ARLJ, NSgJ, R4Zt], and the the strong empirical results [ARLJ, LWSo, NSgJ].

In response to reviewer comments, we have added new experiments and results to strengthen the paper in the PDF, and all modifications are highlighted in **orange**. We provide global response for the following questions:

**More benchmark results [ARLJ, NSgJ]**

> We acknowledge the importance of evaluating our method on more complex task domains and conduct further experiments on ManiSkill2 [1]. We leverage *MoveBucket-v1*, *OpenCabinetDrawer-v1*, *PegInsertionSide-v0*, *PickCube-v0*, *PickSingleEGAD-v0*, *PlugCharger-v0*, *StackCube-v0*, and *TurnFaucet-v0* as training tasks and evaluate the  of VLP on *LiftCube-v0*, *OpenCabinetDoor-v1*, *PushChair-v1* tasks. The following table summarize the average VLP label accuracy on the three test tasks compared to scripted labels and the results demonstrate the strong generalization capabilities of VLP. To provide a comprehensive evaluation of VLP on ManiSkill2 benchmark, we will conduct further experiments and add the results in the future version.
> 
> | Task | VLP Accuracy |
> | - | - |
> | LiftCube-v0 | 100.0 |
> | OpenCabinetDoor-v1 | 100.0 |
> | PushChair-v1 | 93.8 |
> | Average | 97.9 |

**Train-test task split [R4Zt]**

> 1) We conduct experiments on the standard ML45 benchmark, training the vision-language preference model on its training tasks and evaluating on its test tasks. The results shown below demonstrate the strong generalization capability of our method on unseen tasks in ML45. This reinforces the robustness and adaptability of our framework regardless of task split. We have updated the manuscript and included these results and discussions in Appendix E.
> 
> | Task | VLP Accuracy |
> | - | - |
> | Bin Picking | 95.0 |
> | Box Close | 90.0 |
> | Door Lock | 100.0 |
> | Door Unlock | 100.0 |
> | Hand Insert | 100.0 |
> | Average | 97.0 |
> 
> 2) We believe that there is no universally accepted "gold standard" for how to split tasks in multi-task setting and the train-test split is just a design choice instead of a standardized procedure.  CriticGPT [2], which is prior preference-based RL method evaluated in similar multi-task setting, also uses the Meta-World benchmark and manually selects its own train-test task split. Lift3D [3] also select 15 custom tasks for experimental evaluation. Additionally, the baselines are evaluated using the same split for fair comparison.

We believe these new results and clarification will help address the concerns about the evaluation benchmark and train-test task split. We thank the reviewers' time and feedback in improving the quality of our work and we hope the revisions further highlight the contributions made. Please let us know if any clarification or additional experiments would further strengthen the paper. We would be happy to incorporate all these suggestions in the final version.

**References**

[1] Maniskill2: A unified benchmark for generalizable manipulation skills.

[2] Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models.

[3] Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a video-based, vision-language-interleaved preference learning approach for robotic control. Reviewers raised concerns, particularly regarding the limited novelty in the architecture and the weakness of the experimental results. While the paper shows promise, it is not recommended for acceptance in its current form. The authors are encouraged to address the reviewers' feedback and refine the work for submission to other venues.