Few-shot In-context Preference Learning using Large Language Models

Authors: Chao Yu, Hong Lu, Jiaxuan Gao, Qixin Tan, Xinting Yang, Yu Wang, Yi Wu, Eugene Vinitsky
Primary area: reinforcement learning
Keywords: Large Language Models, Preference-based RL, Reinforcement Learning from Human Feedback, Reward Design

Abstract:
Designing reward functions is a core component of reinforcement learning but can be challenging for truly complex behavior. Reinforcement Learning from Human Feedback (RLHF) has been used to alleviate this challenge by replacing a hand-coded reward function with a reward function learned from preferences. However, it can be exceedingly inefficient to learn these rewards as they are often learned tabula rasa. We investigate whether Large Language Models (LLMs) can reduce this query inefficiency by converting an iterative series of human preferences into code representing the rewards. We propose In-Context Preference Learning (ICPL), a method that uses the grounding of an LLM to accelerate learning reward functions from preferences. ICPL takes the environment context and task description, synthesizes a set of reward functions, and then repeatedly updates the reward functions using human feedback over videos of the resultant policies over a small number of trials. Using synthetic preferences, we demonstrate that ICPL is orders of magnitude more efficient than RLHF and is even competitive with methods that use ground-truth reward functions instead of preferences. Finally, we perform a series of human preference-learning trials and observe that ICPL extends beyond synthetic settings and can work effectively with humans-in-the-loop.

Decision: reject
Reviewer scores: 5, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1730414831001

thread_type:
official_review

<thread_object>
creator: reviewer_9tx3

<content>
summary:
This paper proposes a reward design method. It uses LLMs to generate reward functions to calculate the reward, and the prompt of the LLM is learned through human feedback of the policy rollouts and other historical information in the loop.

It replaces the implicit reward model in traditional RLHF with an LLM and its output reward function. This enhances the interoperability and capacity of the reward design.

strengths:
It replaces the implicit reward model in traditional RLHF with an LLM and its output reward function. This enhances the interoperability and capacity of the reward design.

weaknesses:
(1) ICPL involves human labor, but does not show any significant gain over Eureka, which doesn't require any human feedback.

(3) For challenging tasks, true human feedback does not work better than proxy human feedback. This undermines the necessity of involving humans.

(2) For challenging tasks, like humanoid jump task, ICPL does not have any solid comparisons with other baselines.

questions:
(1) Why is it necessary to use pair-wise human feedback (a good example and a bad example) if RTS is available? Why not just use all the reward functions with their RTS as prompt (maybe together with other information like reward trace, differences, etc) to generate reward functions?

(2) Could you please explain the counter-intuitive results in Table 2? It seems the more prompt components you remove (from w/o RT, to w/o RTD, to w/o RTDB), the better performance it gets (w/o RT wins 2 tasks, w/o RTD wins 3 tasks, and w/o RTDB wins 4 tasks), but adding all the components back, i.e., ICPL(Ours), it wins all the tasks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628996225

thread_type:
official_review

<thread_object>
creator: reviewer_u5zu

<content>
summary:
The paper proposes In-Context Preference Learning (ICPL), a method using LLMs for more efficient preference-based reinforcement learning. ICPL uses LLMs, such as GPT-4, to synthesize reward functions based on environment context and task descriptions. These generated functions are refined through human feedback iteratively. The approach shows significant efficiency gains, requiring fewer preference queries compared to traditional RLHF and achieving comparable or superior performance to methods using ground-truth reward functions. The experiments validate ICPL's performance across various simulated and real human-in-the-loop RL tasks, showcasing robustness in complex, subjective environments.

strengths:
* Demonstrates a substantial reduction in the number of human queries needed for preference-based learning which is sorely needed since human-in-the-loop approaches should ideally just require a handful of preferences.

* It's appreciated that the evaluations of the method is done both in synthetic data and with real humans.

* The paper is well written and the provided method is explained well. Even tho generating reward functions from LLMs is not novel, the way the iteratively make use of human preferences to update their prompt is.

weaknesses:
* As with all works that uses LLMs to generate reward functions from human feedback I question how well it will perform with more complex tasks which is one of the big reason for using human feedback.

* The synthetic experiment uses completely noiseless preferences while the standard in these kind of control environments are typical a noise of let's say 10%. What is the rationale for using noiseless preferences and what would be the effect of noisy preferences for your method? 

* While the authors uses B-Pref from Kimin et al for some reason they use only the PPO version even tho the repository is more associated with PEBBLE the SAC version. Why is SAC not used as well?

* 6 participants are very low for a study with humans. Still, it is better than some papers that run their method with just the authors feedback. It would be nice with some more information about the experiment like demographic data as well as discussing the limitation of a smaller sample size when it comes to generalizability.

Minor things:
* You introduce the same abbreviation on multiple occassions.

* To make the related work more complete, there is another paper using LLMs with preferences.
1. Holk, S., Marta, D., & Leite, I. (2024, March). PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning. In Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (pp. 259-268).

questions:
* What was the demographic data for the human provided feedback? 

* It seems like Eureka has very similar performance to ICPL, what would you say is the benefit of your method compared to Eureka? Eureka seems to have some constraints but it would be nice to show in experimentation or come up with a scenario where it would fail.

* It would be good to justify the length of the paper. For example, what sections do you believe require the additional space? You are of course free to use all the space but the readability of the paper could improve by making it more crisp.

* Why did you not use PEBBLE as a basline given that you made use of BPref? Also, did you consider any other baselines as there are more recent works [1,2,3] (To name a few)? It would be great if you discuss how you determined which baseline to use and if you considered any others.
1. Kim, C., Park, J., Shin, J., Lee, H., Abbeel, P., & Lee, K. (2023). Preference transformer: Modeling human preferences using transformers for rl. arXiv preprint arXiv:2303.00957.
2. Park, J., Seo, Y., Shin, J., Lee, H., Abbeel, P., & Lee, K. (2022). SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. arXiv preprint arXiv:2203.10050.
3. Marta, D., Holk, S., Pek, C., Tumova, J., & Leite, I. (2023, October). VARIQuery: VAE Segment-Based Active Learning for Query Selection in Preference-Based Reinforcement Learning. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 7878-7885). IEEE.

I am more than willing to up the score given reasonable answers to these points.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730671819959

thread_type:
official_review

<thread_object>
creator: reviewer_cq3w

<content>
summary:
This paper introduces ICPL, a method for iteratively improving reward functions for RL problems with human feedback. The method has LLMs generate reward functions specified by code, trains and executes these rewards, and then ranks the final trajectories with human feedback to then update the reward functions again.

------
After author response, my main concern (motivation above EUREKA) was clarified and my more minor concerns were addressed or clarified as well, so I have increased my rating and think the paper should be accepted. On balance I think it's a really interesting problem that's being tacked and the experiments (esp human experiments) are really interesting and compelling.

strengths:
The idea is generally really interesting and compelling. The idea of having LLMs generate an initial reward function and then iteratively repeat it is really interesting.

The human study was really compelling and thought out. It's really good that this was actually tried and not just assumed it would work with real human feedback.

Paper really well presented, ideas presented very clearly. Motivation clear and compelling.

weaknesses:
I am struggling to figure out what the compelling advantage is of this method over the baseline Eureka. As far as I understood reading the paper, Eureka operated from the same set of assumptions about the environment as did ICPL. And in the non-human experiment performed very similarly. In the related work it says that EUREKA requires humans to give feedback in text, whereas ICPL only requires ranked preferences. During the description in 5.2 it also says that sparse rewards are used to select the best candidate reward function. Does that mean that this is additional assumptions EUREKA needs. There was also not a comparison to EUREKA in the human study. Was that because it would not work without these other assumptions? I think it's possible I'm just misunderstanding here, so if authors could clarify this point it would really help me understand the paper and potentially improve my rating.

It's stated in the intro and conclusion that ICPL surpasses RLHF is efficiency, but RLHF is not mentioned anywhere in the experiments. Is this an experimental finding of the paper, or are authors just saying based on known findings about the efficiency of RLHF. Could a direct comparison be made in the first (non-human) experiments since you don't need actual humans and can thus potentially run more. More clarity on this point would really help.

Based on 5 iterations, I'm not sure that you can make the claim that it will monotonically improve much past that point. Did authors try past 5 (10, 20).

One sort of undiscussed thing here is that, requiring new models to be trained every iteration does mean that loop is pretty slow. Was 5 iterations chosen for that reason (so it wouldn't take multiple days). This should be maybe discussed as a weakness. E.g. for human studies or using humans, doesn't that mean the humans need to wait hours or else get new humans to provide feedback for every iteration?

questions:
Please clarify the points mentioned above, that would really help me make a better decision about the paper. In particular explaining why this method would be better in some way that EUREKA
(Either because ICPL doesn't require some assumption made by EUREKA or it's better in some other way).

Minor:
Why GPT-4o for the human experiment only? I'm not sure how much it matters actually, but found it curious.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693558939

thread_type:
official_review

<thread_object>
creator: reviewer_fib6

<content>
summary:
This paper presents a novel framework called In-Context Preference Learning (ICPL), which automatically generates dense reward functions by utilizing an LLM capable of querying humans for preference data. The authors find that their method greatly outperforms one baseline, PrefPPO, with respect to sample efficiency (PrefPPO requires far more human preference queries) and task performance. The authors also find performance comparable to that of Eureka, a baseline that also utilizes an LLM for generating dense reward functions but relies upon access to ground-truth sparse reward data rather than human preference data. The authors argue, since ICPL does not require access to a ground-truth sparse reward function, it has a clear advantage for tasks that are less well-defined or require human intuition. Additionally, they argue that training with human preferences will enable greater model alignment.

strengths:
I appreciated the fact that this paper took steps to optimize their baseline methods within reason. For instance, for Eureka, the authors continued generating candidate reward functions until the LLM had generated 6 executable ones (to make things fair for comparison against their own method).

The comparison against PrefPPO was strong.

weaknesses:
According to table 1, ICPL performance seems no better than that of Eureka. Furthermore, substituting in the values from table 3, ICPL performance with real human preference queries does not exceed Eureka’s performance on any task except Ant. Since ICPL does not outperform Eureka, ICPL’s benefit relies upon the ease of obtaining human preference queries in comparison with a ground-truth sparse reward function. I’m not convinced that this benefit is significant.

One argument, from the introduction, is an appeal to the success of preference-based training in other domains. I’m not convinced that this success generalizes to the domain of LLM-generated reward functions. 

The other core argument in favor of preference-based training is that human insight—expressed through preference queries—can better align agent behavior with human preferences. The authors motivate this through their custom HumanoidJump task, wherein the task is “to make humanoid jump like a real human.” They argue that this is a domain in which designing a sparse reward function would be difficult due to the nuances/subjectivity of mathematically defining jumping “like a real human.” In my mind, the paper largely hinges on this argument, however the authors only offer one case-study as evidence of the efficacy of human preference data in this domain.

I could be convinced otherwise, but I think there would need to be a more thorough analysis of human preferences in comparison with sparse reward functions in order to be certain.

Also, I found section 5.3.2: Evaluation Metric to be very confusing. I wasn’t sure what an “environment instance” was. I also didn’t understand which set of task metric values was used to compute the maximum for the RTS.

questions:
On page 1, I was confused by the phrase “tasks are distinct from the training data.” What does this mean?
Are there any other reasons to account for why human preference data might be preferable to sparse reward functions?
How do you actually generate the 6 reward function candidates? Do you randomly sample from the LLM? If so, how?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731568456115

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall response to all reviewers

comment:
We sincerely thank the reviewers for their valuable feedback, which has significantly enhanced the quality of the paper. We have made revisions based on their suggestions to improve the clarity and rigor of our work. However, we want to draw the attention of all reviewers to a misunderstanding that is consistent across reviews: **a common criticism that our method does not outperform EUREKA**. Our method should ***NOT*** be expected to beat EUREKA as EUREKA **has access to the actual reward instead of just preferences**. We only included EUREKA as an approximate upper bound on the expected performance ICPL could achieve. Rather than being a negative (i.e. we failed to outperform EUREKA), it is instead surprising that our method is able to match it. **We have revised the paper to make this clear.**

Also we want to clarify that our goal is **NOT** to show that human preference or preference-style signals are inherently better than directly using rewards. Rather, we are addressing tasks that lack any rewards by incorporating human feedback to guide the learning process. This has also been revised to be clearer in the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732556333021

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Addition of two new baselines

comment:
Based on the reviewer U5zu's helpful suggestion, we have added two additional preference learning baselines and uploaded them in the new PDF. We have added PEBBLE [1] as they suggested and additionally added SURF [2] , a version of PEBBLE that adds data augmentation. In both cases, our conclusions are unchanged and the performance of PrefPPO, PEBBLE, and SURF are all similar and sharply below ICPL. We hope that this strengthens our conclusion that ICPL is an effective approach to preference learning. 

[1] Lee, Kimin, Laura Smith, and Pieter Abbeel. "Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training." arXiv preprint arXiv:2106.05091 (2021).

[2] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF:
Semi-supervised reward learning with data augmentation for feedback-efficient preference-based
reinforcement learning. In International Conference on Learning Representations, 2022. URL
https://openreview.net/forum?id=TfhfZLQ2EJO.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel algorithm for using LLMs to synthesize reward functions for RLHF (focusing on applications to robot control). They build on Eureka, an existing approach, modifying it to handle human preference feedback instead of ground truth sparse rewards. One point of contention is the connection between this work and Eureka. The authors point out that Eureka assumes access to the ground truth reward function as feedback. In principle, this "ground truth reward function" could be implemented in the RLHF setting by querying the human to obtain their reward; however, providing numerical reward values is likely difficult for humans. Thus, the authors modify Eureka to instead work with human preference feedback in the form of rankings instead of numerical rewards. This contribution appears somewhat incremental; indeed, the methodology appears to simply modify the Eureka prompt to accommodate ranking feedback.