RLHF with Inconsistent Multi-Agent Feedback Under General Function Approximation: A Theoretical Perspective

Authors: Ming Shi, Yingbin Liang, Ness Shroff, Ananthram Swami
Primary area: learning theory
Keywords: RLHF theory, inconsistent multi-agent feedback, regret analysis

Abstract:
Reinforcement learning from human feedback (RLHF) has been widely studied, as a method for leveraging feedback from human evaluators to guide the learning process. However, existing theoretical analyses typically assume that the human feedback is generated by the ground-truth reward function. This may not be true in practice, because the reward functions in human minds for providing feedback are usually different from the ground-truth reward function, e.g., due to diverse personal experiences and inherent biases. Such inconsistencies could lead to undesirable outcomes when applying existing algorithms, particularly when considering feedback from heterogeneous agents. Therefore, in this paper, we make the first effort to investigate a more practical and general setting of RLHF, where feedback could be generated by multiple agents with reward functions differing from the ground truth. To address this challenge, we develop a new algorithm with novel ideas for handling inconsistent multi-agent feedback, including a Steiner-Point-based confidence set to exploit the benefits of *multi-agent* feedback and a new weighted importance sampling method to manage complexity issues arising from *inconsistency*. Our theoretical analysis develops new methods to demonstrate the optimality of our algorithm. This result is the first of its kind to demonstrate the fundamental impact and potential of inconsistent multi-agent feedback in RLHF.

Decision: reject
Reviewer scores: 5, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1729698545589

thread_type:
official_review

<thread_object>
creator: reviewer_psxu

<content>
summary:
The authors provide a theoretical study on how feedback heterogeneity impacts the learning outcomes in RLHF. Specifically, it formulates an inconsistency model where the difference between feedback from each of the m different sources and the ground truth reward function is bounded above. The authors propose an algorithm with favorable regret guarantees under this model.

strengths:
1. The problem set up is novel and important.

2. Theoretical work on RLHF is important.

weaknesses:
1. Conceptually, I believe addressing the heterogeneity of feedback is an important direction for RLHF. However, I'm struggling with the notion of a "ground truth" reward function in this context. Based on your inconsistency model, it seems that the ground truth reward function can lie anywhere within a region, with the distance between it and each agent's feedback bounded.

However, considering your Example 1, if two agents have fundamentally different preferences, why should we even assume that a single ground truth reward function exists? It seems more reasonable to assume a ground truth reward function in situations where all agents share similar preferences, but their feedback is noisy or inconsistent. Unfortunately, both Example 1 and Figure 1 didn’t clearly convey a message to me.

2.  There are few claims I don’t really follow. Possibly due to my lack of knowledge. I will ask them in the question section.
3.   Writing can be more coherent. See minor suggestions below: 

Line 182, maybe $P_m(\tau_k \succ \tau_0)$ since it is the probability of agent m’s choice.

Line 219,$V_1^{\pi_k}(\tau_o)$ is not clearly defined, my guess is that it is $E[\sigma(R^*(\tau^{\pi^k})-\sigma(R^*(\tau_0))]$. And why we have $V_1$ instead of just $V$?

Line 286, $\bar{R_\alpha}$ shows up without definition (after reading I realise it has shown up on line 280)

Line 369, $R_k$ is not defined until next page (line 382). And I am assuming $R_0=R$?

Line 454 what is $\Pi$?

4. In the conclusion part, maybe some suggestions of further work and limitations of your own approach will be great.

questions:
1. Could you explain figure 1 to me please? And what does example 1 trying to illustrate there? (c.f. the first comment in the weakness session)

2. Why mutual information between two agent’s feedback helps learning? (c.f. your line 331 to 341)

3. How is your approach bringing benefits compared to the original RLHF approach? I.e. if I just directly learn the reward model based on all the data, why the regret will be worse?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729749259698

thread_type:
official_review

<thread_object>
creator: reviewer_2ec7

<content>
summary:
This paper studies the problem of reinforcement learning from human feedback (RLHF), where the feedback may come from $M$ different sources each with different reward functions. All these reward functions could be inconsistent with the ground truth reward, where the inconsistency level is denoted by $\xi$. This work focuses on regret minimization under the general function approximation setting. The authors propose four new ideas for the algorithm design, namely "Steiner-Point-Based Confidence Center", "Sub-Importance Sampling for Reducing Complexity", "Scaled Policy Weights for Reducing Biases", and "Optimism in the Face of Policy Uncertainty". The regret scales with $\sqrt{K/M}$ (here $K$ is the number of episodes) with a constant term depending linearly with $\xi$. This suggests that a low inconsistent level and abundant feedback sources are beneficial to RLHF.

strengths:
1. The problem of RLHF with inconsistent feedback sources is novel, interesting and grounded.
2. The algorithm design techniques are highly technical.

weaknesses:
1. The presentation of the paper makes it hard to understand. For example, the authors do not give results in special cases such as tabular MDP (or even bandits) and linear MDP.

questions:
Since I'm having trouble understanding this work, I do not have technical questions.

1. If the algorithm is adapted to tabular multi-armed bandit setting, will the algorithm admit time-efficient implementation (without enumerating the reward transition model in the confidence set)?
2. Can the algorithm be adapted from comparing $(\tau_i, \tau_0)$ to $(\tau_i, \tau_j)$?
3. What will the algorithm and its regret upper bound be for tabular multi-armed bandits?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730566246375

thread_type:
official_review

<thread_object>
creator: reviewer_kmhh

<content>
summary:
The paper investigates reinforcement learning from human feedback (RLHF) in a setting where feedback is generated by multiple agents whose internal reward functions may differ from an objective ground-truth reward function. This introduces inconsistency, as each agent's feedback reflects their own subjective understanding, influenced by biases or varying priorities.

strengths:
The authors propose a novel algorithm designed to handle such inconsistencies, featuring the development of a Steiner-point-based confidence set that leverages feedback from multiple agents and a weighted importance sampling method to manage complexity. The paper also establishes theoretical guarantees demonstrating that multi-agent feedback can still be beneficial even under inconsistencies, though with limitations based on feedback quality.

weaknesses:
I. The paper's readability is notably poor, presenting significant challenges for comprehension. Specifically:
1. From pages four to ten, the text is densely packed with high-complexity mathematical formulas, making it difficult for readers to follow the theoretical flow without substantial effort. The sheer volume and intricacy of the equations hinder accessibility, even for those familiar with reinforcement learning theory.
2. The pseudocode on page six is highly complex, introducing elaborate steps and technical constructs that are challenging to parse. The dense presentation of the algorithm complicates understanding, as it requires careful and repeated examination to fully grasp its structure and function.

I suggest breaking up dense sections of equations with more explanatory text, providing intuitive explanations alongside technical details, and restructuring the algorithm presentation to be more step-by-step.

II. Furthermore, the definition of Equation 2 seems wrong, as it defines the concept of inconsistency as an epsilon neighborhood, which, in contrast, means consistency.

questions:
1. Why is inconsistency defined as an epsilon neighborhood in Equation 2? What is the rationale behind using this specific formulation, and how does it capture the nuances of multi-agent feedback inconsistency?
2. Why does Equation 3 include a maximization over the inconsistency defined in Equation 2? Could you explain the necessity and impact of this choice on the regret analysis and how it influences the theoretical guarantees?
3. Does your theoretical work translate into a feasible, implementable algorithm with corresponding experimental results? If so, can you provide details on the practical application and any empirical evidence supporting the algorithm's effectiveness?
4. It seems $\sigma$ is misused in Equation 21.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730735881723

thread_type:
official_review

<thread_object>
creator: reviewer_9vyv

<content>
summary:
The paper focuses on the problem of using inconsistent feedbacks for RLHF under general function approximation due to possible human biases and diverse experiences. The real-world human feedbacks are expected to be different than the ground-truth reward function and possible be inconsistent with them and other human feedbacks. Therefore, the investigation is on whether the inconsistency helps the learning or exacerbate it. The performance is evaluated by the regret under inconsistency definition. The paper presents their RLHF algorithm based on Steiner point to solve this problem and present 3 ideas to tackle the issue and provide a theoretical analysis on regret due to inconsistent multi-agent feedback.

strengths:
* A novel algorithm RLHF-IMAF is proposed.
* Inconsistent feedback for RLHF is indeed a topic that is understudied and worth more attention due to the nature of human feedback.
* The problem is well motivated.
* The reasoning behind proposed ideas are well described and the necessary proofs are provided.

weaknesses:
* The paper is not well written. There are alignment issues with the equations. Due to the nature of the paper, it is heavily notated; but this makes it harder to focus on the message of the paper. 
* Providing some visualizations would be helpful to convey the proposed ideas. The visualization in Figure 1 is insufficient to get the message across. Since the paper is fully theoretical, there may not be any experiment to validate the approach. But a visualization consisting of iterative figures could be good.
* The explanation for the results in appendix and the references to the appendix are missing. The claims in the main paper and proofs in the appendix are disconnected from each other, and the information put in appendix is not explained. The proofs in the appendix and the theoretical analysis in the main paper all look disconnected.
* A toy experiment to validate the algorithm would be useful.

questions:
* How do you evaluate that you achieved what you proposed? Given the theoretical analysis, you suggest that multi-agent feedback even under inconsistency is helpful; but then you assert that if the feedback has poor quality, even when M is large, the regret is still bad although the feedback is helpful. How can the feedback be helpful when the regret is bad if we evaluate the benefit of the method with the regret?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper explores RLHF with heterogeneous multi-agent feedback. The main concerns raised by reviewers are as follows:

- Poor Presentation: All reviewers found the current version of the paper poorly presented and unsuitable for acceptance. Its highly technical and math-heavy nature makes it difficult for the general ML audience to follow. A major restructuring of the paper would significantly improve its clarity and accessibility for future submissions.

- Weak Justification for the New Preference Model: The motivation for introducing this new preference model is not well justified. It remains unclear how this model offers advantages over the standard BT model in handling heterogeneous human preferences.

Additional Note: The BT model might already be capable of addressing heterogeneous human preferences, as it can potentially capture the distribution of preferences at the population level, even if individual preferences differ.