Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF

Authors: Zhaolin Gao, Wenhao Zhan, Jonathan Daniel Chang, Gokul Swamy, Kianté Brantley, Jason D. Lee, Wen Sun
Primary area: reinforcement learning
Keywords: Reinforcement Learning, Reinforcement Learning from Human Feedback

Abstract:
Large Language Models (LLMs) have achieved remarkable success at tasks like summarization that involve a single turn of interaction. However, they can still struggle with multi-turn tasks like dialogue that require long-term planning. Previous works on multi-turn dialogue extend single-turn reinforcement learning from human feedback (RLHF) methods to the multi-turn setting by treating all prior dialogue turns as a long context. Such approaches suffer from covariate shift: the conversations in the training set have previous turns generated by some reference policy, which means that low training error may not necessarily correspond to good performance when the learner is actually in the conversation loop. In response, we introduce REgressing the RELative FUture (REFUEL), an efficient policy optimization approach designed to address multi-turn RLHF in LLMs. REFUEL employs a single model to estimate $Q$-values and trains on self-generated data, addressing the covariate shift issue. REFUEL frames the multi-turn RLHF problem as a sequence of regression tasks on iteratively collected datasets, enabling ease of implementation. Theoretically, we prove that REFUEL can match the performance of any policy covered by the training set. Empirically, we evaluate our algorithm by using Llama-3.1-70B-it to simulate a user in conversation with our model. REFUEL consistently outperforms state-of-the-art methods such as DPO and REBEL across various settings. Furthermore, despite having only 8 billion parameters, Llama-3-8B-it fine-tuned with REFUEL outperforms Llama-3.1-70B-it on long multi-turn dialogues. Implementation of REFUEL can be found at https://github.com/ZhaolinGao/REFUEL/, and models trained by REFUEL can be found at https://huggingface.co/Cornell-AGI.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730642192281

thread_type:
official_review

<thread_object>
creator: reviewer_gjrw

<content>
summary:
This paper introduces REFUEL (REgressing the RElative FUturE for reinforcement Learning), a novel approach for multi-turn Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs). The key contributions include:
1. A new algorithm that addresses covariate shift in multi-turn dialogue by using self-generated data and treating the task as a sequence of regression problems
2. Theoretical guarantees showing REFUEL can match any policy covered by the training set
3. Strong empirical performance, where an 8B parameter model fine-tuned with REFUEL outperforms a 70B parameter model on long multi-turn dialogues
4. A novel approach that eliminates the need for an explicit critic network by using a reparameterization trick and relative future rewards

strengths:
1. The paper addresses a significant limitation in current RLHF approaches by properly handling multi-turn dialogue scenarios instead of treating them as single-turn tasks.
2. The theoretical analysis is thorough and proves that REFUEL requires weaker conditions than previous methods like Natural Policy Gradient.
3. The empirical results are impressive, showing that a smaller model trained with REFUEL can outperform much larger models on long dialogues.
4. The method is computationally efficient by avoiding the need for an additional critic network.

weaknesses:
1. The experimental evaluation is limited to dialogues of 5 turns or less, which may not fully demonstrate the method's capability for very long conversations.
2. The simulator used for evaluation, while sophisticated, may not fully capture real human interaction patterns.
3. The paper lacks ablation studies on different components of the algorithm and their contributions to the final performance.
4. The comparison with PPO-based methods is omitted due to computational constraints, which leaves some questions about relative performance against all potential baselines.

questions:
1. Does REFUEL's approach work for other multi-turn tasks beyond dialogue, such as multi-step reasoning or sequential decision-making tasks?
2. Can you provide ablation studies showing the individual impact of key components in REFUEL? Specifically, how much does the relative future reward regression contribute versus the on-policy data generation?
3. Can you demonstrate REFUEL's effectiveness on significantly longer conversations (e.g., 10-20 turns)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730742221264

thread_type:
official_review

<thread_object>
creator: reviewer_glhj

<content>
summary:
This work introduces REFUEL, a method addressing the covariate shift issue appearing when commonly applying single-turn RLHF methods to multi-turn dialogues. It frames the problem as a multi-step MDP and merges the two-step of actor-critic procedures into a unified procedure. It shows improvements on benchmarks such as UltraInteract and AnthropicHH when having sample conversations evaluated head-to-head by GPT4.

strengths:
# Originality

To the best of my knowledge, the refuel technique and its application to multi-turn dialogue tasks is novel, as most existing work leverage an actor-critic framework.

# Quality & Clarity

The paper is well-written and easy to follow. It states clearly the limitations of current approaches and presents its contributions in a straightforward manner. The chosen hyperparameters and detailed proofs are clearly described in the appendix.

# Significance

The task of designing high-quality dialogue agents has become ever more visible in the past couple of years. Controlling generations over multiple dialogue turns is critical to the success of such agents, as it could enable more powerful and human-like capabilities with dialogues that carry an extended amount of context and information in previous turns and that are strategically generated in order to achieve a macroscopic goal. The paper is making a significant stride towards that.

weaknesses:
The evaluations proposed could be more extensive. In particular, it may be interesting to show evaluations on more common single-turn dialogue tasks (e.g. QA tasks) in order to show whether the quality is degraded for these tasks. It is also surprising to see a strong degradation in quality for turns #0 and #1 compared to the raw Llama 8B model, and the paper does not provide an explanation for this.

questions:
* Why are dialogues filtered to a maximum of 5 turns? It would seem that the performance improvement would be larger on longer dialogues.
* Why is performance degraded over most alternative setups for turns #0 and #1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730763662162

thread_type:
official_review

<thread_object>
creator: reviewer_iitf

<content>
summary:
The manuscript introduces a training technique, grounded in RL theory/practice, for improving multi-turn dialogue generation using large-language models. The key problem addressed is that in standard multi-turn dialogue generation, training is done usually by relying on single-turn techniques, i.e. optimizing the last step of a dialogue conditioning on all previous steps. The manuscript calls out as this being a problem, since the policy generating previous data is not the one being optimized and refers to it as *covariate shift*. The proposed solution is grounded in Q-learning, avoiding the need for training separate actor / critic networks which are cumbersome to maintain and unstable to train.

The manuscript points out that with Transformers state can easily be restored (by just feeding back in a partial sequence) and alternative (counterfactual) steps can be generated (by just sampling). Then through a series of tricks the policy training can be tied to the change in relative rewards.

The evaluation setup leverages UltraInteract, making use of a LLama-8B for experiments and LLama-70B for simulating a user. Each turn quality (i.e. win-rate over other responses) is determined using GPT-4. The results clearly indicate improvements on turns 3, 4, 5 but are below baselines on turns 1, 2.

strengths:
Some strengths include the following:

* The construction of the optimization algorithm I find novel and elegant.

* Very strongly grounded in theory of RL with the most being entirely explained from the common formulations of Q-learning paired with optimiziation techniques. Rather clearly written overall, as well taking into account less familiar readers with the details of the theory through the presence of a "Intuitive explanation" section.

* The use of two benchmarks (UltraInteract, AnthropicHH), as well as open-source models (LLama, ArnoRM) make results reproducible and ground the contributions.

weaknesses:
Although I could follow the theory behind the paper and I'm familiar with most of it, I'm not the best reviewer for the correctness of the derivation of the formulations and final objective trained. Therefore my confidence is rather low on this aspect.

Some weaknesses I'd like to ask authors about:
* Why not extend beyond H=5? This seems rather limiting and although improvements are noticeable in the GPT-4 evaluations, presumably the benefits should be much stronger at say H>10.

* Although it is commonly accepted practice for GPT-4 to evaluate win-rates (or quality as judge), are there other measurements that can help quantify usefulness of REFUEL? For example, the qualitative examples in Appendix D. indicate repetitive text by other baselines. Yet looking at green highlights of long responses for math problems, seems to indicate some reasoning improvements. 

* Can the authors comment on utility beyond multi-turn dialogue generation, e.g. perhaps for multi-step reasoning or tool-use?

questions:
See weaknesses discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730802632626

thread_type:
official_review

<thread_object>
creator: reviewer_lbbb

<content>
summary:
Long-term and multi-turn planning remains a recognized limitation in large language models (LLMs), limiting their effectiveness in complex, goal-oriented scenarios such as extended dialogue interactions. While reinforcement learning from human feedback (RLHF) with extended context lengths has been employed to address this, these methods encounter high computational costs and are vulnerable to covariate shift. This work presents REFUEL, a novel reinforcement learning framework designed for long-term planning. REFUEL optimises the model for the relative future, thus mitigating covariate shift.

strengths:
- Presents an interesting contribution to the area of RL in LLM's, making a stride in solving the problem of longer term planning.
- Illustrates the effectiveness of the technique not only on large models but also on smaller LLMs.
- Presents and proves performance guarantees of the approach.

weaknesses:
- REFUEL was only applied to the Llama 3 family of LLMs and further only the instruction tuned versions of these models. It would be more complete if the approach was also tested on models with different pretraining, e.g. the supervised language modelling pretrained models, models such as Gemma or Phi etc.
-  The datasets used for evaluation involve reasoning and harmful and helpful conversations with LLMs, but true goal oriented datasets such as those in task-oriented dialogue would present more of a long term planning challenge to test this approach.

questions:
- Could this approach also be applied to LLMs only trained with the language modelling objective?
- How would this approach perform in more challenging planning scenarios such as task-oriented dialogue?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors address the problem that LLMs have with multi-turn tasks (such as dialogue) due to covariate shift when using standard RLHF approaches. They propose a framework called REFUEL that addresses this by framing multi-turn RLHF as a series of regression tasks on self-generated data, achieving superior performance to existing methods and even enabling a smaller model to outperform a much larger one in long dialogues. The reviewers appreciate the novelty and theoretical grounding, the empirical results, and the clarity of presentation. However, they also raised some significant concerns primarily on the limited scope of evaluation (model and task diversity, evaluation metrics, ablations), performance degradations not adequately explained, and lack of comparison to PPO. The authors respond to the comments but the reviewers are not convinced.