ICLR 2025

Published as a conference paper at ICLR 2025
REGRESSING THE RELATIVE FUTURE:
EFFICIENT POLICY OPTIMIZATION FOR MULTI-TURN RLHF
Zhaolin Gao1, Wenhao Zhan2, Jonathan D. Chang3∗, Gokul Swamy4,
Kiant´e Brantley5, Jason D. Lee2, Wen Sun1
1 Cornell University, 2 Princeton University, 3 Databricks Mosaic Research,
4 Carnegie Mellon University, 5 Harvard University
ABSTRACT
Large Language Models (LLMs) have achieved remarkable success at tasks like summa-
rization that involve a single turn of interaction. However, they can still struggle with
multi-turn tasks like dialogue that require long-term planning. Previous works on multi-turn
dialogue extend single-turn reinforcement learning from human feedback (RLHF) meth-
ods to the multi-turn setting by treating all prior dialogue turns as a long context. Such
approaches suffer from covariate shift: the conversations in the training set have previous
turns generated by some reference policy, which means that low training error may not
necessarily correspond to good performance when the learner is actually in the conversation
loop. In response, we introduce REgressing the RELative FUture (REFUEL), an efficient
policy optimization approach designed to address multi-turn RLHF in LLMs. REFUEL
employs a single model to estimate Q-values and trains on self-generated data, addressing
the covariate shift issue. REFUEL frames the multi-turn RLHF problem as a sequence of
regression tasks on iteratively collected datasets, enabling ease of implementation. The-
oretically, we prove that REFUEL can match the performance of any policy covered by
the training set. Empirically, we evaluate our algorithm by using Llama-3.1-70B-it to
simulate a user in conversation with our model. REFUEL consistently outperforms state-
of-the-art methods such as DPO and REBEL across various settings. Furthermore, despite
having only 8 billion parameters, Llama-3-8B-it fine-tuned with REFUEL outperforms
Llama-3.1-70B-it on long multi-turn dialogues. Implementatio...