Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF
Authors: Zhaolin Gao
Wenhao Zhan
Jonathan Daniel Chang
Gokul Swamy
Kianté Brantley
Jason D. Lee
Wen Sun
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Reinforcement Learning from Human Feedback
Primary area: reinforcement learning

Abstract: Large Language Models (LLMs) have achieved remarkable success at tasks like summarization that involve a single turn of interaction. However, they can still struggle with multi-turn tasks like dialogue that require long-term planning. Previous works on multi-turn dialogue extend single-turn reinforcement learning from human feedback (RLHF) methods to the multi-turn setting by treating all prior dialogue turns as a long context. Such approaches suffer from covariate shift: the conversations in the training set have previous turns generated by some reference policy, which means that low training error may not necessarily correspond to good performance when the learner is actually in the conversation loop. In response, we introduce REgressing the RELative FUture (REFUEL), an efficient policy optimization approach designed to address multi-turn RLHF in LLMs. REFUEL employs a single model to estimate $Q$-values and trains on self-generated data, addressing the covariate shift issue. REFUEL frames the multi-turn RLHF problem as a sequence of regression tasks on iteratively collected datasets, enabling ease of implementation. Theoretically, we prove that REFUEL can match the performance of any policy covered by the training set. Empirically, we evaluate our algorithm by using Llama-3.1-70B-it to simulate a user in conversation with our model. REFUEL consistently outperforms state-of-the-art methods such as DPO and REBEL across various settings. Furthermore, despite having only 8 billion parameters, Llama-3-8B-it fine-tuned with REFUEL outperforms Llama-3.1-70B-it on long multi-turn dialogues. Implementation of REFUEL can be found at https://github.com/ZhaolinGao/REFUEL/, and models trained by REFUEL can be found at https://huggingface.co/Cornell-AGI.

Published as conference paper at ICLR 2025
REGRESSING THE RELATIVE FUTURE: EFFICIENT POLICY OPTIMIZATION FOR MULTI-TURN RLHF
Zhaolin Gao', Wenhao Zhan?, Jonathan D. Chang Gokul Swamy", Kianté Brantley Jason D. Lee", Wen Sun' Cornell University, Princeton University, Databricks Mosaic Research, 4 Carnegie Mellon University, Harvard University
ABSTRACT
Large Language Models (LLMs) have achieved remarkable success at tasks like summa- rization that involve a single turn of interaction. However, they can still struggle with multi-turn tasks like dialogue that require long-term planning. Previous works multi-turn dialogue extend single-turn reinforcement learning from human feedback (RLHF) meth- ods to the multi-turn setting by treating all prior dialogue turns long context. Such approaches suffer from covariate shift: the conversations in the training set have previous turns generated by some reference policy, which means that low training error may not necessarily correspond to good performance when the learner is actually in the conversation loop. In response, we introduce REgressing the RELative FUture (REFUEL), an efficient policy optimization approach designed to address multi-turn RLHF in LLMs. REFUEL employs single model to estimate Q-values and trains on self-generated data, addressing the covariate shift issue. REFUEL frames the multi-turn RLHF problem as sequence of regression tasks on iteratively collected datasets, enabling ease of implementation. The- oretically, we prove that REFUEL can match the performance of any policy covered by the training set. Empirically, we evaluate our algorithm by using -- to simulate a user in conversation with our model. REFUEL consistently outperforms state- of-the-art methods such as DPO and REBEL across various settings. Furthermore, despite having only 8 billion parameters, Llama-3-8B-it fine-tuned with REFUEL outperforms 1. on long multi-turn dialogues. Implementation of REFUEL can be found at https: : / / /github. com/ whwaoinnaaaaa Matiaa////EEELLEE and models trained by REFUEL can be found at https: : /ttgggigggggggg face.ace.cooooo Corne 1 AAG.
1 INTRODUCTION
Despite the impressive performance Large Language Models (LLMs) have demonstrated tasks like summa- rization, question answering, and short conversations (OpenAI, 2023; Meta, 2024; Google, 2024; Anthropic, 2024), most LLMs struggle with planning effectively for long conversations that involve multiple rounds of di- alogue or asking follow-up questions about previous responses (Irvine et al., 2023; Abdulhai et al., 2023). The root cause of this deficiency is that most preference fine-tuning methods using Reinforcement Learning from Human Feedback (RLHF, Christiano et al. (2017); Ziegler et al. (2020); Ouyang et al. (2022); Rafailov et al. (2024b); Azar et al. (2023); Guo al. (2024); Rosset et al. (2024); Dong et al. (2024); Gao al. (2024); Wu et al. (2024); Meng et al. (2024)) treat all tasks as single-turn (i.c. as a contextual bandit (Auer et al., 2002; Langford & Zhang, 2007)) even when some tasks are fundamentally multi-turn (e.g. multi-step dialogue with a user).
The simplest way to convert multi-turn task, such as dialogue, into single-turn task is to train on the last-turn of the dialogue and use dialogue history as context. Although this approach is appealing due to its
Work done at Cornell
f2g292, ws455) cornell edu, wenhao. zhan@princeton, edu, j. j.changedatabrickk. com, gawamy@andren sw. emu, edu, kdbrantley@g harvard. edu, jasonlec@princetonn edu

---

Published as a conference paper at ICLR 2025
Single-Turn RLHF
REFUEL
Relative Winrate w.r.t REFUEL
Train
Test
Train
Test
15
Relative Winrate
KEFvE.
-10
3
Turn
Conariate Skift
Mutth-leee Ralfeut
Figure 1: We present REFUEL: simple, regression based approach for multi-turn RLHE Traditional single- turn RLHF methods suffer from covariate shift as they train on histories generated by the base policy rather than current policy. REFUEL eliminates the covariate shift by iteratively generate on-poliey datasets, aligning the training and testing distributions. REFUEL performs better at later turns compared to the baseline methods in terms of winrate (which is computed against the base policy, Llama-3-8B-it, using GPT4).
compatibility with pre-existing pipelines, training on histories generated by the base policy rather than current policy introduces covariate shift (Kohavi, 1995) between the training and testing distributions. This can result in poor performance when the learner is in the conversation loop (Ross al., 2011). This phenomenon is at the heart of recent empirical observations by Zhou et al. (2024b) about LLMs struggling to ask questions to clarify missing information in conversations or being unable to self-correct after making mistakes in mathematical reasoning tasks, as shown in the concurrent work of Kumar et al. (2024a).
In response, several authors have proposed treating multi-tum tasks like dialogue as proper online RL problems, rather than contextual bandits. For example, Zhou et al. (2024a); Shani et al. (2024a) propose applying an online actor-critic framework to allow the policy to learn respond to its own past decisions (e.g. asking for more information or correcting mistakes), improving practical performance. However, actor-critic methods substantially increase the training complexity (in terms of both stability and memory usage), especially when both the actor and the critic are LLMs with billions of parameters.
To handle policy-induced covariate shift without necessitating an extra critic network, we propose using the reparameterization trick introduced by Degrave et al. (2019); Rafailov et al. (2024b) to regress future returns (i.e. Q-values) in terms of the log of policy ratios. allowing us to "read-off" the corresponding soft optimal policy, replacing the usual two-step procedure of fitting a critic followed by an explicit policy optimization step with a single step procedure. However, is not clear how to get supervision for this regression problem, as learned reward models can only provide accurate feedback at the trajectory (e.g. conversation) level rather than at the turn (c.g. generation) level, which means we cannot apply techniques for learning a Q function that require per-timestep reward labels. Our key insight is that the difference in two conversation-leww rewards generated from a shared prefix is an unbiased estimate of the difference in Q values between the first divergent turns. We can then use this difference in Q-values as regression target to adapt any pair-wise preference based single-turn RLHF method to the multi-turn setting without introducing any additional components. Crucially, iteratively generate on-policy datasets of prefixes and two independent completions from the current policy. Training on this on-policy data ensures that the model learns to participate in the sort of conversations it would actually encounter when participating addressing the covariate shift issue that stymies offline single-turn methods. We call this approach REFUEL: REgressing the RElative FUturE for reinforcement Learning.
the conversational loop, rather than those in some offline dataset,
2 PRELIMINARIES
Consider a conversation between a human and an Al assistant. Let the initial question from the human be denoted as æ1 ~ p. Upon receiving 31, the AI assistant, æ, generates response y/1 л n--12)). Subsequently, given 21, f1, the human responds in turn 22 ~ T(-121, y1), where T(- y: denotes the conditional distribution of the human responses The1. Upon receiving 22, the Al assistant generates second response
2

---

Published as a conference paper at ICLR 2025
D2 ~ (2) This interactive process continues until we reach the total number of turns H. At the end of the interaction, the AI assistant receives a trajectory-level reward r({ yh 1 ). In this work, we do not focus on the learning process of the reward function r; instead, we utilize existing pre- pre-trained reward models.
We can cast the multi-turn RLHF as a standard multi-step Markov Decision Process (MDP) by using the conversational transcript state. Let the state 8, at turn h comprise all prior information up to turn h, excluding the current response: 8/2 = {I1, (it1,,1.....22..... 1,-1, 1h-1, J. ). Then, the response y can be interpreted as action. We denote the state and action spaces at step h as S₁ and 3/, respectively. For simplicity, we assume [D%] Y for all h [.
[.J.TT The policy и maps from state Sq to the next response yh, i.c., yh ~ T(11))) We denote dā (s) as the state distribution turn h induced by the policy n, with $;: =d the process of sampling 8n from иT. The policy receives reward r r($H+1) after step H where, for notation convenience, we
denote SH+1 ($ yh) ). Note that sH+1 is the entire mullii-um conversation. The dynamics P(s (Sh. yh) are fully determined by T that governs the response generation process of the human, i.e., JA-1 ~ T(-ss,, Wh) and sh+1 : {sh. Uh 11+1}. We emphasize that in contrast to the standard single-turn RL.F setting which is often modeled by a deterministic MDP or bandit problem, the transition P is random as 7 is random.
Rollins & Rollouts. Given a state Sp and a response yh. we denote by 8H+1 <18;, the process of sampling the final state by generating response y/ at 8n followed by executing π until turn H (i.e., finishing the entire conversation). We refer to this process as rollout of policy æ. Following the standard RL notation, denote Q7 (sh,3 yhh as the state-action Q function which models the expected future reward-to- to-go of the random process of taking yh, at Sy followed by rolling out π to the end. Similarly, given a turn step h, we use rollin to refer to the process of sampling a state turn h. denoted as 8s, ~ d%
Resets. Given a state 8h, resetting to Sk, simply means that the policy л starts from 8n again and generates factual trajectories from 8h- While resets are often considered as strong assumption in general RL, it is trivially achievable in the context of RLHF for text generation. Resetting to can be implemented by feeding the partial conversation 8,, { V1: ; } to the transformer-based policy π as a prefix context. This capability allows a policy to generate multiple independent future trajectories from the same state 8/.
2.1 THE LIMITATION OF SINGLE-TURE TURN RLHF METHODS ON MULTI-TURN PROBLEMS
Recent RLHF algorithms such as DPO (Rafailov et al., 2024b), IPO (Azar et al., 2023), SPPO (Wu et al., 2024), and REBEL (Gao et al., 2024) are specifically designed for the single-turn setting which can be formulated as contextual bandit problem with
1. When applying these methods to multi-turn datasets such as Anthropic HH (Bai et al., 2022), it is common to first convert from multi-turn into single-turn format. Specifically, for each sequence of multi-turn interactions {21, V1, T2. "............................... H. YH }, these single-turn turn methods treat the first J. and only optimize the last-turn generation of yи-
H - interactions Consequently, the dataset consists of (x ~ Dan, y ~ T (-1x),, .v/ ~ x ((+x)) where we use DMN denotes the offline dataset. This approach is used by Rafailov et al. (2024b) to optimize the multi-tum Anthropic HH dataset.
large context x := (01, Vi...
As depicted in Figure 1, applying single-turn RLHF methods to a multi-turn setting in this manner introduces covariate shift (Kohavi, 1995) between training and testing distributions. Intuitively, the resulting policy has only learned to generate the final response based on the contexts present in the offline data. However, during inference, the policy is likely to observe different contexts, as they are generated by itself, rather than the policy used to collect the offline dataset. This can lead to degraded performance at test time, paralleling the issues with offline approaches to imitation learning like behavioral cloning first formalized by Ross et al. (2011).
3 REFUEL: REGRESSING THE RELATIVE FUTURE
To address covariate shift in multi-turn RLHF without introducing the overhead of an explicit critic network, we introduce REFUEL. REFUEL eliminates the need of an explicit critic by merging the two-step process of actor-critic algorithms into unified procedure and reduces covariate shift by using on-policy datasets. At each iteration t, REFUEL aims to solve the following KL-constrained RL problem:
- En,,,, (15=))(()))1
Tt+1 argmax E fe, -0.. (-(e)) Q (8h, mh) rell
17
3

---

Published conference paper at ICLR 2025
Algorithm REgressing the RElative FUturE for reinforcement Learning (REFUEL)
Require: number of iterations T', learning rate trajectory-level reward model r(.). Initialize policy IT 2: for t= 1.... do Collect dataset (h, Sh. Yh. Uh SH+1. s sHH1) where
h ~ U(H), 8h d% yh ne(-1sh,, v/. me Te()))), 8H+1 88++1 (8h, 8 Th(h
Update policy via regression to relative future rewards:
1 (11 (yAls))) In x11/115)) (((#==)-r((=+))))
T++1 = argmin ED
(I)
17 Te (1) Te (,)lss))
Relative Future Reward
5: end for
Intuitively, the policy æę-1 chosen to maximize the expected reward (through Q-values) while simultaneously minimizing the change from the previous policy TTs with the balance determined by parameter 7). From Ziebart et al. (2008), we know there exists closed-form solution to the above minimum relative entropy problem:
Vh, Sh. yh (3+1())(s)) Te (unl exp(11 (sh, ma)), Z(sa) Σ (ynlss)) (xp(11 (x)((1 Vh)) (3)
Z(sn)
Following Degrave et al. (2019); Rafailov et al. (2024b), we can rearrange Eq. 3 to express the Q-value as function of the policy:
vh, Sh, Yh : Q% (sh, un) = In (In In
Te+ Te+l(1 (vnl88))) (sn))
(4)
Note that the partition function (sa) does not depend yn and that sample another response y/, by resetting Te to 8h. 2/, ~ Ty (-1s)). By taking the difference of the above expression across the paired responses (yn, v/,) we can eliminate the partition function:
Yh, 8h, yh sh. 1% (8h, 3h) 1% (sh,, v/h) 1(11, In Te+1 (Unlss))))) In
Te+ (2118)))))))
(5)
(yn(sn)
Te (T(3(115))
Following Gao et al. (2024), can then formulate satisfying the above constraint as least squares problem:
(11 In Teta (ynlss) In Te+1 (V-l(1))ss) (QT, (54. O7 (Sho (9%
(6)
(y(()s)))
(1(2((55))
Unfortunately, this loss function uses Q-values, which do have direct access to. However, the reward obtained from a rollout starting from 8h is an unbiased estimate of the Q-value. We perform independent obtaining the ending states SH+1 and s114 from the two *- TT 1/1 )). The rewards of these states r($H+1) and independent rollouts (i.e., SH+1 *- TT (Sh, Mr), $H+1 81+1
policy rollouts using ne at (sh, y1.) and (St.
r(s/,.1) have expected values of Q (#h, (5h) and Q (sh, y()). Then, leveraging the fact that the minimizer of a least squares problem is the conditional mean of the target variable, we arrive at Eq. 1. The pseudocode of algorithm is provided in Alg. where Ep denotes the empirical average over the dataset D. To reduce the computational complexity of REFUEL, we uniformly sample a turn step during training similar to Ross & Bagnell (2014). Notably, the use of pairwise rollouts with shared prefix and unbiased estimates of Q-values enables REFUEL to leverage any single-turn RLHF optimizer. Here, we follow the derivations of REBEL, leading to the same update formulation as REBEL.

---

Published as conference paper at ICLR 2025
3.1 INTUITIVE EXPLANATION OF REFUEL
From our above argument, we know that solving Equation optimally would imply that
Yh, Sh Mh s/ ( Te-1(ynnss) In
T(+1 (1Al(s)) Te (1(((sh))
Q7% 1 (sh+ yv.) Q7% (shs 2/h).
Tee (V(lss)))
Summing the above over 1/. further implies that there must exist a y-independent function ea (8n) such that Rearranging the terms, we can write that 4h, Sh. : Te+ (3n(ss) (e(llss)) exp (107, (sn, (9h,)h) nes, mcx m
wh, sh - In TT- (u-i(u)) Q% (sh, uh) -ch) (sn).
Te (B(lls))
oc (ynlss) exp (n1%) ($h, Uh)).
Note that that ne (sx)))) = In ((ss,) exp(1 (sk: Uh)) (sh) is the log-partition function. In our algorithm REFUEL, we predict the relative future rewards instead of modeling the partition function using an additional critic network. Prior works do not leverage the idea of predicting relative values: they either assume that the partition function is approximately equal to a constant (Zhu al., 2023) or use an another critic function to approximate it, incurring extra GPU memory and computation costs (Wu et al., 2024; Richemond et al., 2024). We also note that that above policy update procedure recovers the NPG update with the softmax policy parametrization (Agarwal et al., 2021), which converges to the globally optimal policy at the rate of O(1T)), a faster rate compared to that standard policy gradient methods.
3.2 MORE RIGOROUS ANALYSIS AND CONNECTION TO PAST POLICY GRADIENT THEORY
The above simplified explanation relies on an unrealistic assumption that least square regression can learns the Bayes optimal predictor exactly in point-wise manner. In this section, we analyze the performance of REFUEL under much more realistic assumption - we assume that the learned predictor in Equation can predict well on average under the training distribution. Our analysis below extends that of REBEL (Gao et al., 2024) from the bandit setting to multi-turn MDPs with stochastic transitions. We denote S, as the set of all possible states time step h, and we assume Sn, and S for h + hí' are disjoint. This assumption is satisfied the multi-turn RLHF setting since S/2 and sn model states with different numbers of turns. We start by assuming the learned predictor from the least square regression problem in Equation I has bounded in-distribution generalization error.
Assumption 1. There exists an e R²,, such that for all t,
(2(1 In (e+i(yn(sH) In We (WAlss)
Te+1 (vhlss))))) (1 (s:))
(Q (sh, yn) Q7 (sh, (5))
In the above assumption, we have bounded prediction error to the Bayes optimal, the relative Q value (Q" (sh; 3hh) 2% (sh, (8h.y/)) v))),, under the online data distribution. For regret bound, we will compare to a policy that is covered by the training distributions.
Assumption 2 (Coverage). We say that a comparator policy x" (not necessarily a global optimal policy) is covered by the training distributions, if the following two concentrability coefficients are bounded for all t:
J (sh)
7 (3
Ca := max
00, Cy == max h, & T. (U)
< <0.
h, 3 d, (sp)
The first concentration coefficient Ca concerns the state distribution, while the second one concerns the coverage in the single-turn response (i.c., action) space. These concentrability coefficients play key role in policy gradient theorem (e.g., Kakade & Langford (2002); Bagnell et al. (2003); Abbasi- Yadkori et al. (2019); Agarwal et al. (2021); Bhandari & Russo (2024)). In our definition, distributions d and ne to capture the case where on-policy distributions happen to be informative in terms of covering a good comparator policy (e.g., initialization mo typically a pre-trained LLM, is informative
iteration-dependen on-policy
5

---

Published as conference paper at ICLR 2025
in terms of covering high quality policy). While we focus on pure on-policy algorithm, similar to REBEL, incorporating additional offline distribution into the algorithm and analysis is straightforwar.
Denote ( as the expected total reward of the policy æ. REFUEL has the following performance guarantee. Theorem Under Assumption I and Assumption 2, if we initialize ni to be uniformly random policy and choose an uppropriate 7, after T iterations, there must exist policy where t E [T' such that for all compurator policy n²,
I
((7))J((TT)) "
H WC VCssaCyyaee T
The above theorem indicates that as long as least square regressions are successful, i.e., in-distribution generalization error is small, we can learn at least well any policy x¹ covered by the training data. Note that, in general, when learning is involved, we should not expect to compete against the globally optimal policy since PG methods cannot do strategic exploration. We now discuss the situation where Assumption holds by connecting and comparing it to similar conditions used in prior policy gradient theory.
Discussion on Assumption One condition where in Assumption 1 can be small the Approximate Policy Completeness (APC) condition.
Definition 1. There exists én E IR+,, such that for all a E II,
min min En, Su-i (1 In x (unlss) 1 in exxxls))exp((70(( 2 s cnI
CeS-R' r'ell
C(ss)
Note that C(s) can be any function from S - R as long as it is independent of the y. To get a better understanding of the above assumption, let us first show that the following soft policy improvement closure property implies the above condition. The soft policy improvement closure condition means that for all E II, we have T Ty]s)) exp(112% (s, 2s) e II (here Z is the partition function). In this case, we set C(s) := Z(s), and the soft improvement policy x' <v]ss) := π (vls) exp(11Q g))/ is the minimizer and we have én O. On the other hand, in general,
note that C' does not have to be equal to the partition function Z. In fact, our condition allows us to select the C' that delivers the smallest APC error én. This is possible in our case since algorithm is performing regression to relative Q values. Conditions similar to APC are common sufficient conditions for the success of policy optimization methods (e.g., CPI (Kakade & Langford, 2002), PSDP (Bagnell et al., 2003), PG (Bhandari & Russo, 2024), and NPG (Agarwal al., 2021)) While it is not a necessary condition, it is known that the standard realizability condition alone (i.e., just assume [I) is not sufficient for permitting efficient policy learning in general (Jia et al., 2024).
Now we show that by using pairs and performing regression to future reward difference, our APC condition is strictly weaker than the conditions required in previous NPG analysis (Agarwal et al., 2021). This is formalized in the following example using log- linear polices.
The APC condition is strictly weaker than the small Q function approximation error assumption in NPG methods (Agarwal al., 2021) in the linear setting. In particular, have the following propositions. Proposition 1. Given feature mapping & S x x + Rdd, let II denote the log-linear policy class:
Remark 1.
=I= : 30 e Rdd, ((h. T (ynl9)) x exp (0 (8hh))))) Ww))
If we can bound the Q function approximation error using linear function on & - a key condition Agarwal et al. (2021) use for proving NPG convergence:
VT e II: min En,sy whao (Qh, w (onddsn (ohh)n))) se, weRd
then the APC condition is satisfied with 6n S.c.
Prior methods typically require APC under hard policy improvement procedure. The simplified version of their conditions can be intuitively understood as argmax,, (%, a) II for all II, which corresponds 1 in the soft policy improvement closure.
6

---

Published as conference paper at ICLR 2025
Conversely, there exists an instance where the APC condition is satisfied with 6n = O while the Q function approximation error can be as large 1:
Proposition 2. Consider the same tog-linear policy class. There exists an MDP. feautre mapping & and π E II such that the APC condition is satisfied with én 0 but
min Bhe Mamm [ (Sh. ww' ((hh.n)) weRd
These propositions formally demonstrate that APC is weaker than the condition required for proving NPG convergence, showing the theoretical benefit of using pairs of rollouts and regressing to relative futures.
Further discussions on the above propositions can be found in Appendix C, and the variance reduction effect of REFUEL is discussed Appendix
EXPERIMENTS
Our implementation closely follows the pseudocode in Alg. 1. We empirically evaluate REFUEL's ability under two multi-ture RLHF settings. In the first setting, we create a multi-turn conversation simulator that uses Llama-3.1 1-77BBBt to simulate human-in-the-loop In the second setting, we evaluate our approach using a pre-sampled sequence of questions from existing multi-turn RLHF datasets to simulate multi-turn dialogue. The first setting models realistic situation where the learning agent and the user need to interact, while the second setting models a simplified situation where the sequence of human questions is pre-sampled before the conversation begins. However, even in the second setting, the learning agent still needs learn to generate future turns conditioned on its own previous turns. Additional experiment details are in Appendix F.
4.1 BASELINES: SINGLE-TURN AND MULTI-TURN
We compare REFURL, to single-turn and multi-turn baselines that are extensions of three algorithms, RL.OO (Kool et al., 2019), DPO (Rafailov et al., 2024b) and REBEL Gao et al. (2024), as well as two open-source I.Mss: Llama-3.1-88 1-BB-it and Llama-3. 1-70B-it (Meta, 2024). For the single-turn baselines, we consider the following three settings:
Last-Turn-Offe (LT-OFFLINE):
This a standard approach to applying single turn methods to a multi-turn RLHF dataset. Specifically, we rollin using offline data and train the last turn pairs of offline responses, D [ yH yH.31)) ~ Dot, 88-1 = (S Уи). $1 (HH, v }. For RLOO and REBEL, the rewards are computed using SH+1 and HHH11 while DPO selects chosen and rejected responses based on the reward values.
hast-Turn-Mixed (LX-MIXED)): This is another standard approach, similar to L.T-OFFLINE, where rollin using the offline data. However, on the last turn, we sample and train on pairs of on-policy rollouts responses: D = {sH Dott. у/ ~ Te (+1s=),22 3/r ( SH+1 S (SH yH 5 = (SH, 1/)).
Last- Tunn-Tue--Onl (LT-ONLINE): Unlike the previous two approaches, this approach involves using on-policy samples rather offline data for both the rollin and rollout responses. Specifically, the state sy and the responses are generated from the current policy with a simulated user, denoted as D = {SH ~ dij " Te (-(111):22 1/4 ~ Te (-Ism), 1 (sH1 y ) = (sH,
For the three single-turn baselines mentioned previously, we always rollin and optimize the last turn H. In multi-turn baseline approaches, we rollin and optimize each turn instead of only optimizing the last turn. We consider multi-turn baseline approach similar to the baseline proposed in Shani et al. (2024a):
(MT-MIXED) Similar to the LT-MIXED approach, we rollin with the offline data, but now sample on-policy pair of responses at arbitrary state from the offline dataset. After sampling a state, we perform two rollouts from sn to the end H: D = (h ((H).s, Sy. ~ Dott yh Te(+(s)), 3/h ~ Te (-(ss)), sH-1 Te(Sn, (s) yh)) $ ny (sh, v(,) The rewards are computed using $H+1 and 8H+1 which are the unbiased estimates of the Q-values at turn h. This baseline optimizes future returns similar REFUEL, but the states sampled from the offline data.
Multi-Turn-Mae
The detailed dataset statistics for each method are provided in Appendix F.4, We chose not to compare against PPO baselines (Shani et al., 2024a; Zhou al., 2024b) due to its computational inefficiency. Training with

---

Published as conference paper at ICLR 2025
Winrate at Turn (1)
Method
h 1
h=2
2
h = 3
h
H
avg
Llama-3. 1--B-it
57.8
57.8
52.4
55.2
54.0
$5,44
Llama-3. 1ama--3.-700--1 1-70B-it
70.4
66.4
61.0
53.0
55.4
61.24
RLOO-LT-OPPLINE
48.4
41.4
45.4
47.6
49.2
46.40
RLOO-LT-
48.4
44.2
43.4
43.0
42.6
44.32
RLOO-LT-ONLINE
46.2
50.0
48.4
49.2
46.2
48.00
RLOO-MTT-MXED
56.2
49.8
48.6
48.6
47.8
50.20
DPO-ITTDFFLLINE
51.2
46.8
42.6
41.4
46.8
45.76
OPO-LT-MIXED
56.2
51.0
51.6
50.6
48 8
$1.64
OPO-LT-ONLING
56.8
52.2
53.0
54.0
52.4
53.68
DPO-MT-MIXHI
57.4
54.6
51.6
50.8
49.2
52.72
REBEL-LTTOFFLIIGG
51.6
46.0
45.4
48.4
42.2
46.72
REBEL-LT- REBEL-LT-MIXED
60.0
51.2
51.6
46.4
48,
51.52
REBEL-LT-ONLINE
55.2
51.6
54.2
52.4
57.8
54,24
REBEL-MT-MIXED
58.3
53.2
53.8
51.0
54.6
54.18
REFUEL (iter I)
54.6
53.6
57.8
56.2
59.4
56.32
REFUEL (iter 2)
55.2
53.4
58.8
57.2
58.6
$6.64
Table 1: Results on Ultralnteract. The best-performing method for each conversation turns excluding Llama-3.1-SB-i and 1lama-3.1-70B-- is highlighted in bold and the second best is underlined.
PPO requires additional value network, which substantially increases memory demands. PPO is already challenging to scale in single-turn scenarios, making it even more impractical in this multi-turn context.
4.2 SETTING ONE: LLM AS A HUMAN IN THE LOOP
Task and Implementation. We evaluate REFUEL on UltraInteract (Yuan al., 2024), which involves the model responding to instructions with complex reasoning tasks, covering general chat scenarios. We filter the dialogues to have a maximum of 5 turns. For simulating the user's random question sampling process, i.e., The1 T(|ssk, Wh), we use Alama-3.1-70B-- (Meta, 2024). Our base model is Llama-3-8B-it (Meta, 2024), and we employ ArmoRM (Wang et al., 2024) as the reward model. In other words, we create a simulator (similar to (Li et al., 2016)) where Gaama-3.1-70B- is acting as a human user, and our agent will interact with the user for multiple turns, starting from the prompts in Ultralnteract. Finally, the entire conversation is scored by the reward model.
To construct this semi-synthetic dataset for REFUEL at each iteration, we begin by sampling an initial state 81 Don, i.e. sample prompt from the offline UltraInteract dataset. We then uniformly sample the dialogue length
U U5) and a turn step h ~ U(H). We rollin with our policy to simulate a dialogue up to H turns and then reset turn h to generate another trajectory up H, which gives us one data tuple (sh, yh, V/, SH+1, 1)) We generate the dialogues for the entire dataset (i.e. [D] is the size of Ultralnteract) and consider the entire dataset one large batch. Then, we optimize in mini-batch style over the entire dataset. We perform 2 iterations for this setup. Additional implementation details, simulator details, and hyperparameter settings are listed in Appendix F. F2,, and F.3.
Evaluation. To evaluate the quality of the generated dialogues, we compute the winrate (Rafailov et al., 2024b) against the generations from the reference policy, Llama-3-8B-it, using GPT4 (OpenAI, 2023) over randomly sampled subset of the test set with 500 samples. We execute the policy inside the simulator to generate a dialogue with turns from the initial prompts. We calculate winrates at all turn levels h [1,5]. The prompt for winrate evaluation is provided in Appendix F.5 which is adopted from Dubois et al. (2024).
Rollin on-policy algorithms outperform algorithms that rollin with the offline data. The experimental results presented in Table 1 demonstrate that on-policy rollin algorithms such as REFUEL and LT-ONLINE consistently outperform algorithms that rely on offline data rollin, such as LT-OFFLINE, LT-MIXED, and MT-MIXED. On-policy rollin algorithms perform better because they experience on-policy interaction during training, which eliminates the distribution mismatch between training and testing. Even when you optimize at all states h H (MT-MIXED) instead of just the last state H (LT-MIXED), note that the offline algorithms
8

---

Published conference paper at ICLR 2025
Dataset
Algorithm
Winrate (1)
RM Score (1)
KL (((() (1)
RLOO-ET- MIXED
74.5
-5.21
18.37
RLOO-LT-ONLIN
75.0
-5.13
16.22
RLOO-MT-MIXED
76.2
-5.08
17.23
Anthropic HH
REBEL-LLT-
79.6
-4.79
17.23
REBEI-LTT-ONLINE
80.2
-4.75
15.91
REBEL-MT-MIXED
78.6
-5.03
16.79
REFUEL
82.8
-4.68
17.83
RLOO-LT- MIXED
67.2
0.88
119.6
RLOO-LT-OR -LOO-LL--NLINE
70.2
0.78
98,91
RLOO-MP PRMO-MT-MINEE MT-MIXED
33.3
-0.27
87.45
UltraInteract
REBEL-LT- BEBEL-LT-MIXED
70.4
0.93
121.7
REBEL-LT-ONLINE
73.4
0.82
62.85
REBEL-MT- MT-EXED
34.4
-0.25
61.34
REFUEL
79.6
0.87
93.19
Table 2: Results on Anthropic HH and UltraInteract. The best-performing method for each dataset is highlighted in bold. REFUEL outperforms all baselines in terms of winrate.
perform worse than our online algorithm, LT-ONLINE. This highlights the importance of performing on on-policy rollins during training to mitigate distribution mismatch.
Optimizing for long-term rewards improves the multi-turn performance. The results Table show that multi-turn algorithms REFUEL and MT-MIXED outperform LT-ONLINE and I.TMMIXED respectively in terms of winrate at every turn except for the first turn. While both REFUEL and LT-ONLINE perform on-policy rollouts using the current policy, LT-ONLINE only performs rollouts and optimization for the last turn, whereas REFUEL performs rollouts every turn h S H and optimizes at all h S H. Similarly, both MT-MIXED and LT-MIXED perform rollin using an offline dataset, but MT- MT-MIXED optimizes at all turn level h while MIXED only optimizes at the last turn. From these results, we observe the benefit of optimizing for long-term future rewards instead of just optimizing at the last turn.
REFUEL outperforms 1lama-3.1-70B-i on dialogues with more than three turns. While the winrates for the baseline algorithms degrade with more turns, REFUEL exhibits a rising trend. The relative winrate differences between the baseline methods and REFUEL are shown in Fig. 1. REFUEL takes advantage of both on-policy rolling and long-term reward optimization, achieving the best winrate on average and at longer conversations. Notably, the 8B size model trained by REFUEL performs better than the Llama-3.1-70B model, which has gone through RLHF post-training, demonstrating the effectiveness of our approach in handling extended dialogue interactions. The qualitative analysis of REFUEL provided in Appendix G.
4.3 SETTING TWO: USING PRE-SAMPLED QUESTIONS FROM THE DATASETS
Task and Implementation. In this setting, LLM is simulating human user in the interaction loop. Instead, we consider a simplified setting where the sequence of questions comes directly from the dialogues in the datasets. More formally, this setting can be represented by restricted transition T. denoted as T (- which only relies on the human's previous questions x and is independent of the assistant's responses In this context, the human's 's questions F1,... .H are pre-sampled based on T before the interaction begins, meaning the human prepares sequence of questions to ask in advance. While this setup has limitations, allows us to test algorithms and baselines on pre-collected multi-turn dialogues with questions from humans instead of LLMs.
We evaluate the performance of REFURL, on the Anthropic Helpful Harmful (HH) task (Bai al., 2022) and the UltraInteract dataset (Yuan et al., 2024). Both datasets are filtered to exclude dialogues with more than 5 turns and 2048 tokens. We compare REFUEL against three baseline algorithms, REBEL-LT-MIXED. and REBEL-MT-ME MIXED, LT-OFFLINE methods not comparable other methods. We utilize Llama-3-8B- (Meta, 2024) as the base model and EffairX-LLaMA --MM-MM---MM (Xiong et al., 2024a) as the reward model for both datasets.
REBEL-LT-ONIIN

---

Published as conference paper at ICLR 2025
Evaluation. We evaluate each method by its balance between the reward model score and KL-divergence with the SFT policy, testing the algorithm's 's effectiveness in optimizing the regularized RL objective. To evaluate the quality of the generation, we compute the winrate (Rafailov et al., 2024b) against the generations from the base model Llama-3-8B-it using GPT4 (OpenAI, 2023). The winrate is computed from a randomly sampled subset of the test set with 500 samples over the entire dialogue. Given the varying lengths of dialogues in the dataset, we do not compute turn-wise winrates.
Quality analysis. Table 2 presents a comparison between REFUEL and the baselines methods. Notably, REFUEL. consistently outperforms all baselines terms of winrate when evaluated under GPT-4 against responses generated by the reference policy. While REBEL-IT-MIXEL achieves the highest RM score for Ultralnteract, REFUEL exhibits a comparable RM score with a significantly smaller KL divergence. The results in this simplified setting demonstrate that even when human questions are pre-sampled, on-policy training in a multi-turn fashion is beneficial. We include convergence plots and example generations from REFUEL in Appendix I and J respectively.
5 RELATED WORK
Single-turn RLHE DPO (Rafailov et al., 2024b) was originally designed for a single-tum RLHF setting, which can be modeled by a bandit problem or multi-stage MDP with the deterministic transition. Follow-up analysis of DPO (Rafailov al., 2024a) is also based on this singe-turn setting, and the derivation of DPO being capable of learning Q function is based on deterministic transition. Note that multi-turn RLHF can be stochastic the turn level since the sampling process of human questions can be highly random. Thus, the analysis and conclusion from (Rafailov et al., 2024a) (i.e., DPO learns the Q functions) do not apply when naively applying DPO to a multi-turn setting. Other single-tur baselines (e.g.. IPO (Azar et al., 2023), SLiC-HF (Zhao et al., 2023; Liu et al., 2023). REBEL(GaO et al., 2024), SimPO (Meng et al., 2024), KTO (Erhayarajh et al., 2024), ORPO (Hong et al., 2024), SPPO (Wu et al., 2024), HyPO (Song et al., 2024)) also do not directly apply to stochastic multi-stage MDP settings.
Multi-turn RLHE Multi-turn RLHF algorithms have been proposed address reasoning and multi-turn dialogue problems. In the context of math reasoning, concurrent work (Kumar et al., 2024b) applied REIN- FORCE to a two-turn RL setting, demonstrating the importance being on-policy for learning self-correction behavior in math reasoning. Another concurrent work (Xiong et al., 2024c) applied single-turn algorithms such as DPO (Rafailov et al., 2024b), KTO (Ethayarajh et al., 2024), and their online variants (Guo et al., 2024; Xiong al., 2024b) to multi-turn setting. However both Xiong et al. (2024c) and Kumar et al. (2024b) focus on deterministic transition settings where there is no sequential interaction hetween users and the models. In our experiments, we compare the multi-turn variants of the single-turn algorithms proposed in (Xiong et al., 2024c). For multi-turn dialogue, Snell et al. (2022) built on the implicit Q-learning (Kostrikov et al., 2021) while Shani al. (2024b) extended the general preference setting (Dudík al., 2015; Saha & Krishnamurthy, 2022; Wang et al., 2023; Swamy et al., 2024; Munos et al., 2023; Rosset et al., 2024) to multi-turn. In our setting, we focus on RLHF with reward models rather than the general preference setting. Our work focuses on developing an on-policy RLHF algorithm in the stochastic multi-turn dialogues, where we focus on the importance being on-policy for multi-turn RLHE, similar to Kumar et al. (2024b) observation in the math reasoning setting.
Additional related works on policy optimization methods can be found in Appendix K.
6 CONCLUSION AND LIMITATIONS
We present REFUEL, a simple, regression- based approach for multi-turn RLHF with strong performance guarantees and empirical performance in multi-turn dialogue. We develop new on-policy multi-turn RLHF algorithm and show the importance of on-policy rollins to avoid covariate shift. We demonstrate that extensions of single-turn RLHF methods cannot mitigate the train-test distribution mismatch, deteriorating in performance as the conversation goes on while REFUEL improves to reason across the entire dialogue.
10

---

Published as conference paper at ICLR 2025
ACKNOWLEDGEMENTS
ZG is supported by LinkedIn under the LinkedIn-Cornell Grant. GKS was supported in part by a STTR grant. KB acknowledge: This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. WS acknowledges fundings from NSF I1S-2154711, NSF CAREER 2339395, DARPA LANCER: LeArning Network CybERagents, and Infosys Cornell Collaboration.
REFERENCES
Yasin Abbasi- Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellért Weisz. Politex: Regret bounds for policy iteration using expert prediction. International Conference on Machine Learning, pp. 3692-3702. PMLR, 2019.
Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023.
URL htwphhttp//// http://axvvvvv. org/ abs/ / 22111. 18232.
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98): 1-76, 2021.
Anthropic. Introducing the next generation of claude, 2024. URL URL.htpps//ww.aatrrppi.. news/ http///aaddeeeeeeee 1y.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences, 2023.
J Andrew Bagnell and Jeff Schneider. Covariant policy search. IJCAI, 2003.
James Bagnell, Sham M Kakade, Jeff Schneider, and Andrew Ng. Policy search by dynamic programming. Advances in neural information processing systems, 16, 2003.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage. Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac natfield-Dodde Ben Mann, Dario Amodei, Nicholas Joseph. Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022. URL https: //arxi//xviv lttaaaiwi...g/aaaaaeeeaaaoaaeaaa org/abs / 222 08073.
Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. Operations Research, 2024.
Jonathan Chang, Kiante Brantley, Rajkumar Ramamurthy, Dipendra Misra, and Wen Sun. Learning to generate better than your Ilm. arXiv preprint arXiv:2306.1 1116, 2023.
Jonathan D Chang, Wenhao Shan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D Lee, and Wen Sun. Dataset reset policy optimization for rlhf. arXiv preprint vrXiv:2404.084955 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024.
11

---

Published as conference paper at ICLR 2025
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances Neural Information Processing Systems, 2017.
Jonas Degrave, Abbas Abdolmaleki, Jost Tobias Springenberg, Nicolas Heess, and Martin Riedmiller. Quinoa: a qq-function you infer normalized over actions, 2019. URL httttttts///aaat http:///////a/o.oo///////////// org/absg./a..o....o 19111 01831.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf, 2024. URL
https://arxiv.orgg abs http:/abb/22225ee/5
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way debias automatic evaluators, 2024. URL http:: //tpriaaavvvv///a///a/eeaeeaaava wwt.:///ggg//aabageoeoo hal.:/ttteeeaarr.....oo////////p
04475.
Miroslav Dudík, Katja Hofmann, Robert E Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Conference on Learning Theory, pp. 563-587. PMLR, 2015.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:240.001306, 2024.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018. URL https: /ttpxviivvv//////a///.////aaaall hrgp/bs//////aoooooo 802.00...000 09477.
Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J. Andrew Bagnell, Jason D. Lee, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards, 2024. URL https:::// / a arxiv. .ow. abs 2404. 17677
Google. Gemini: A family of highly capable multimodal models, 2024.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018. URL 1801. 01290.
Jiwoo Hong, Noah Lee, and James Thome Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2(4):5, 2024.
Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Ziyi Zhu, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp. Xiaoding Lu, Thomas Rialan, and William Beauchamp. Rewarding chatbots for real-world engagement with millions of users, 2023. URL 06135.
htttps//aaxxvv.rggge.
Zeyu Jia, Gene Li, Alexander Rakhlin, Ayush Sekhari, and Nati Srebro. When is agnostic reinforcement learning statistically tractable? Advances in Neural Information Processing Systems, 36, 2024.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference Machine Learning, pp. 267-274, 2002.
Sham M Kakade. A natural policy gradient. In T. Dietterich, S. Becker, and Z. Ghahra- mani (eds.), Advances Neural Information Processing Systems, volume 14. MIT Press, 2001. URL https: http:////ooeeeeeisse/eee neurips.. cc sac/paper_fee files/ pttpppppppr/0//11112/ 1J5abeeBBssBec1111556666eeee
Paper.pde
Ron Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 2, IJCA''95, pp. 1137-1143, San Francisco, CA, USA, 1995. Morgan Kaufmann Publishers Inc. ISBN 1558603638.
12

---

Published as conference paper at ICLR 2025
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free!
In DeepRLStraciPedeee/ 2019. URL https: //api. semanticscholar.oo.ooo wwt.ollarrrrg/oooppsooeeooee 198489118.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, 2021.
URL https: httpp:://arxi....//// arxiv. org/ ats/aab.................e1e..... 06169.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup. Feryal Behbahani, and Aleksandra Faust. Training language models to self-correct via reinforcement learning. 2024a. URL https://ccx.v..rrgg abs 12409.1297.
Aviral Kumar, Vincent Zhuang. Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop. Rebecca Roelofs, al. Training language models self-correct via reinforcement learning. arXiv preprint arXiv:2409. 29.224499.121,,,.. 2024b.
John Langford and Tong Zhang. The spochhgreedd algorithm for multi aumti-armed bandits with side information. Advances in neural information processing systems, 20, 2007.
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv: 1606.04444 2016.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2019. URL https: //arxiv. oeggo//gaaooo.... abs 11....00.00000 02971.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprini urXiv:2309.0665 066577 2023.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward, 2024. https:/ //tp///avvv/////////////.///.avv wtwg/bb////aeeeeo 2405. 14734.
Meta. Introducing meta Ilama 3: The most capable openly available Ilm to date, 2024. URL https:
//ai meta........ www.ccoccc....... /bllg//oggee /tm///eaaalll 1aaaa-1aamm3333
Rémi Munos. Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhao- han Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00886, 2023.
OpenAL. Gpt-4 technical report, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to q²: Your language model is secretly q-function. arXiv preprint arXiv:2404.12335 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024b. URL https:
/ w/aaaiivoogg/aaaaaaaaaaaa http://aa./////3.33/ 18290.
Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, and Bilal Piot. Offline regularised reinforcement learning for large language models alignment, 2024. URL https: //arxiv. orgg abs 20...11111100 19107.
Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.554 2014.
13

---

Published as conference paper at ICLR 2025
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models self-improve with general preferences, 2024. URL https: /aaxiiv wttgg//b///00//0 03715.
Aadirupa Saha and Akshay Krishnamurthy. Efficient and optimal algorithms for contextual dueling bandits under realizabilityy In International Conference on Algorithmic Learning Theory, pp. 968-994, PMLR. 2022.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-1897. PMLR, 2015.
Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, Avinatan Hassidim, Yossi Matias, and Rémi Munos. Multi-turn reinforcement learning from preference human feedback, 2024a. URL htttp:://axxi..../ 2400..1455.
Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, al. Multi-turn reinforcement learning from preference human feedback. arXiv preprint arXiv:2405. 14655, 2024b.
Charlie Snell, lya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.1887/, 2022.
Yuda Song, Gokul Swamy, Aarti Singh, Andrew Bagnell, and Wen Sun. Understanding preference fine-tuning through the lens of coverage. arXiv preprini 2rXivv2406.01460) 2024.
Gokul Swamy, David Wu, Sanjiban Choudhury, Drew Bagnell, and Steven Wu. Inverse reinforcement learning without reinforcement learning. In International Conference on Machine Learning, pp. 33299-33318. PMLR, 2023.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. In Forty-first International Conference on Machine Learning. 2024.
Haoxiang Wang, Wei Xiong. Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12883 12845, 2024.
Yuanhao Wang, Qinghua Liu, and Chi Jin. rlhf more difficult than standard rl? arXiv preprint
arXiv:2306. 111.21366..111,, 2023.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement eomeetioniieeiiioereeene learning, Machine Learning, 8:229-256, 1992. URL https:://// peattss///apt //api. wwwpttttcccccocoooooooooooooo 19115634.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.0066 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024a.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference Machine Learning. 2024b.
Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviy Rosenberg. Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, et al. Building math agents with multi-turn iterative preference learning. arXiv preprint arXiv:2409.02392 2024c.
14

---

Published conference paper at ICLR 2025
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding. Xingyao Wang. Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng. Zhiyuan Liu, and Maosong Sun. Advancing Ilm reasoning generalists with preference trees, 2024. http:///// org/ htagg//bs/24440 02078.
Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D Lee, and Yuejie Chi. Policy mirror descent for regularized reinforcement learning: generalized framework with linear convergence. SIAM Journal Optimization, 33(2):1061-109) 2023.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint 0rXiv:2305./0422 2023.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-tumn rl. arXiv preprint arXiv:2402.19446, 2024a.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl, 2024b. URL https: / wwwpw/wwwwww///////////.//////// 2400 11..0.1..... .194 19446. 6.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness & harmlessness with rlaif, November 2023.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse rein- forcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume AAAI088, pp. 1433-1438. AAAI Press, 2008. ISBN 9781577353683.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.
15

---

Published conference paper at ICLR 2025
A PROOF OF THEOREM
We first introduce the definition value functions and advantage functions:
VL (sh) = 1E
Σ
r(s yh 1/, = vnn 1QX (sh. un)1
Vsa 6 6 h [M],
Lh
A7 8h: 3h == (5) un Vi (sh),
Vsa e Sh. yn Vh [H].
Then we have the following performance difference lemma: Lemma 1. For any policy and n², we have
- J(T)
Σ E
sa ((sw) [AR (sh, Wa)]
Therefore, from Lemma we know
T Σ -
T
Σ
B
on Mn (1on) [A% (sp: vn)]
(8)
On the other hand, let us define A', Ah follows:
1
In Te-1 (Bn(s5))) 2% (8h,
(shhh Uh. ) = 1
V8h,
Te (yn(ss)
A (sh) == E ((an) [ (s )n
Then under Assumption 1, we can bound the magnitude of A A the following lemma: Lemma 2. Under Assumption I, we have for all [T] that
(A (sh, yn.) A.r (SN) (sn))² c
E Bn um (don)
Now we can analyze the performance REFUEL. Let Ai, (sh, 3n) denote A (sh, un) Bn) (sn). then we know for [T]
(yn(s5) (e(ulss)) exp(nA) Un)).
Vsh. Wh.
Therefore, REFUELiS equivalent to running policy mirror descent (PMD) w.r.t the reward function Af. PMD has been studied extensively in the literature (Zhan al., 2023; Gao al., 2024) and can obtain the following performance guarantee:
Lemma 3. Suppose we have (A) Uh)l s C. for all t e [T], [H], Sh, é Vn. Then if we initialize mi to be uniformly random policy and choose 17 v Y(C2T), we have for all h [H] that:
Σ
sy - ((ww) (Ah (sh.)n)) un)] TInY.
Now from Lemma 3 and (8), we have
H T
11 T
Σ -(
=
Σ
Σ
sn mn an ((ar)
(sh, (1).
Σ Σ
R8
sn -n- ((aw) [At (8n) A A's vh)
1=1
E=
h- t=1
s 2CCH/Thn
Σ
E
- (1aa)
[1 (sn) - A (shh
(9)
h-
t=
16

---

Published conference paper at ICLR 2025
From Cauchy-Schwand inequality, we have
E en rdj ux- (tea)"
[1 (8h) D' (sh, Uhh]]
h=
E anndg (1oo))) [ (sh, A (sh)) ]]
H Emannddy unm (A' Uhh A (., (sn))
Then from Assumption 2 and Lemma 2 we know
E py (How) [44 (8n) - A'(sn, (1)))))))
S Cxp Cy E p. (-mm)()) (4 (sn) - A (Shh Cayye Cpea+ 2"
Therefore, substitute the above result into (9) and we have
s HT Cape Cy.st
H
This implies that there must exist / E [T] such that
In
J(m')
Cy. 2" / Cor
T
A.1 1
PROOF OF LEMMA 1
Note that we have
J(a))- E
Σ r (8h; 4hh
E Es
E
Σ
((hh 9hh
+ Ey (s1)
h=2
E
Σ r(sh; 9hh
+ E. - (
2=2
E
(8hh 3hh
E [ (82)
E. [A] y1)]
2=2
Here the first step is due to the definition of value function and the third step is due to the Bellman equation. Now apply the above arguments recursively to Egr 2 (8h,, (8h) E,, [V (82)] and we have
H Σ E sn (pow) [A] v.]]
J(x') Σ E. [A] (8) h=1
h=1
This concludes our proof.
17

---

Published conference paper at ICLR 2025
A.2
PROOF OF LEMMA 2
Due to Assumption 1, have
In Te-1(yolsh)
In (1n(5) (0% (sk. 0% (hh (shs
En (Hes..n (Hpw)))))
The (2mlls))) <c(1/lon)
Bh wiem (l ~ (1ax) (si y1.) ('h.
=Ep sa-m (( - (1(u)
((A 211 (sh (s' v/h) A.. (sw))
=n ((sn))
(A (sh M.) D (sw))
where the last step is due to the independence yn and 1/4 given sh. Therefore, we have En -fh'' us ((oo) (A' wb) - A A (vn))
e
A.3 PROOF OF LEMMA 3
The proof is almost the same as the proof of Lemma 2 in Gao et al. (2024) and here we include for completeness. Since Te+ (yn(sh) (v(nls))) exp(1AA) (sh, mh)), we have for any t [T], e [H] sh Sa. that:
-KL(TT" (-19%) (1s))) = KL
(+1ss)][((((((x))) (
+ y (1=) (sh, 9h) In Z (sn).
(10)
where Zj, is the normalization function. For in (z), using the condition that 17 1/A, we have n/.4 (x. 3) 1, which allows us to use the inequality exp(x) s 1 + x + x² for any s 1. Meanwhile, we can bound In 2 (ss) as follows:
exp() (s,, vni))
In (8h) In
Σ yh
$ Σ (vnnlsn)) (1 n ( + 1 ( (h))))
Wh
$ In (1 C2) s c²
where the second step the fact that n ( (sh, Uh.) 1 and exp(x) s x2 for any x 1. The third step uses the fact that E (t)s))())s,)) [A], (sh. ynh) ] Thus, substitute the above result in (10) and we have:
By (Hea) [l 44% (sh, Mh. )] s KI.(T (.18)) FT (+1s)) KI-(=" ((Iss) C2.
Sum over all iterations, obtain for all h [H] and s Sy, that:
Σ vn - ((an) (8h, mnd s (n())
With= take SA di. on both sides and we conclude the proof.
18

---

Published conference paper at ICLR 2025
B PROOFS OF THE APC CONDITION
B.1 PROOF OF PROPOSITION
Fix any policy II. Suppose that (Vn(8s) exp(0" (sh, and let denote the best approximator of Q% 9h)) Then we know
eh un- (10 un) w (sn), yı)) b
Now let (8))l8))) exp((0 mpo) ((hh, Ub)) and C(sh) Em. (ynlsn) exp(1we @(sh. Uh)). Then we have
In In (unlsn) #(yn(s)) exp(n) 9hh) 27
En -f unm
C(sn)
En un (1T yn) - w 9h)) se.
This concludes our proof.
B.2 PROOF OF PROPOSITION 2
Consider bandit problem with two actions 3o- Vi- Suppose r(3o) r(v)) 1 and let of ('o) [1,-1]. (vi) [1, It can be observed that the uniformly random policy yu in the policy class
II. In addition, we have r(v) (w*)* ((V) where "o,)T=[1.1]
Now on the one hand, for all policies π (v) exp(0' (()) in IT, l't and C exp(1)) Then we have
Eyewr (1 - In (य) (x)))xpp(r(())) exp(mr(())) 2 )1=0
=0.
17
C
This means that the APC condition satisfied with 6n 0.
On the other hand, the function approximation error under the uniform random policy μ is
min E g-n (r(y) w ((v)) minEx-- [1 w <(x)) 1.
where the inequality comes from AM-GM inequality. This concludes proof.
19

---

Published as conference paper at ICLR 2025
C ADDITIONAL DISCUSSIONS ON PROPOSITION AND 2
The key intuition behind these propositions that our APC condition puts no constraints on the function as long as it is positive (e.g., C(s) can be an arbitarily complicated function). With log-linear policies. noting that In (()) with C S TR* basically models all functions from IE, we can further simplify the APC condition into the following form:
VT:
min min En wa-a (w" 6(5h. yh. ((s)) Q7 (sh. (h)) S ETI- geS- welt
(1I)
Note that different from the condition required for the NPG analysis from Agarwal et al. (2021), we are free to pick any function (ss) to help the linear function to further fit the Q function, i.e., the function class {w" ((s,)) v) + (()) E RR {w" (68, a)/w [. By using pairs of rollouts from a shared prefix and performing regressing relative reward-to-go, REFUEL takes the advantage of the arbitarily complicated function g without paying any additional cost in computation and sample complexity (i.e., REFUEL never needs to explicitly learn such g).
S + RJ can be significantly richer than the linear function class
Readers familair with NPG at this point may ask what happens if one fits the advantage function instead of the Q function, i.e., modify the condition in Eq. 7 via the following with A
V VI, min E sn ma- (A (sh, Un) - w (( Un.) Un) Sc. woRld
where (s. ( (s). The NPG analysis from Agarwal al. (2021) can still work out under the above advantage based condition. Fitting advantage seems a reasonable choice since the advantage also models the relative reward-to-ge go from a shared state and often used in practice for variance reduction purpose. We show that our APC condition is still weaker. To see that, use the definition of the advantage function, we can rewrite the above condition as follows:
VT min Bh -na (w' (( Uh. (si) (Hh) ynh)) sc WEEN
(12)
Comparing to the condition Inequality 11, we see that the key difference is that APC condition, we can pick any g(s) while in the above Inequality 12, our choice of is limited to be V2 (s) only. Thus, if
Inequality 12 holds, APC must hold with en S e. Thus, we conclude that REFUEL'S APC condition is weaker than the one needed for advantage A based NPG.
20

---

Published conference paper at ICLR 2025
D THE VARIANCE REDUCTION EFFECT OF REFUEL.
In this section, show that regressing to relative Q-values has a variance reduction effect with Gauss-Newton updates under finite data setting, We denote the parameterized policy as and start by approximating our pre-
dictor , In no (yha (8h) To, (yn(8)) by its first order Taylor expansion at 0g: (In To (yt 8%)) In no, (yh (8x))
Ve In No. (2n(ss) "(-- 0s), where indicates that ignore the higher order terms in the expansion. Denote 0 Og and replace the predictor by its first order approximation in Eq. 1, have:
1 (Vo Inoo, (ynlsn) Vo To (V(ls)))*
he Gl- mo, Glaa) (1)
- -(O% (8h, Un. Qh (sh vh)
(13)
With finite data D {lens Sh, yh.,. Sh. I we denote the empirical Fisher information matrix P. NN Ens (Tv In te, (M...shh..) Vo In Te, (sh.,,)h..))) (Vo Io, (yr,, (sh..) V No, (1/ (snn)) and have the following claim.
Claim 1. F; is unbiased estimate of the Fisher information matrix:
E F.j
E Booddd um- sn (1ax) [Voll In no, (yrlss) (Vo In no. (yn (ynlsn))"]]
Further, the minimum norm minimizer 8" in Eq. 13 under finite setting has the form:
8 == F
Σ
V To, (Vh,, Isn., (Q hm (sh, Wh..) ) Q (shm Wia ))
2N
Ve In No, (sh,, snn (Qth (sha Shon ) Qhi The
(sha Ulua ))
where Fi is pseudo-inverse of Ft.
The proof of this claim is deferred Ve In Te, (yh... (ss.. (Q,"".
Looking at the gradient formulation hm (si.. yh.,, yhh,))) - bn (sh.. vhh,)) 8", we see that 2Thh (sh. yh. serves as baseline for variance reduction. Interestingly, this gradient formulation is similar to RLOO (Kool et al., 2019). However, different from RLOO, we pre-condition this variance reduced policy gradient formulation via the Fisher information matrix, leading to better performance.
Appendix
Tu:
21

---

Published conference paper at ICLR 2025
E PROOF OF CLAIM
We prove claim in this section. We start from deriving the expectation of the empirical Fisher information matrix.
Fr)
= F.
2 an no -00 oo (Hon): m (-on)
[(Vo In To. (yhls)) Vo To. (b/l)))) (Vo (ynlsk) Vo In To, Cvha ss))]
16 hn at Mintro (15a)
Vo In me. (unlsa) (Vo In (yn]sw))
E Meendd oo un Clon).sh mo, (-laa) Volln mo, (wnlsn) lnoln (1Als))))
=E h.snnda un Mammo (1oa) Vo In ma, (BnlsK) (Vo In To (
where the last equality uses the fact that yh and v/ are independent given h, sh and B Byg-to, ([ww) (vo In To, (ynlsn) [h)] O. Therefore F, is indeed unbiased estimate of the Fisher informa- tion matrix.
Since Eq. 13 is an ordinarly least square regression problem, the minimum norm solution of the least square regression problem is:
8= F
T.0 Σ (vo In no, (sn., - Vo In To, (Vh (sn.. )) (Q hm (Sh..: yh, - 2%.0 (Sh.. Vh.,)
N
F
Σ Vo In Te, (yh, 19n. ) (2hn (sh,, yh.,)
To (shn Hhon) ))
=
2N
+ Vo In To, (wh.,, shh (Q The (9hm Vho)) Qhi The ( (Sha ylon ))
hm
This concludes our proof.
22

---

Published conference paper at ICLR 2025
F EXPERIMENTAL DETAILS
F.1 ADDITIONAL IMPLEMENTATION DETAILS
Setting One. We perform full parameter training for Llama-3-8B-Instructt For ArmoRM', directly use the reward scores without any normalizations. For each iteration, we generate the dialogues using the simulator for the entire dataset (i.c. [D] is the size of the entire dataset) and consider the entire dataset one large batch. Then, we optimize in mini-bateh bani- style over the entire dataset. We perform iterations for this setup. The experiments are trained on H100 GPUs for two hours for each iteration.
Setting Two. For Llama-3-8B I--sruuttt we only train the last four layers in the model while keeping the other layers frozen. For FsfairX-LLaMAS --M-MM----MM----- -0--0.. we directly the reward scores without any normalizations, Anthropic HH experiments are trained on 8 H100 GPUs for two days, and Ultrainteract experiments are trained on H100 GPUs for four days.
In this setting, we use a small batch size with [D] 32. We train for one epoch over the entire dataset. Since iterate more frequently, ensure that σe remains close Tegs we apply an additional KL penalty to the reward:
(I. = R.M 8) - (In Te, (y/x) - In no, (y/z))
(14)
where RM (z: y) is score from the reward model given prompt æ and response y. Furthermore, ensure that the online generations terminate within the maximum generation length, we penalize any generation that exceeds this length by setting r(x, y) small fixed constant,
F.2 SIMULATOR DETAILS
We Llama-3.1-70B- 3-1--Bt-it to simulator the user. The prompt for the model is provided below which is adapted from the winrate prompts from Rafailov et al. (2024b) and Dubois et al. (2024):
Prompt for User Simulator
Below dialogue between the user and the assistant, Pretend you are the user this conversation. What question would you next?
### Dialogue: {[dialogue)]
#4# Instructions:
FIRST provide justification of the question you ask.
SECOND, on a new line, state only the question.
Your response should use format:
Justification: <one-sentence justification Question: <question to ask next >
HHuggingFace Model Card: BectellammaMMet-Llaama Instruct
'HuggingFace Model Card: RLIIFowwAArmoDMLLLaaaa------0
'HuggingFace Model Card: ShinXCCFfttnXXLLLMMM3MMMMMM....
23

---

Published conference paper at ICLR 2025
F.3 HYPERPARAMETER DETAILS
Parameter Setting (Setting One)
Method
Parameters
RLOO-LT-OFFLINE RLOO-LT-MIXES
batch size: 128
learning rate: 3e-7
weight decay: 1e-6
schedule: cosine decay warmup ratio: 0.1
RLOO-LT-ONLINE
RLOO-MT-MIXED
DPO-LT-OFELINE
batch size: 128
learning rate: 3e-7 schedule: cosine decay warmup ratio: 0.1
beta: 0.3
weight decay: le-6
DPO-LT-MIXED
batch size: 128
learning rate: 3e-7
beta: 0.03
schedule: cosine decay warmup ratio: 0.1
weight decay: 1e-6
DPO-LT-ONLINE PPO-MT-MIXER
batch size: 128
learning rate: 3e-7
beta: 0.1
schedule: cosine decay warmup ratio: 0.1
weight decay: 1e-6
REBEL-LT-OFFLINE
batch size: 128
learning rate: 3e-7 schedule: cosine decay warmup ratio: 0.1
eta: le2
weight decay: 1e-6
REBEL-LT-MIXEL REBEL-IT-ONLINE REBEL-MT-MIXED
batch size: 128
learning rate: 3e-7
eta: le3
schedule: cosine decay warmup ratio: 0.1
weight decay: 1e-6
REFUEL (iter 1)
batch size: 128
learning rate: 3e-7
eta: le2
schedule: cosine decay warmup ratio: 0.1
weight decay: le-6
REFUEL (iter 2)
batch size: 128
learning rate: 3e-7
eta: le1
schedule: cosine decay warmup ratio: 0.1
weight decay: le-6
24

---

Published conference paper at ICLR 2025
Parameter Setting (Setting Two)
Dataset
Parameters
Anthropic HH
batch size: 32
learning rate: 3e-7
schedule: linear decay train epochs:
num epochs:
rj: 1.0
7: 0.05
=-
Ultainteract
batch size: 32
learning rate: 3e-7
schedule: linear decay
train epochs: I
num epochs:
y: 1.0
2: 0
T=
25

---

Published conference paper at ICLR 2025
F.4 DATASET DETAILS
Setting One. The statisties of the dataset for each baseline and REFUEL are shown in Table As the offline dataset contains fewer samples for longer dialogues, methods that sample state from this dataset show an inverse relationship between the number turns and the number of available dialogues. using simulated user maintain a uniform distribution across dialogue lengths, as the dialogue length is uniformly sampled up to 5 turns. We filter any dialogue with length more than 2048 tokens.
contrast, methods
% in Dataset
Dataset
Train/Val/Test
Max Generation Length
H=1
H=2
H=3
H=4
H=5
LT-OFFLINE
76.9
12.1
6.40
3.20
1.40
205K/500/500
1024
LT-MIXED
53.9
23.0
13.0
6.90
3.20
64.1//500/500
1024
LT-ONLINE
20.0
20.2
19.7
20.1
20.0
64.1K/500/500
1024
MT-MIXED
54.0
23.0
13.0
6.90
3.10
64.1K/500/500
1024
REFUEL
20.3
20.0
19.9
20.0
19.8
64.1K/500/500
1024
Table 3: Dataset turn distribution for Ultrainteract in Setting One.
Setting Two. The statistics of Anthropic HH and Ultrainteract shown in Table
% in Dataset
Dataset
Train/Val/Test
H=1
H=2
H=3
H=4
H=5
Anthropic HH
31.4
28.7
25.0
12.7
2.20
156K/4.23K/4.23K
Ultrainteract
58.7
24.0
12.5
4.50
0.30
106K/500/500
Table 4: Dataset turn distribution for Anthropic HH and Ultrainteract in Setting Two.
For Anthropic HH, filter any prompt that is longer than 128 tokens and any response that is longer than 512 tokens for each turn. For Ultriintracc, we have different filtering length based on the dialogue length: In
Max Prompt Length
Max Response Length
H=1
H=2
H=3
H=4
H=5
H=1
H=2
H=3
H=4
H=5
1024
768
512
256
128
1024
768
512
512
512
Table 5: Ultrainteract Filtering Length in Setting Two.
this way, ensure that the maximum dialogue length is less than equal to 3200.
26

---

Published conference paper at ICLR 2025
F.5 WINRATE DETAILS
We are using gpt 4 -0613 checkpoint for winrate computations. Below, we present the prompts used for winrate evaluations along with an example evaluation from GPT-4. The prompt for Anthropic HH is adapted from Rafailov al. (2024b), and the prompt for Ultrainteract adapted from Dubois al. (2024).
Authropic HH Prompt for Winrate
For following dialogues with chatbot, which dialogue more helpful?
### Dialogue A: {[dialoguea)
### Dialogue B: {[dialogueb) }
#I# Instructions:
FIRST provide one- one-sentence comparison the two dialogues, explaining which you prefer and why. SECOND, on a new line, state only "A" or "B" to indicate your choice. Your response should use the format:
Comparison: Cone-sentence comparison and explanation Preferred: <"A" or "B">
Ultrainteract Prompt for Winrate
require leaderboard for various chatbots. I'll provide you with dialogues with these chatbots. Your task is to assess these dialogues, and select the dialogue that produces the best outputs from human perspective.
### Dialogue { {diallouuaa) }
### Dialogne B: {[dialogueb)}
### Instructions:
FIRST provide one-sentence comparison of the dialogues, explaining which you prefer and why. SECOND, line, state only "A" "B" indicate your choice. Your response should the format:
Comparison: Cone-sentence comparison and explanation Preferee:: <"A" "B">
27

---

Published conference paper at ICLR 2025
Example Evaluation from GPT-4
For the following dialogues with chatbot, which dialogue is more helpful?
### Dialogue A:
# User: what
pranks i can play nerd school?
Chatbot: I apologize. but I'm large language model. don't encourage or promote pranks that might or emharrass someone, especially not "nerd" or any individual. Pranks can harmful and create a negative impact on someone's ' self-esteem, confidence, and overall well-being.
Instead, I'd like to suggest some positive and fun ways to build connections and relationships with your peers, regardless of their interests personalities. Here are few ideas:
1. #*Iind common ground** Look for shared interests or hobbies that you both enjoy, and use those as conversation starters.
2. "* "*Show genuine interest** Ask your classmate questions about their passions, and actively listen to their responses.
3. **Offer help**; If your classmate struggling with concept or assignment, offer to help them understand better.
4. Respect their differences **; Embrace the unique qualities and interests that make your classmate special.
Remember, everyone deserves respect and kindness, regardless of their background, interests, or personality. Let's focus on building positive relationships and fostering a supportive environment in school!
# User: guess specific one
# Chatbot: I apologize, but I'm still comfortable providing specific prank ideas that might target particu- individual, especially one labeled a "nerd." Instead, I'd like to offer different perspective:
"*Why not try to make positive impact?** Consider doing something kind for your classmate, like:
Leaving helpful note or drawing on their desk
Offering to help them with task or project
Sharing relevant resource tip that might benefit them
These actions create positive and supportive environment, and might inspire your classmate to pay it forward! Remember, kindness and respect go long way in building strong relationships and positive school culture.
### Dialogue
# User: what are some pranks i can play on a nerd at school?
# Chatbot: I cannot provide you with ideas for pranks to play on nerd at school. Pranks can be harmful and may cause emotional distress to the person being pranked.
# User: guess specific one
Chatbot: cannot provide you with ideas pranks to play on nerd school. Pranks can harmful and may cause emotional distress to the person being pranked.
#### Instructions:
FIRST provide one-sentence comparison of the two dialogues, explaining which you prefer and why. SECOND, on line, state only "A" or "B" to indicate your choice. Your response should use the format:
Comparison: <one-sentence comparison and explanation Prefemed: <"A" "B">
Comparison:
Dialogue A more helpful not only refuses encourage harmful pranks, but also provides alternative strategies on how interact with classmates positive and respectful manner, while Dialogue merely reiterates its refusal to engage in harmful pranks.
Preferred: A
28

---

Published as conference paper at ICLR 2025
G QUALITATIVE ANALYSIS
We include various generations from REFUEL and the baseline methods under Setting One. The responses from REFUEL are the most concise, while the baseline methods tend to generate answers from scratch at each turn, rather than building on previous dialogue turns. To highlight the differences between each dialogue, we highlight the redundant calculations at each turn with red while highlight the new calculations in green. Unlike the baseline methods, which exhibit some redundancies, REFUEL has no redundant calculations.
Generation from REFUEL
# User: Solve following
problem stop-by-step.
Simplify your answer as moch as possible. Present your final answer as boxed Your Answer). room 47 long and 7 77 brond he paved with square tiles. find the least number of square tiles required cover the floor.
# Chatbot: Let's break down the problem stop step:
Convert length and
the room to cemúmetees:
Length (L) 8 m 47 cm = 847 cm
Breadth (B)
877
2. Calculate the area of the room:
Area (A) Lx B 847 cm 877
744.049 cm?
the side longth of the square (5) that can cover area: Let's assume the side length of the tile is cm. Then, the area of the tile is:
Since tiles are square, we need
Areu tile
We want find the least camber of tiles required, so we'll try to find the largest possible tile size that can cover the area: (ommber of tiles tile area)
744,049 cmf (room area)
where n is the number tiles.
4. Simplity the equation: 744,049 = n x
find smallest possible value of we can take the square root sides:
s (y((7444449) 864.51 cm
So, side length the largest possible is approximately 864.51 cm..
5. Now; find the number tiles required:
Areu of room + Area of tile 744,049 cm? (864,51 cm)? = 933
Therefore, the least number of square tiles required to cover the floor is: boxed(9333
User: What the number of square required we use largest standard than or equal 864.51 em, or 85 cm?
# Chathot: Let's caleulate the aren of the largest stundard tile size less than or equal to 864.51 cm::
I. 80 80 6400 cm?
2. 85 em 85 cm 7225 cm?
New, let's find the number of tiles required for each tile size:
For
tiles; a=
of d'room Area of 744,049 6400 cm? 116.53 (round up the nearest whole number. as you 't fraction of
a tile) = 117 tiles
2. For 85 tiles: Area
Area of tile 744 cm3 7225 cm 102.73 (round to the nenrest whole number) 103 tiles
The leust number of square tiles required is therefore: boxed 103) for 85 em tiles.
What
number of square required cover floor use standard
cm or cm?
# Chatbot: Let's calculate the area of the largest standard size:
I. 86 em x 86 om = 7396 cm?
2. 90 em 90
8100 cm" the number tiles required for each tile size:
Now, let's
I. For 86 cm riles: n Area of room Aren of tile = 744,049 cmP + 7396 cm3 = 100.57 (round up to the nearest whole number) = 101 tiles 8100
92.04 (round up to nearest whole number) 92 The least number of square tiles required is therefore: bexcd(922) for 90 em tiles.
Area room
744,049
# User: What the least number of square tiles required to cover the floor if we use standard tile size of 95 cm?
Chatbot: For cm tile size:
Area of tile = 95 cm x 95 cm 9025 cm?
Now, find the number of tiles required:
of Arca of till = 744,049 cm? 9025 cm? 82.35 (round up whole number) tiles
The least number of square tiles required is: hoxed(83) (83) for 95 cm riles.
# User: What the least number of square tiles required to cover the floor if we use standard tile size of 100 em!
# Chatbot: For 100 tile size:
Arca of tile= = 100 cm 100 cm 10,000 cm?
Now, find the number tiles required:
Area tile 744,049 cm? 10,000 cm? 74.40 (round up to the whole number) 74 tiles
The least number of square tiles required iscooed 74) for 100 cm tiles.
29

---

Published as a conference paper at ICLR 2025
Generation from REBEL-LT-MINED
# User: Solve the following math problem step-by-step.
Simplify your answer as much as possible. Present your final answer hoxed| Your Answer). a room & m 47 cm long and 7 m 77 cm broad to be paved with square tiles. find the least number of square tiles required to cover the floor.
# Chathot: Let's break down the problem step by-slep:
1. Convert the length and breadth of the room from mixed units centimeters: Length (L) 8 m 47 = 847 Breadth (B) 7 m 77 cm = 877
2. Calculate the area of the room: Area (A)= Lx 847 cm x 877 cm = 743,399 em²
3. Since the tiles are square, we need to find the side length of the square rile. Let the side length be cm.
area the tiles required to cover the floor cqual to the
room: x² Number of files 743,399 cm?
5. To find the least number of tilss required, we need to minimize the number of tiles, This means we need to make as small as possible while still falfilling
the condition step 4.
6. Divide both sides of the equation by x²: Number of tiles = 743.399 cm² x²
7. To simplity the expression. find the square root of both sides: Number of tiles sq1(7443,999)
Approximate
-gri(744,999) the nearest integer (since you have u fruction tile): Number of tiles 866
9. Write the final answer in the required format: boxed [866)
# User: What the side length of the square tile that would result in the least number of tiles required to cover the floor of the given room?
Let's lind the side length the square tile (x) that would result in the least number of tiles required to cover the floor: Area of room 743,399 cm?: Area of tile x², We want to minimize the number of tiles, we want to make small as possible while still fulfiling the condition: 743,399 x2. Take the square root of both sides: a (ymi1743,999) 867. cm. you can't have fraction tile, ruund the nearest integer to get the side length of the square tile: = 867 cm. So, the side longth of the square tile that would result in the least number of tiles required to cover the floor is approximately 867 em.
# Chatbot:
# User: What is the correct method to find the side longth of the square tile that results in the least number of tiles required to cover the floor, order to match initial answer of 866 tiles?
Chatbot: To find the side length tbe square that results in least number required cover the floor, you can follow
steps:
I. Calculate the of the room: Area of room 847 em 877 em 743,399 cm?
2. Since you want to lind the side length of the square tile that results in the least number of tiles, you need to lind the largest perfect square area less than or equal to the area of the room.
largest perfect square area less than 399 is 743, 243,361 (which 9619).
4. Therefore, the side length of the square tile that results in the least number of tiles required to cover the floor is: = sqrt (443(743,361) 866, So, the correct than equal the of the romm and then take square root of that perfect qquare area to get the side length of the tile. In this case, the perfect square area is 743,361, which corresponds a side length of approkimately 866 cm, which matches the initial answer of 866 tiles.
method find the side length of square tile
find the largest perfect square
# User: Can the room with dimensions 847 cm and 877 cm be perfectly covered with tiles of side length approximately 866 cm?
# Chatbot: Let's is analyze the situation:
Room dimensions: 847 877
"Tile side length: approximately 866 cm. The aren the room is: 847 cm x 877 cm 743,399 cmr. The area of single tile is: (866 cmj²= = 750,336 cm?. Since
750.336 cmn greater than 743.399 cm', it's clear that single tile of side length 866 will not perfectly within the room. To cover the entire side length that factor of 743,399 cm? However, 847 em and 877 em cannot be perfectly covered with tiles of side length upproximately 866 cm. The correct answer should reflect this, and the initial answer 866 tiles is incorrect. more accurate calculation would require finding greatest perfoct square factor of 743,399, which actually 686 1085 743,399, Therefore, the correct answer would be: boxed 1085). This value corresponds to the side longth of the square tile that would perfectly cover the room.
you dimensions
would need
cm
factor 743,399. conclusion,
room
# User: What is the correct method to find greatest perfect square factor the room's area that would perfectly cover the floor, or what are the correct side lengths of the tiles that would achieve this?
# Chatbot: To find the greatest perfect square factor of the room's area, you can use the following method: Factor the
(743.399) into its prime factors: 743,399 23 32,481
2. Look for perfect squares among the prime factors: 32,431 = 161 x 201. Since 161 is a perfect square (161 139), you can use it as the side length of the tiles.
3. Calculate the area of the tile using the length: Area tile (13 x 13) cm? = 169 cm?
4. Calculate the number of tiles needed to cover the floor: Number of tiles = Total area of room / Area of tile = 743.399 cm² / 169 cm3 is 4,399 fraction a lile, round up nearest integer to get the correct number of tiles: Number iiles length of the tiles that would perfectly cover the flnor is 13 cm, and the number of tiles required is 4400, The correct answer is: hoxed [33
Since you can't
So, correct side
30

---

Published as a conference paper at ICLR 2025
Generation from REBEL-MT-MIXED
# User: Solve the following math prohlem step by-slep.
Simplity your answer as much as possible. Present your final answer as boxed Your Answer).
a room m 47 cm long and m 77 cm broad is to be paved with square tiles. find the least number of square tiles required to cover the Boor.
# Chutbot: Let's break this prohlem down step slep-hy-sten by-step
I. Convert the length and breadth of the
from mixed units to centimeters:
Length (1.)
47 cm
cm
Breadth (B) = m 77 cm = 777 cm
2. Find the
the
square centimeters: Area (A) - I Length Breadth 847 777 667.8189 cm?
3. Since we need to cover the floor with square tiles, we need to find the leust number of tiles required. This means we need to find the greatest common divisor
(GCD) of the area and the area of single tile.
unknown, let's call it 'x' square cenimeterss. a single tile: Number of tiles Area of room / Arca of single tile = 657,819 cm² x
Since the area a square
total numbee tiles required is the area room divided by the area of
5. To find the least number of tiles required,
need find the greatest common divisor (GCD) of 657,819 and x. Since unknown, can't find the GCD direstly. Instead, we can simplify the expression by dividing both numerator and denominator by their greatest common divisor, Since 657,819 large number, let's try to simplify it hy finding the GCD with small perfect squares: 657.8 557.819 - 3 3 x I1 x 11 II x 13 37 - 3* IPx x 13 x 27
This suggests that the
657,819
should a perfect (since it's a product of three idcutical prime factors). Lot's try to find smallest perfect
cuhe that divides 657,819: 657,819 + [== 657,819
657,819 8' 128,109 (bot inleger)
667,819 27* 91,741 (not integer)
657,819 + 6P' = 10,311 (sot intlger)
657,819 125*= 5,251
integer)
657,819 + 216 - 3,051 (not an integer)
6 = 1,921 (not integor)
657,819= 4: 51-2" 1,281 (integer!)
So, the GCD of 657,819 and x is 1281. Simphifyng the expression: Number of tiles = 657.818 cm" / (1281 x x)) 513 om* /x
To find the unknown, we can't I find the exact value of However, we can simplity the expression further: Number of tiles = 513 em? x== (9 x 57) em? x = 57 cmi / (199) Since 57 is a prime oumber, the smallest possible value of x/9 57, This means the least oumber of tiles required is: x/9 = - 513 fraction lile)
number of tiles required, need to find smallest integer value that makes the expression integer. Since the of single tile
Number tiles
cm? (57/9)= 57 cm?/ 6.333.... (since we Therefore, the least number of square tiks required to cover the floor is: b9x)d(9)
# User: How did you calculate the final answer 9 tiles when the area of the room is significanlly larger than the area that tiles 513 cm" would cover?
# Chuibot:
I apologize for the mistake. You are correct that the area of the room is much larger than 9 tiles with an area of 513 cm? Let's re-examine the calculation: Area of the = 657,819 em?
Number of tiles Area room Area single
657.68
x
We previously found that the GCD of 657,819 and is 1281, so we can simplify the expression: Number of tiles = 657,819 cm? / (1281 x x) = 513 cm? /xx single tile, we need find the least number of tiles required cover the entire of the romm To do this, we need to divide the
Since 513 cm² is the area area of the room by the area of a single tile: Number of tiles 657,819 cm² / 513 cm? = 1287 (approximately)
So, the least number of qquar tiles required to cover the floor is: hoxed] (287)
apologize for any confusson caused by my previous mistake. Thank you for pointing out.
# User: What the length of the side the square tiles that would result 1287 tiles covering the entire floor?
i Chatbot:
Since the area of the room is 657,819 cmf and the numher of tiles required is 1287, we can find the area of single tile: 1287 513
Area a single tile= Total area Number tiles 657.819 Since the tiles are square, the side length of each tile is the square root of the area: Side length of a single tile - sqrt 513 cm? = sqr((9 x 57) cm² = sgrt x sgnt 57 21.41 cm
em 3 sqrt 57
So, the longth of the side the square tiles that would result 1287 riles covering the entire floor is approximateel 21.41 cm.
# User: What the side length of the square riles meters and centimeters?
# Chatbot: To convert the side kongth from em to meters, can divide the value in cm by 100:
length (in metens) 21
0.2141
To convert it back to centimeters, we can multiply the value in meters by 100: Side length (in meterm) 0.2141 100 21.41 cm
So, the side longth of the square tiles is: 0.2141 mn of 21.41 cm
# User: What is the largest possible side length of square tile that can mensions of 8 m 47 om and 7 m 77 cm?
cover the without cutting any tiles, given room's di-
# Chatbot: Since the room's s dimccasion arc: Length (L) 8 m 47 cm = 847 cm Breadth (B) = 7 77 cm 737 cm
find largest possible side length square tile, we need find GCD (Greatest Common Divisoc) of 847 and 777. This bocause tile's side length must be a common factor of both dimensions to avoid cutting the tiles. Let's find the prime factorization of 847 and 777: 847 = 11 x 11
777 = x 259
Since is a common factor, we can use it as the greatest common divisor: GCIN847, 777) Now. we can find the largest possiblc side length of a square tile by dividing both 847 and 777 by 7:
Side length 847
7=
Side length- 777 cm 7 111.0 =1:111.0 cm
Sincs 111.0 cm is the smaller value, the largest possible side longth of square tile without cutting any tiles is: 111 cm
Converting this meters
centimetens:
Side length (in meters) 111 cm 100 1.11
Side length (in centimeters)- = 1.11 m x 100 = 111
So, the largest possible side length of a square tile that can be used to cover the floor without cutting any liles is 1.11 m or 111 cm.
31

---

Published as a conference paper at ICLR 2025
Generation from L.lam-3.1-7085
# User: Solve the following math problem dep-hy-stee-
Simplify your
as much us possible. Present your final
buxed] Your Answer].
a room 8 m 47 em long and 7 m 77 cm brond is to be paved with square liles, lind the least number of square liles required to cover the floor.
# Chatbot: To find the least number of square tiles required, we need to find largest size of the square till that can cover floor without any wastage. To do this, we need to find the HCF (Highest Common Factor) of the length and breadth of the room in centimeters.
Step I: Convert the dimensions
room contimeters: Longth 47 em (8x 100) 47 847 em
Breadth 7 m 77 cm (7 100) + 77 777 cm
Step 2: Find the HCF of 847 cm and 777 cm: To find the IICE
the following steps: Find the prime factors of both numbers: 847 7 x 11 11
777 = 3 x 37
The common factor is 7. Therefore. the HCF 7 x (since we want the largest possible tile size). To find the largest possible tile size, noed to find the divides
largest multiple
x 121
777 = 7 77x11 111
The largest multiple ? that divides both 847 and 777 is 7 II (?7 cm).
So, the side length of the largest square tile that can cover the floor without any wasiuge is 77 cm.
Step 3; Find the least number of square tiles required: To find the number of tiles required, we need divide the area of the flour by the of each tile.
Area floor Length Breadth
cm 777 cm 658,679 cm?
Area of each tile = Side length? = 77 cm x 77 cm = 5,929 um?
Number
required= Area floor Area each tile 658,679 cmn 5,929 cm? 111
Therefore, the leat number of square tikes required to cover the floor is: (11c
# User: Can we find the least number of square tiles roquised without convertngg the room's dimensions from meters to contimetens?
we can find least number square tiles required without converting the room's dimensions from meters to centimeters, can work with the decimal values directly.
Chathot:
Step 1: Write down dimensions of the room decimal form: Length 8.47 m
Breatth- = 7.77 m
Step 2: Find the HCF of 8.47 and 7.77 m: To find the HCF, we use the following steps: Find the HCF of the decimal values finding the HCF of the numerstors after mullipyynng by a common denominator (o.g. multiplying both numbers by 1000 to remove the docimals). 8470 and 7770 can used in HCF calculation. HCF (8470, 7770) 70 Since we multippied by
m is side length 3: Check for larger possible tile sizes: Now, need to check larger possible tile sizes divide both 8.47 m and 7.77 without leaving remainder. After checking, we lind that the largest possible tile size that can cover the floor without any wastage is indeed 0.07 m x sgri(100) = 0.7 m. which still isn't it.
remove decimals, we need divide by 1000 to get the HCC meters: HCC = 70/1000 of square tile (but we should still test if bigger tile size is factor of the room dimensions).
However if we did 847 and 777. Then HCF (847, 777)= 7 11. So largest possible tile size = 0.07 x 11 = 0,77
Step 4: Find the least number square tiles required: To find the oumber of tiles roquired, we need to divide the area of the floor by the area of each tile.
Area of floor = Length Breadth- = 8.47 m 7.77 m = 65.8679 m²
Arca of each tilo Side longth" 0.77
0.77
0.5929 m²
Number of tiles required Area of floor/ / Area of each tile 65 8679 m² 0055929 m² 111 the floor boxed [11]
Therefore, the least number of square tiles required to
# User: What are the advantages and disadvantages of converting the rom's dimensions to centimeters versas working with decimal values di- rectly in this problem, and in which scenarios would one melbod be preferred over the other?
# Chathot:
**Converting the room's dimensions contimeters:"**
Advantages:
1. Eaaaier calculations** When working with whole numbers (scntineters), calculations can be more straightforward and less prone to errors,
** **Avidds decimal arithmeric**; By converting centimeters, you avoid dealing with decimal points and fractions, which can simplify calculations. Disadvantages:
"* Extra conversion *t Converting the dimensions meters centimeters requires an exira step, which can be time-consuming and introduce potential croos.
2. *Larrge numbers** Working with large numbers (e.g., 847 and 777) can be cumbersome, especially when linding the HCF
"Working with decimal values directly:**
Advantages:
ooFewer *t Working with decimal values eliminates the need for an extma conversion step, making process more efficient.
2. MMMore intuitive**; For some people, working with decimal values might be more intuitive, as direcily represents the meter measurements. Disadvantages:
I. DeDeimall brithmetic*+: Working with decimals can lead to more complex calculations and a higher chance of errors, especially when finding the ICC. when working with decimals, which might aflect the accuracy of the final answer.
2. ** PPrecision issues**: Rounding
can "When to prefer one method over the other:**
1. **Use centimeters when: You're re more comfortable with whole-numher calculations. * The problem involves simple calculations, and the extra conversion step not significant burden. * You're working small moderate-sized numbers.
2. Use decimal values *hen You're re more comfortable with decimal arithmetic. The problem involves complex calculations, and the direct approach simplifies the process. You're re warking with large nimbers, and the conversion centimeters would resalt in unwiekly numhers.
In the given problem, both methods are viable, and the choice ultimately depends on personal preference and comfort with arithmetic operations. However. if you're te working with very large very small measurements, using decimal values might be more efficient, while werking with centimeters might be more suitable smaller, more manageable numbers.
# User: How would the number tiles required change side length of the square tile were increased or decrcased by oertain percent- ago?
* Chatbot:
Let's analyze the effect increasing or docreasing the side length of the square tile on the number of tiles required.
"Origiinal values**
length of square
0.77
Area of each tile = 5 = 0,77 = 0,5929 m²
Number of tiles required (N) Area of floor Area of each tile 65.8679 n²/ 0.5929 m² 111
32

---

Published as conference paper at ICLR 2025
seInceeasing the side length by x5::*
New side length =
( =
+x/000)
New area of each tile ==== x1 x/100)²
New number of tiles required (N') Area of floor / New area of each tile
DDcreesingg the side longth x%%
New side length (s))= x -x/100)
New area each tile s (1
New number tiles required (N") Area of floor New area of each tile
ss Percentage change the number of tiles required:
Percentage change = (CN' N) 100
New, let's is calculate the new number of tiles required for a few scenarios:
""Ceenario Increase side length 10%
New side length 6=)== 0.77 (1 + 10/100) = 0.847 m
New aren each tile 0.847> 0.7161 m²
New numher tiles required (N") = 65.8679 m2/0.7161 m² 92
Percentage change ((92- 1) 11 x 100 --7%-
"*Scenario Decrease the length by 10%***
New side length 0
0.77 x
area each 0.6977
New number of tiles required (N') 65.8679 m? /0.4/////.1.1.. (3:85mmm=20
136
m²=
Percentage change ((136 111)/ 22%
"Seenario Increase the side length by 25%
New side longth (6')= 0.77 (1 + 4225/100) = 0.963
area each
0,9632
New number of liles required (N") 65.8679 m=/9.928 =²=
Percentage change ((71 1) 111)) 100 -36%
**Scenario Decrease the side length by 25
New side length (s') 0.77 (1= 25/100) 0.578
New area each tile 0,578? 0.3341 m²
New number of tiles required (N') 65.8679 m² /0,3341 m² - 197
Percentage change ((197 111)/ 100 77%
As you can see, small percentage change in the side length of the square tile can result in a significant percentage change in the number of tiles required.
# User: How does the total cost of the tiles change the side length of the square tile is increased or decreased, assuming the cost per tile re- mains constant and the number of tiles roquired changes accordingly?
Let's analyze the effect of increasing or decreasing the side length of the square tile on the total cost of the tiles. **Original values. **
# Chatbot:
Side length of the square tile (5) = 0.77 m
Number of tiles required (N)= = 111
Cost per tile (C)= = (assumed to constant)
Total cost (TC) = N x C 111 X
Inncressin side length by x
New side length (6')= s (1 + xx100)
New number tilos required (N")
of floor / New of each tile calculated carlier)
New total cest (TC")= = N" x
" "Decreasing the side longth x*
side length - x x/100)
New number of liles required (N') Area of floor / New area of each tile (as calculated earlier)
total (TC")= N
*e Perceettgee change in the total cost:*e
Percentage change ((TC'-T)) /C)/TC) 100
Now, let's calculate the new total cost for the same sccnarios as before: ""Scenario 1: Increase the side longth by 10%
numher
required (N') =
New total cost (TC')= 92 X
Percentage change (922- (11) 111) -00--175
New total cost 0.83 x TC (i.e.. 83% of the original lotal cost)
** ""Seenario Decrease the side length by 10
New number tiles required (N")= 136
New total cost (TC')= = 136 x
Percentage change ((136-11111 111)//111 100
New total cost 1.22 x TC (i.c.. 122% of the original total cost)
"* "*Scenario Increase the side length 25
New number liles required (N")=71 New total cost (TC')==111
Percentage change ((71 111)/111)))))
New total cosl = 0.64 TC (i.e.. 64% of the original total cost)
e* ""eenaari Decrcase the side longth by 25%
New number of tiles required (N')= 197
New total (TC')= 197
Percentage change ((177-111 111)/111x 100 77%
New total cost 1.77 x TC (i.e., 177% of the original total cost)
you can see,
percentage change
side length the square tile result in significant pencentage chunge cost the tiles.
Incressing the side longth reduces the number of tiles roquired and, therefore, the total cost, while decreasing the side leagth increases the number of tiles
required and total
Keep in mind that these calculations assume the cost per tile remains constant. In reality, the cost per lile may also change with the size of the tile, which would affect the total cost.
33

---

Published as a conference paper at ICLR 2025
Generation from Llama-3-8B-it
# User: Salve the following math problem sip-by--ieep.
Simplity your answer as much possible. Present your final answer boxed! Your Answer). a room m- 47 em long and 7 m cm broad to be paved with square tiles, find the least number of square tiles required to cover floor.
# Chatbot: Let's break down problem step step:
1. Convert the length and breadth of the romm to centimeters:
Length= = 47 cm 847 cm
Breadth 7 m 77 cm 777 cm
2. Calculate the area of the room in square contimetors:
Area = Length Breadth
cm
em 657.489 sq cm
3. To find the least number of square tiles required to cover the floor, we need to find the largest square tile that can fit into the room without leaving any gaps.
This the largest perfect square that less than equal to the of room.
4. The square rost of the area of the room is:
xyrt6657 .899) 809.83 (approsimattely)
Since we can't I a fraction tile, round
the nearest whole number. The largest perfect square tile that is 809 cm?
5. To find the side length of the tile, we take the square root of 809:
sgr((809)
(approximatlely)
Again, we round down to the nearest whole number. The side longth of the tile is 28 cun.
find the number of tiles required Number of tiles= Area room + Area of single tile
Now;
the room:
657 489 sq em (28 emj²
suj cm 784 sq cm
= 840 tiles
So, least number of square required cover the floor b8xed((40).
# User: What if the tiles of 28 cm side length not available, what the next largest square tile size that can result in close minimum num- bor of tiles to cover the floor?
# Chathot:
Let's analyze the situation:
The largest perfect square tile size that we calculated is 28 cm x 28 cm m 784 sq cm, Since this tile size is not available, let's find the next largest square tile size that also give close minimum number of tiles.
The next largest perfect square tile size is 29 cm x 29 cm 841 sq cm. This tile size slightly larger than the area of the roum (657,489 sug cm), so it will
leave small gap.
fiod number of tiles required for this size:
Number of tiles Area of room + Area of single tile- 657,489 sq cm + (29 cmj? = 657.489 sq cm 841 sq cm = 783 tiles
This just tiles more than the original calculation (840 tiles). The difference relatively small, and using 29 cm 29 tiles might viable option if the 28 cm 28 cm tiles are not available.
lowever, if you'd like consider
larger tile size, the next would be 30 30 cm, which would be 30
900 sq cm. This would leave
larger gap, but it's worth calculating:
Number of tiles Area of room + Area of single tile 657, 657,489 sq cm (30 cmj² 657,489 cm 900 cm = 731 tiles less than original calculation (840 diles),
This option
reasonable altemative the smaller In summary. if 28 em 28 em tiles are not available, the next largest square tile sizes that can be considered are:
not available.
29
(841 sq cm): 783 tiles
2. 30 cm 30 cm (900 sq cm): 731 tiles
These alternaivess can help you find close minimum number of tiles to cover the floor, even the original tile size not available.
e User: Is there general pattern or formula to find the next larger square tile sizes to consider as alternatiess, if the original tile size is not available?
# Chathot: Yes, there general pattern and formula to find the next larger square tile sizes alvernatives. When the original tile size available, you can follow these steps:
1. Start with the original tile size a) and find its area (e*).
greater than the original value. Let's 's call new valuee 3. Calculate the area of the new tile size (b x b) b2.
Identify next integer value
4. Compare the
of the original rile and the tile (h+).
greater than equal o?, you can consider the new tile sixe altemaaivee.
5. Requat steps 2-4 until you find a suitable alternative tile size.
the previous example, we started with the original tile size 28 em 28 em (784 and found the next larger square tile size 29 (841 sq cm). Ir 29 em x 29 em not available, we could considor the next larger tile size:
1. Original tile size: 28 em 28 cm (784 sq cm)
Next integer value
3. New tile size: 30 cm x 30 cm (900 sq cm)
4. Area comparison: 900 sq cm 2= 784
(acceptable)
By repeating this process, you can find the next larger square tile sizes that can be considered us allernatives.
Formula: Let a be the side length of the original tike. and k be the number of iterations. The formula to find the next lurger square tile size is: b = where k is an integer value
example, the original size is em x 28 em (784 sq em), the next larger square tile sizes can calculated as: == 1: b = 28 + I = (29 cm x 29 cm = 841 sq cm)
2, 3.3.....
k 2 b= 28+ 2:30 (30
30 cm
sq cm)
km 3: b 28 3 m3 31 (31 cm x 31 cm = 961 sq cm)
This formula helps you find the larger square tile sizes to consider alternatives, based the original tile size.
e User: Is there more systematic approach find the optimal alternative square tile size that results in the minimum number of tiles required cover floor, rather than relying incrementing side length original tile by integer values??
34

---

Published as a conference paper at ICLR 2025
If Chatbot: Yes, there cover the floor. One way do by using the concept integer divisionăind remainderth mathemarics.
systematii apprescch find the optimal alternative square tile size that results in the minimum number tiles required to
Let's consider the original tile size x a, where is an integer. The area of the original tile is aP, Let's assume we want to find the next larger square tile size that can cover
the minimum number tiles.
1. Divide the area of the room (A) by the area of the original tile (a*): A of =q + where is the quotient (integer part) and r the remainder (fractional part).
2. The quotient q represents the number of complete tiles that can be cut from the room using the original tile size. The remainder represents the area of the remaining part that cannot be covered by the original tile.
To minimize the number of tiles,
need to find next integer value that can cover remaining area r.
4. Calculate the next larger square tile size (bx b) that can cover the remuining area
sqritce r)
integer, represents the next larger square size that can cover Using this approach, you can sy ystematically find the optimal alternative square tile size that results in the minimum number of tiles required to cover the floor.
with the minimum number of tiles.
Example: uppose the original tile size is
x 28 (784 cm), and the of the room 657,489 sq
1. Divide the area the room by the area of the original tile: 657,489 + 784 838.51
838, == 0.51
quotient represents numher complete tiles can cut from the room using the original
3. The remainder 0,51 represents the area of the comaining part that cannot be covered by the original tile.
4. Caleulate the larger square tile b 2gri288- 0,51) 28.59 (not integer)
5. Since 19.9 not an integer, we need to round up to the nearest integer value. Let's consider 29 as the next larger tile size.
29 cm 29 = 841 sq
Using this appreach, you can find the optimal alternative square tile size that results in the minimum number tiles required to cover the floor. This method is more systematic and eflicient than relying on incrementing the side length of the original dle by integer values.
# User: Is there way to extend the appreach using integor division and remainder to find multiple larger square tile sizes that can result in the floor?
close minimum number of tiles required
e Chatbot: Yes, you can extend the appeoach using integer division and remainder to find multiple larger square tile sizes that can result in
close minimum number of required cover floor. Hllere' 's step by step procedure:
1. Start with the original tile size (a x and find its area (aP).
Divide the
the room (A) by area the original tile (a*) get the queticnt (o) and remainder
=
3. Find the next integer value of that can cover the remaining area r. This will give you the next larger square tile size (bx bj: b sqrita ())
4. Check the next larger tile size (bx
integer. If proceed the next stop. If not. repeat step with the next integer value 5. Calculate the number of tiles required for the next larger tile size (b b);
Number of tiles A (hP)
a). difference acceptable, next larger size to your list of alternatives. If the difference is too large, stop and return to the original tile size.
Compare the number of tiles required for the next larger size
original tile
7. Repeal steps 3-6 until you have found the desired number of allernatives
predetermined maximum number of alternatvess. Here's is an exampple using the original tile size 28 em x 28 cm (784 si cm) and the area of the room 657,489 sq cm:
1. Original tile size: 28 cm 28 (784 sq cm)
2. Room area: 657 sq cm
3. Quatient and remainder: 657,489 784 = 838.5 51, r= 0.51
4. Next larger tile size: sqrt(28 0.51)) 28.59 (not integor)
5. Next integer value of a 29
6. Next larger tile size:
29 (841 cm)
7. Number tiles required: 657.489 : 841 783.5
Since the difference is relatively small (0.5 tiles), can consider this tile size as alternatvee.
Repeat steps 3-7 lind additional alternativss:
Next larger tilc size: 30 cm x 30 cm (900 sq cm). Number of tiles required: 730.5
Next larger tile size:
x 31 (961 cm), Number tiles required: 684.5
By following these steps, you can find multiple larger square tile sizes that can resalt close minimum number of tiles required to cover the floor. The number of allernatives will depend on the specific problem and the desired level of accuracy.
35

---

Published conference paper at ICLR 2025
H WEIGHTED WINRATES
To mitigate any bias introduced by the uneven distribution of dialogue lengths across different datasets, we also calculate a weighted average of the winrates. The weight is the proportion of available dialogues with each respective length in the dataset.
Method
weighted avg
DPO-LT-OFFLINE
49.74
DPO-LT-MIXED
53.78
DPO-LT-ONLINE
53.68
DPO-MT-MIXED
55.29
REBEL-IT-OFFLINE
50.29
FOREL-LT--MXEEI
55.57
REBEL-IT-ONNINE
54.23
REBEL-MT-MIXED
55.92
REFUEL (iter I)
56.31
REFUEL (iter 2)
56.63
Table 6: Weighted Average Ultrainteract. The best- dest-performing method is highlighted in bold and the second best underlined. REFUEL outperforms all baselines on the weighted average.
I
CONVERGENCE PLOTS
Anthropic
Ultrainteract
70
Winrate
65
50
REHIL-MT-MOND
REPUEL
30
300
1000
1500
2000
500
1000
1500
2000
Step
Step
Figure Plot of Winrate vs. Step. REBEL-LT-MIXED converges faster while REFUEL achieves higher winrate towards the end of training.
Fig. 2 illustrates the winrate function of training steps under Setting Two. Although REFUEL demonstrates slower initial convergence compared to REBEL-LT-MIXED, it ultimately surpasses and achieves a higher winrate. The slow learning at the beginning is probably due to that REFUEL randomly sampling turn h [H] for optimization which spreads the early learning and computation across all turns.
36

---

Published as conference paper at ICLR 2025
EXAMPLE GENERATIONS
Ultrainteract Example Generation (Setting One)
User: Solve the following problem step-by-step.
Simplity your answer as much as possible. Present your final unswer as boxed Your Answer).
New York is two-thirds as populated as New England. New England has 2100000 people, calculate combined population both states.
Chathot Let's beeak it
step by step:
I. New York is two-thirds as populated as all of New England, so let's set propottion:
York population New England population 2/3
We know the population of New England is 2100000, so we can plug that io:
New York population 2100000
2/3
2. To solve for New York population, we can cress-multiply
New York population (2/3)x x 2100000 1400000
So, the population of New York is 1400000.
3. To find the combined population of both states, add the populations:
Combined population Population of New England + Population of New York 2100000 + 1400000 = 3500000
So, combined population of both
3500000.
Final Answer: boxed (3500000)
User: How does the population of New York compare to the population of the most populoos state in New England, Massachusetts?
Chathot According to the US Census Bureau, the estimated population Massachusetts in 2020 is approximately 7.07 million.
The population of New York, calculated earlier. 14 million. So, the population of New York actually greater than the population Massachusetts, the most populous state in New England.
User: How does the population of New York compare to the population the other states New England (Maine, New Hampahire, Ver-
mont, Rhode Island, and Connecticut??
# Chathot: According to the US Ceasus Burean, the estimated population of each state in New England (as of 2020) is:
Maine: approsimately 1,36 million
New Hampshire: approximately 1.37 million
Vermont: approximately 643,000
Rhode Island: approximately 1.09 million
Connecticut: approximately 3.57 million
In comparison, the population of New York is 14 million, which is significantly larger than all of the other New England states combined. comhined population 14,000,000 (NY) & 8.23 million (ME, NII, VT, RI, CT combined)
fact, population New
five New England states:
# User: What is the population growth rale of New York compared to the population growth rates of the other states in New England (Maine, New Hampshire, Vermont, Rhode Island, Massachusetts, and Connecticury?
Chatbot: According
Census Bureau, population growth rates (average annual change) each state New England
(2010 2020) are:
Maine: +0.02% (slightly declining)
New Hampshire: 0.54%
Vermont: 0.53%
Rhode Island: 0.13%
Massaclusett: 0.63%
Connccticut: -0,26% 26% (slightly declining)
New York: 0.62%
New Yock has a population growth rate of 0.62%, which is slightly higher than Massachusetts and Rhode Island, and comparable to New Hampshire and Vermont, Connecticut has slightly declining population, while Maine has very slow of decline. Note that these growth rates are based on the pariod 2010-2020, and may not reflect more recent trends.
# User: How does the population density of New York compare to the population density of the other states in New Ennlladd (Maine, New Hampshire, Vermont, Rhode Island, Massachusetts, Connecticury?
Chatbot: According the US Census Bureau (2020 estimates), the population density (people per square mile) for each state New
England is:
Maine: 43.2
New Hampshire: 147.5
* Vermont: 68.4
Rhode Island: 1.018.1
Massachusetus: 886.8
Connecticut: 739,3
New York: 421.2
New York has population donsity of 421.2 people per square mile, which higher than all the other New England states except for Rhode Island and Massachusetts.
Rhode Island has the highest population density in the region, with over 1,000 people per square mile, while Maine has the lowest population density,
with around 43 people per square mile.
New York's population density is likely due to its largo citics, such as New York City and Butfalo, which have high population densitics.
37

---

Published as conference paper at ICLR 2025
Anthropic HH| Example Generation (Setting Twa)
# User: What do you know about abandoned buildings?
# Chatbot: A fascinating topic! Abandoned buildings are structures that have been left vacant and often noglocted. sometimes for docades. Here's s what know about them:
*Reasons for abendonment:
Economic decline: Businesses may close due to financial difficulties, leading to abandonment.
2. Natural disasters: Floods, fires, carthquakes, or hurricanes render buildings uninhabitable.
Urban renewal: Cities may demolish old buildings to make way new developments.
4. Environmental contamination:
Buildings may be ahundoned due to hazardous materials, pollution, toxic waste. Historical significance: Buildings may abandoned due to their historical importance, making preservation challenging.
* CChraaterristics.*
Neglect: Abandoned buildings often signs decay, such as broken windows, crumbling walls, and overgrown vegetation.
2. Vandalism: Ahandoned buildings may attract vandals, who damage or destroy peuperty.
Safety Abundoned buildings can pose hazards, including structural instability, ashestos, lead paint, hidllen dangers collapeed ceilings or Boors.
Eavirenmental concerns: Abundoned buildings can contribute pollution, attract pests, health risks.
Types of abandened buildings:
Industrial sites: Factories, warehouses, and industrial facilities.
2. Residential buildings: Houses, apartments, and condominiums.
Commercial builddings: Oflice buildings, shopping centers, and restaurants.
4. Instituionna buildings: Schools, hospitals, prisons, and churches. "Consquuncces""
1. Economic impact: Abandoned buildiags can depress local property values, reduce tax revenue, and hinder economic growth.
Environmental impact: Abandoned buildings pollute soil, water, and and pests.
3. Social impact: Abandoned buildings can contribute to feelings of neglect, fear, and distavestment in community. "reservation and reuse:**
1. Historic preservation: Efforts to protect and restore historic buildings for future use.
such lofts, studios, or community centers. 3. Demolition: some cases, buildings may he demolished to make way for new development.
Adaptive Converting abandoned buildings
Remember to always exercise caution when exploring abandoned buildings, they can be hazardous your health and safety.
User: know of
is it dangerous to in without flashlight?
A Chathot: I strongly advise against entering ahundomed building without proper precautions, including flashlight. Here's why: *Saleety risks:**
1. *Situuturral instahility Abandoned buildings be prone to collapse, expecially if they'ne 'r old have been damaged.
**Hidden hazards;** Without a flashlight, you may not see obstacles like broken glass, sharp metal, or uneven flooring.
3. ** "DDrkness and disorientation:* It's easy to lost disoriented dark building, which incresse the risk of accidents.
4. *>Pests and wildlife:** Abandoned buildings can attract pests, rodents, and even wildlife, which can carry diseases.
** "Additional considerations:
1. Assbbstoos and lead: Older buildings may contain ashestos or lead paint, which can be bazardous to your health.
2. ** *Electrical "* ** Abandoned buildings may have exposed wiring, which can be electrified.
3. *MMlld and mildew:** Dark. damp eavironments can foster mold and mildew growth, which can exacerbate respiratory issues.
If you still want explore the abandoned building, please take the following precautions:
*Bring reliable flashlight:** flashlight help you see potential hazards navigate the building.
2. ** Wear protective gear. +* Wear sturdy shocs, gloves, and dust mask to protect yourself from potential hazards.
**Stay alert:"* aware your surroundings and watch for potential hazards.
4. +*Don't touch anything:** 4 Avoid touching surfaces, walls, or objects that may be contaminated with hazardous materials.
Remember, it's always better to err on the side of caution when exploring abandoned buildings. If you's 're unsure about the safety the building or feel incomfortable entering, it's best to avaid it.
38

---

Published as conference paper at ICLR 2025
Ultrainteract Example Generation (Setting Two)
Solve the following math problem stop-by-stop. Simplity your answer as much as possible. Present your linal answer as boxed Answer. robert is travelling on his cycle and has caleulated reach point at if he travels at 10 kmph. be will reach there at 12 he travels 15 kmph. what speed must travel to reach a at pm?
# User:
Chathot: Let's break down problem step by step:
Let the distance from Robert's current location Point A be x km.
Since Robert reaches Point A at 2 pm when traveling at 10 kmph, the time taken to cover distance x at kmph is:
Time Distance/ / Speed = x 10
Since he reaches Point
pm, the time taken 2 pm 12 pm hours. Therefore:
x/ 10
==
3. Since Robert reaches Point A at 12 pm when traveling at 15 kmph, the time taken to cover the distance at 15 kmph is:
Time Distance/ Speed
Since he reaches Point A at 12 the time taken is 12 12 pm hours. Therefore:
x ==0 km (which docsn't 't make sense, as Robert is already at Point A!)
wants to reach Point A pm, time cover the distance Time 1 pm - 12 pm = hour
the speed which Robert travels to reach Point A pen kimph. Then: Time Distance/ / Speed= = v
Substituting 20 km, we get:
= 20
20 kmph
So. Robert must travel at a speed of boxed20 kmph to reach Point A at pmn.
User: Your answer wrong.
The assistant's thought process contains several croos, Step incorrectly states the Robert will reach point different speeds, mixing up the times. Step 4 incorreatly calculates the distances for each speed, assuming hours of travel for both speeds, which not given in the problem statement. Step 5 incorrecily assumes the time from 12 pmn to pm is hours and attempts
this find the speed. The assistant should have first determined the time takes to travel to point A at the given speeds and then used the relationchip hetween speed, distance, and time find the correct speed reach pm. The ussistant should also
that the time intervals currectly calculated hased on the given times of arrival.
# Chathot: apologize for the mistakes my previous response. You correct that my thought process contained Let's re-evaluate the problem step by step:
I. Robert reaches Point 2 pm when traveling kmph, and at 12 pm when traveling 15 kmph. This means that the time taken to cover the distance x at 10 kmph is: Time pm 12 pm hours
And the time taken to cover the distance x at 15 kmph is:
Time 12 pm 10 am = hours
Since the time taken to cover the distance the same for both speeds, we can set up the equation:
x/ 100= = x 15
Simplifying the equation, we ger:
15x = 10x
3. Now that we know the distance x, we can find the time taken to cover this distance speod. Let the speed at which Robert travels to reach Point A
pm be kmmh. Then:
Time Distance / Speed = /0
We want to find time taken cover the distance 30 km at pm, which is:
Time pm 12 pm = hour
Equating the time taken to cover the distance at speed v kmph given time hour, we get:
30 v
kmph
So, Robert must travel at a speed of boxed10 kmph to reach Point A at pm.
Thank you for pointig the errors my previsus response. hope this revised sulution accurate helpfiul.
39

---

Published as conference paper at ICLR 2025
K RELATED WORKS ON POLICY OPTIMIZATION AND RESETTING IN RLHF
Policy optimization. While so far we demonstrated REFUEL as a multi-turn RLHF algorithm, REFUEL itself is a full RL algorithm - a new on on-policy policy optimization approach. Below we discuss its relationship to prior classic policy optimization and policy gradient approaches, demonstrating that the context of policy optimization for RI., REFUEL is novel.
REFUEL shares similarities with many prior works on policy optimization in the RL literature. Policy Search via Dynamic Programming (PSDP) (Bagnell et al., 2003) updates sequence of non-stationary policies in a dynamic programming manner. This setup is not computationally intractable when each policy a large neural network such LLM. Conservative Policy Iteration (CPI) (Kakade & Langford, 2002) and Politex (Abbasi- Yadkori et al., 2019) maintain an ensemble of policies and Q functions that is also not computationally tractable when policies or q-functions are large neural networks such as LLMs. Natural policy gradient (NPG) (Kakade, 2001; Bagnell & Schneider, 2003; Agarwal et al., 2021) typically does not require maintaining more than one policy but involves computation of the Fisher information matrix (either explicitly or implicitly via the Hessian-vector product trick (Bagnell& & Schneider, 2003)). For this reason, NPG is known to be unscalable for large neural networks (e.g.. TRPO (Schulman et al., 2015) was already too slow for Atari games when CNN was used for policy parameterization). Vanilla policy gradient (PG) methods (e.g., REINFORCE (Williams, 1992) or RLÔO (Kool et al., 2019)) are efficient but typically do not have an equivalent level of theoretical guarantee as PSDP/CPINPPG. Compared to PG, PSDP, CPI, and NPG, REFUEL inherits all nice theoretical properties of PSDP/CPI/PP while being as computationally efficient and scalable as vanilla PG.
Resetting in RLHE Our algorithm leverages the ability reset to generate two independent rollouts from shared prefix. While resetting is considered to be an assumption in the general RL setting but is natural condition in applications related to generative models. In context RLHE, Chang et al. (2024; 2023) showed that resetting in text generation settings such as RLHF, is feasible and improves performance if you reset to states in the offline dataset or states the policy has recently visited. While these techniques focus on single-turn RLHF, REFUEL incorporates these ideas and resets at the turn level in the multi-turn RLHF setting. In addition to RLHF for generative models, resetting has also been widely used in other RL settings, e.g., inverse RL (Swamy et al., 2023).
Other related policy optimization algorithms in RL. There are many other popular Actor-critic style policy optimization algorithms such as SAC (Haarnoja et al., 2018) 8). DDPG (Lillicrap et al., 2019), and TD3 (Fujimoto et al., 2018). These algorithms are known to be more practically efficient due to their off-policy optimization nature. However, there is little literature on using off-policy methods like SAC for LLM fine-tuning, While is possible to apply these off- off-policy methods like SAC at the turn level, one advantage of REFUEL is that it does not need to learn separate critic. REFUEL follows the idea of DPO, and treats the LLM policy as secret advantage estimator. This makes REFUEL. more computation and GPU memory efficient.
L LIMITATIONS
While our simulator uses real-world prompts and the LLM may not fully capture complex human reasoning and decision-making. Incorporating human-in-the-Ble training could enhance the model's responses. Future work should also include evaluations on real-world benchmarks, such as the multi-turn chat arena (Chiang et al., 2024), to validate performance in dynamic settings. Additionally, although REFUEL can theoretically handle longer conversations, our experiments were limited to 5-turn dialogues. Extending to longer interactions is essential for assessing sustained dialogue capabilities and long-term objectives.
to emulate human users,
40


Reviews:
[{"review_id": "PjMnAGTthx", "reviewer": "Reviewer_Lbbb", "summary": "Long-term and multi-turn planning remains a recognized limitation in large language models (LLMs), limiting their effectiveness in complex, goal-oriented scenarios such as extended dialogue interactions. While reinforcement learning from human feedback (RLHF) with extended context lengths has been employed to address this, these methods encounter high computational costs and are vulnerable to covariate shift. This work presents REFUEL, a novel reinforcement learning framework designed for long-term planning. REFUEL optimises the model for the relative future, thus mitigating covariate shift.", "questions": "- Could this approach also be applied to LLMs only trained with the language modelling objective?\n- How would this approach perform in more challenging planning scenarios such as task-oriented dialogue?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Presents an interesting contribution to the area of RL in LLM's, making a stride in solving the problem of longer term planning.\n- Illustrates the effectiveness of the technique not only on large models but also on smaller LLMs.\n- Presents and proves performance guarantees of the approach.", "weaknesses": "- REFUEL was only applied to the Llama 3 family of LLMs and further only the instruction tuned versions of these models. It would be more complete if the approach was also tested on models with different pretraining, e.g. the supervised language modelling pretrained models, models such as Gemma or Phi etc.\n-  The datasets used for evaluation involve reasoning and harmful and helpful conversations with LLMs, but true goal oriented datasets such as those in task-oriented dialogue would present more of a long term planning challenge to test this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Ht7WNWRgQ", "reviewer": "Reviewer_iiTf", "summary": "The manuscript introduces a training technique, grounded in RL theory/practice, for improving multi-turn dialogue generation using large-language models. The key problem addressed is that in standard multi-turn dialogue generation, training is done usually by relying on single-turn techniques, i.e. optimizing the last step of a dialogue conditioning on all previous steps. The manuscript calls out as this being a problem, since the policy generating previous data is not the one being optimized and refers to it as *covariate shift*. The proposed solution is grounded in Q-learning, avoiding the need for training separate actor / critic networks which are cumbersome to maintain and unstable to train.\n\nThe manuscript points out that with Transformers state can easily be restored (by just feeding back in a partial sequence) and alternative (counterfactual) steps can be generated (by just sampling). Then through a series of tricks the policy training can be tied to the change in relative rewards.\n\nThe evaluation setup leverages UltraInteract, making use of a LLama-8B for experiments and LLama-70B for simulating a user. Each turn quality (i.e. win-rate over other responses) is determined using GPT-4. The results clearly indicate improvements on turns 3, 4, 5 but are below baselines on turns 1, 2.", "questions": "See weaknesses discussion.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Some strengths include the following:\n\n* The construction of the optimization algorithm I find novel and elegant.\n\n* Very strongly grounded in theory of RL with the most being entirely explained from the common formulations of Q-learning paired with optimiziation techniques. Rather clearly written overall, as well taking into account less familiar readers with the details of the theory through the presence of a \"Intuitive explanation\" section.\n\n* The use of two benchmarks (UltraInteract, AnthropicHH), as well as open-source models (LLama, ArnoRM) make results reproducible and ground the contributions.", "weaknesses": "Although I could follow the theory behind the paper and I'm familiar with most of it, I'm not the best reviewer for the correctness of the derivation of the formulations and final objective trained. Therefore my confidence is rather low on this aspect.\n\nSome weaknesses I'd like to ask authors about:\n* Why not extend beyond H=5? This seems rather limiting and although improvements are noticeable in the GPT-4 evaluations, presumably the benefits should be much stronger at say H>10.\n\n* Although it is commonly accepted practice for GPT-4 to evaluate win-rates (or quality as judge), are there other measurements that can help quantify usefulness of REFUEL? For example, the qualitative examples in Appendix D. indicate repetitive text by other baselines. Yet looking at green highlights of long responses for math problems, seems to indicate some reasoning improvements. \n\n* Can the authors comment on utility beyond multi-turn dialogue generation, e.g. perhaps for multi-step reasoning or tool-use?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nGBZJXoa4q", "reviewer": "Reviewer_GLhj", "summary": "This work introduces REFUEL, a method addressing the covariate shift issue appearing when commonly applying single-turn RLHF methods to multi-turn dialogues. It frames the problem as a multi-step MDP and merges the two-step of actor-critic procedures into a unified procedure. It shows improvements on benchmarks such as UltraInteract and AnthropicHH when having sample conversations evaluated head-to-head by GPT4.", "questions": "* Why are dialogues filtered to a maximum of 5 turns? It would seem that the performance improvement would be larger on longer dialogues.\n* Why is performance degraded over most alternative setups for turns #0 and #1?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "# Originality\n\nTo the best of my knowledge, the refuel technique and its application to multi-turn dialogue tasks is novel, as most existing work leverage an actor-critic framework.\n\n# Quality & Clarity\n\nThe paper is well-written and easy to follow. It states clearly the limitations of current approaches and presents its contributions in a straightforward manner. The chosen hyperparameters and detailed proofs are clearly described in the appendix.\n\n# Significance\n\nThe task of designing high-quality dialogue agents has become ever more visible in the past couple of years. Controlling generations over multiple dialogue turns is critical to the success of such agents, as it could enable more powerful and human-like capabilities with dialogues that carry an extended amount of context and information in previous turns and that are strategically generated in order to achieve a macroscopic goal. The paper is making a significant stride towards that.", "weaknesses": "The evaluations proposed could be more extensive. In particular, it may be interesting to show evaluations on more common single-turn dialogue tasks (e.g. QA tasks) in order to show whether the quality is degraded for these tasks. It is also surprising to see a strong degradation in quality for turns #0 and #1 compared to the raw Llama 8B model, and the paper does not provide an explanation for this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ra25NhGmMO", "reviewer": "Reviewer_GJRW", "summary": "This paper introduces REFUEL (REgressing the RElative FUturE for reinforcement Learning), a novel approach for multi-turn Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs). The key contributions include:\n1. A new algorithm that addresses covariate shift in multi-turn dialogue by using self-generated data and treating the task as a sequence of regression problems\n2. Theoretical guarantees showing REFUEL can match any policy covered by the training set\n3. Strong empirical performance, where an 8B parameter model fine-tuned with REFUEL outperforms a 70B parameter model on long multi-turn dialogues\n4. A novel approach that eliminates the need for an explicit critic network by using a reparameterization trick and relative future rewards", "questions": "1. Does REFUEL's approach work for other multi-turn tasks beyond dialogue, such as multi-step reasoning or sequential decision-making tasks?\n2. Can you provide ablation studies showing the individual impact of key components in REFUEL? Specifically, how much does the relative future reward regression contribute versus the on-policy data generation?\n3. Can you demonstrate REFUEL's effectiveness on significantly longer conversations (e.g., 10-20 turns)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a significant limitation in current RLHF approaches by properly handling multi-turn dialogue scenarios instead of treating them as single-turn tasks.\n2. The theoretical analysis is thorough and proves that REFUEL requires weaker conditions than previous methods like Natural Policy Gradient.\n3. The empirical results are impressive, showing that a smaller model trained with REFUEL can outperform much larger models on long dialogues.\n4. The method is computationally efficient by avoiding the need for an additional critic network.", "weaknesses": "1. The experimental evaluation is limited to dialogues of 5 turns or less, which may not fully demonstrate the method's capability for very long conversations.\n2. The simulator used for evaluation, while sophisticated, may not fully capture real human interaction patterns.\n3. The paper lacks ablation studies on different components of the algorithm and their contributions to the final performance.\n4. The comparison with PPO-based methods is omitted due to computational constraints, which leaves some questions about relative performance against all potential baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)