Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward
Authors: Dipendra Misra
Aldo Pacchiano
Ta-Chung Chi
Ge Gao
Venue: NeurIPS 2025 poster ()
Keywords: LLM post-training; Learning from interaction; user-edits; reinforcement learning; preference learning
Primary area: reinforcement_learning

Abstract: We study how to fine-tune LLMs using user-edit deployment data consisting of a set of context, an agent's response, and user edits. This deployment data is naturally generated by users in applications such as LLMs-based writing assistants and coding agents. The _natural_ origin of user edits makes it a desired source for adapting and personalizing of LLMs. In this setup, there emerges a unification of various feedback types namely preferences, supervised labels, and cost that are typically studied separately in the literature. In this paper, we initiate the theoretical investigation of learning from user edits. We first derive bounds for learning algorithms that learn from each of these feedback types. We prove that these algorithms have different trade-offs depending upon the user, data distribution, and model class. We then propose a simple ensembling procedure to jointly learn from these feedback types. On two domains from Gao et al. 2024, we show our ensembling procedure outperforms these methods that learn from individual feedback. Further, we show that our proposed procedure can robustly adapt to different user-edit distributions at test time.

Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward
Dipendra Misra** Databricks Mosaic Research
Aldo Pacchiano" Boston University
Ta-Chung Chi Databricks Mosaic Research
Ge Gao Google DeepMind
Abstract
We study how to fine-tune LLMs using user-edit deployment data consisting of set of context, agent's response, and user edits. This deployment data is naturally generated by users applications such as LLMs-based writing assistants and coding agents. The natural origin of user edits makes it desired source for adapting and personalizing of LLMs. In this setup, there emerges a unification various feedback types namely preferences, supervised labels, and cost that are typically studied separately in the literature. In this paper. we initiate the thcoretical investigation of learning from user edits. We first derive bounds for learning algorithms that learn from each of these feedback types. We prove that these algorithms have different trade-offs depending upon the user, data distribution, and model class. We then propose a simple ensembling procedure to jointly learn from these feedback types. On two domains from Gao al. [21], show our ensembling procedure outperforms these methods that learn from individual feedback. Further, show that proposed procedure can robustly adapt different user-edit distributions at test time.
Introduction
Post-training of LLMs has become critical step in developing large language models (LLMs) in recent times. particular, high-quality data a valuable artifact for making LLMs useful for domains of interest. Modern approaches typically employ annotators to label a variety of feedback ranging from full human written gold responses to labeling binary preferences [17, 37J. However, these approaches are inherently expensive. In contrast, some recent works have attempted to deployment logs generated by natural interactions of users with LLM Agents to improve these models [15, 21, 25]. In particular, for broad class of LLM applications such as coding and writing assistants, user edits [19, 21] form a natural mode of feedback found in deployment logs. Motivated by this, study how fine-tune L.MSs to given task using user-edits in deployment logs.
Consider the application in Figure where an LLM agent is being used as writing assistant. In any given session, user can query the agent any time solve task such generating email given a short description. to fix the deficiencies. For example, the user may prefer certain writing style that the base LLM agent may not be tailored to generate. We emphasize that these edits are naturally generated by the users making it easy to scale this feedback. Further, since the user has stake in solving their own problem, we can assume that the edit feedback is of reasonable quality. contrast, in RLHF applications, feedback such as pairwise preference is collected by hiring annotators label pair of
the LLM's response is unsatisfactory, the user typically edits the response
$ DM and AP contributed equally and are listed alphabetically, GG contributed before joining DeepMind.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Offline Dataset of User Edits
Online Learning Phase
Robust Mixing
Three Feedback Types
Preferenca Feedback
User queries agint
(nerig aninak food ratatak 1bape douurr
OOO
generates resporse
Bery
bor taning alovad bor aigal can aceuddie
Buparesed Peeibbokk
User
aot aad buaight and rahasia dairy natte
responie
fet freiin ve dedee se seficee tummnn pne heveess
Feedback
User Ed1t Cies
recaives edit cost
Cursuatioo Uner Edit Cont: 1290
Figure 1: IIlustrates our learning from edits setup (Protocol 1). An interesting feature of our setup is that it contains three fundamental types of feedback that can be used fine-tune policies. We show that using multiple feedback types leads to better policies.
LLM responses and typically requires quality check. Finally, edit feedback is also convenient to provide since state-of-the LLMs are often able to generate a reasonable response that may only need small edits to make it suitable.
There has been significant progress in the development of LLMs over the last few years. However, LLMs are not oracle and cannot apriori adapt to a given user or task preference [47, 52]. A common solution is to prompt these models with the needed preferences and requirement often as a system prompt [32]. However, users in real applications are not prompt-engineers and may not list all their preferences the prompt, or may even be aware of their preferences. Further, the needs and preferences can be complex, context-dependent, and maybe hard to describe. This necessitates using of user feedback such as edits to improve the agent's 's response over time.
Our learning protocol shown in Figure I consists of an offline learning phase that uses a dataset of deployment logs containing user interactions with the LLM agent. Each data point consists of an original context, along with the agent's 's response and any user edits. The dataset can be collected from a single user if we are personalizing to a given user's preference, or multiple users if we are learning more general preferences. This dataset is used to adapt the LLM agent. The second phase of our learn- ing consists of an online learning phase during which the agent interacts with a user, or multiple users, to generate a response. We penalize the agent during this phase by the amount of edits, as measured by an edit cost, that the user has to perform to make up for the deficiencies in the agent's response. As in waal-world setting, user(s) during the online learning phase can be misaligned with the user(s) that provided data in the deployment logs. The goal of learning algorithm is to effectively perform offline and online learning minimize the cumulative edit cost during the online learning phase.
In machine learning, there a long line of work on development of optimal algorithms given a set of feedback such as from rewards [51, 48), actions [43, 29], or preferences [17, 37]. As shown in Fig- ure 1, an interesting property of our learning setup is that all three aforementioned feedback types co-occur in this setting. Namely, given the context and user edits, one can simply fine-tune the LLM on the user edits. Alternatively, one can use induce a preference data using the user edits as the preferred response over the agent response. Finally, one can also use the edit distance reinforcement learning. This provides a rich space to develop algorithms. In this work, we initiate a theoretical investigation into learning from user edits. Our purpose is two-fold: one is to provide prin- cipled reasoning for our algorithm choices and compare them to other learning settings, and the second is to predict empirical phenomena that can help in deployment. We first provide bounds for three basic algorithms that use the three separate feedback, and demonstrate that these algorithms have different trade-offs. We then propose ensembling approach that can balance these different trade-offs.
reward to perform
We evaluate these different methods and our ensembling approach on two domains from Gao et al., [21]. These domains evaluate the ability of an LLM agent to personalize to simulated user's latent preference based purely on user edit feedback. We show that across a range of different settings, the ensembling approach perform better than algorithms that use single feedback type.
2

---

2 Preliminary
Notation. We use calligraphic letters such as U to denote sets. We define A2U) as the set of all distributions over a countable set U. For any N E N, we define the set [N] = {1, 21,2,... ,N.. We define the set of all texts as )/ and the set of all contexts X. The context can include text along with other multimodal data such as images or videos. We define the set of LLM policies as II { X A())). Given a context z X, and a policy π E II, we can sample a response y ~ 12). We assume that LLMs can accept multimodal input if the context contains multimodal data.
Learning from user-edits. In our setup (Protocol 1), the agent is repeatedly queried by user to generate a response y given a context v. For example, in the writing assistant application, the assistant agent may be triggered to generate an email draft given short prompt. The context will include this prompt along with the content of the existing document and application metadata such as environment variables and the agent's ss calendar information. The agent will then generate a response y, which is the email draft. If the user finds the response unsuitable, they may edit it to y'. If they decide not edit, then we have y' = y. We will always state there is user edit, and if y y then we will call this an empty edit. We assume the user edits can be modeled with an unknown distribution 9: X y -- A(Y) such that given context x' and response y. the user generates the edit s/ ~ q(- v)
We emphasize that the user performs the edits they need the result for their own needs. This contrasts with the more commonly used preference data in the RLHF literature which is collected by asking annotators to label pair of responses. As this is not a natural task, the annotators often compensated. In contrast, user edits provide freely generated deployment data for performing never-ending fine-tuning of LLMs.
The necessity for user edits is due to deficiencies in the agent's weent's response y. Further, the more edits the user has make, the more deficient is the agent's 's response. We can measure this deficiency with an edit cost c = dddn y, 3/) where Aedit : y² - [0, Cmax] is a suitable edit distance metric. The edit cost is higher if the user performs more edits and 0 only if edits are empty (y = y/). Following Gao et al., [21], we use Levenhstein edit distance over sub-tokens for defining A in our experiments. However, our analysis and algorithm development independent of the choice of Aedi- We are now ready to state our formal setup.
Formal Learning Setup. Protocol I states our formal setup. We assume access to a dataset of n user edits D = {(T., Vi, v') and policy class II. For every i [n], we assume x; ~ p(.), Y: T- 2z), and y/ ~ q(- | 24, yt), where p is the context distribution and Teet is reference policy used to generate the agent response. The reference policy can be a pre-trained LLM that is used to warmstart the problem. This is typical deployment dataset that is encountered in writing and coding assistant applications. This data distribution can be used to perform fine-tune policies in the offline learning phase (line 1).
After this, the agent is evaluated over series of T episodes during the online learning phase. In each interaction, the world (which includes the user) will provide a context x, -p (line 3). The agent can generate a response y/ for this context (line 6). Finally, the user generates an edit y/ which leads to an edit cost of to the agent (line 5-6). The goal of the agent is to minimize the total edits performed during these T episodes (line 9). We allow the agent to perform online learning during these T episodes. However, T may not be large enough to perform full fine-tuning. Consequently, any learning algorithm should try to effectively use the edited dataset D.
Our setup can be applied to both settings where edits are performed by an arbitrary set of people or when they are performed by a single person or a particular group. In the first case, we need to learn general preferences, whereas the latter is personalization setting since we need to learn the preferences a particular person or group.
We define a few useful concepts to enable us to state our formal objective. We first define the expected cost of a response y in context x as c(x,, y) (g [Aedi y')]- Given a policy π II, we then define its expected cost J[x) Ex-ps--((a)) [c(2; v)]. We also define a 3-KL regularized objective as:
T( y
xz)
JT( = -E=p.y-((1E) c(x, v) + 3 Blog
(I)
Trorly)
3

---

Protocol Finetuning LLMs from User Edits. An algorithm needs to implement the lines in brown.
Require: Given dataset D ((ri, = of user edits. Also, given policy class IT. Offline learning phase
Initialize agent using 'D
2: for t 3. I'' do
3: World presents context æt
4: Agent generates response yt
// Online learning and evaluation phase
5: User edits the response to Vi
6:
Agent receives cost for edits = Aam(V v/)
7:
Return E1=1. Cz
is well-known that optimal solution for objective in Equation 1 given by (vlx) [erly((a) 2(x) exp (=<g--) Tyel (2)2) where Z(x) "veyy exp
[40]. We assume T. 11 for the given 33 and following RLHF literature [37, 40], we compete with this KI.-regularized optimal
claw/
B
policy. Formally, we define the sub-optimality (SubOpt) of policy æ and the regret (Reg,) of an
agent during the online learning phase as:
Sub0pt(m) Ja(T)) J=( Regr
SubOpt (TT..
where ne is the policy played by the agent in the 1th episode. We also define D(T, r²) = Ex-p [ z) (- 2²) Ty] as the expected total-variation distance between two policies and n. Our goal is to find an interactive learning algorithm that learns from the offline dataset and minimizes the regret Reg, during the online learning phase with high probability.
Principled Learning from User-Edits
There are two stages of learning in Protocol 1 offline and online. We discuss learning in these two stages below where one has access to moderately large dataset of user edits and much more limited number of online learning episodes.
3.1 Offline Learning from User-Edits
Our learning protocol (Protocol 1) contains multiple feedback sources that are typically studied separately in the ML literature. We discuss learning algorithms for each feedback source below.
A. Learning from User Edit Supervision. User edits v/ provide reasonable sample of the desired behavior. Therefore, one can directly learn a policy ĀSUe by fine-tuning on the user-edits:
ASUP arg max (SUP(T)), where (SUP(T)) n Σ log ক (yy x.).
(3)
xell
B. Learning from Preferences. It is possible that users only partially improve the agent's response when editing it to y' and that the optimal response is still far from y'. In this case, fine-tuning on y can lead to sub-optimal behavior. In contrast, one can use the observation that users edited to y this implies that y is preferred over y and we can use this to perform learning from preferences. We
any geference-learinng approach with dataset [[T Vs. 1/)22_ where v y/. For example, we can perform Direct Preference Optimization (DPO) [40] to learn policy uTREE:
PPREF arg max (PREF(T); well
where
1 a n Σ logo BBlog
TVy xz) Trof (v) x.)
( xz)
(PREF(T)
B log
(ye 254)
We emphasize an important difference between the distribution over preferences in Equation 4 and the typical RLHF literature [17, 6, 37], where the preference pair (y. y') is typically generated by independently sampling from the reference policy. In contrast, in our setting only one response is generated from the reference policy whereas the other generated by user edits. This distributional change impacts both the theoretical analysis and empirical performance.

---

Algorithm 1 LateEnsemble of Policies using Upper Confidence Bound.
Require: Given dataset D {Tr Mi, v of user edits. Also, given policy class IT.
1: Create list of policies V [SUP, MPREE, TRI, #gr] by fine-tuning on
// Offline learning
2: Define total cost (TT) = 0 and count N (T)) = for each policy π E y
3: for t = 1, 2, 3.... I'' do
4: World presents context 24
5: I < | then T. wit] else n7 = arg min
Cix) N(m)
VV
log(() N(T)
6:
yt (=+))z+)
I/ Online learning
7:
User edits the response to v/
8:
Agent receives cost for edits C₁ (ed v1)
9: Update cost and counts: C(T,)) = C(T.) + q;: ((T)) N(T)) +1.
C. Learning from Cost. We observe cost Dedir (yt, 3/4) for every datapoint which is sampled from our cost function c(x, = Eyg-at-jan -a(-(a-v) [edi (v. y')] We can, therefore, use this to train a cost model which can be used to perform reinforcement learning. Formally, given cost model family F we first learn a cost model via square-loss regression:
f = arg min fEF
Σ (J c;)²
(5)
A direct empirical approach will be to use this cost function and perform RL on empirical distribution over the contexts in D. However, f need not be well-calibrated for all responses. Motivated by this consideration, we define a pessimistic cost function, f(r, y) = max JE f(x, v) where the set of functions f { = J s.t. El= (S(x,)) f(x,,)) & (F, 0)} for (F,
O Ilog (()) is a confidence set of cost functions that agree with the historical data. We then train the policy नRR to optimize this cost under KL-constraints our input contexts.
T xx))
TRI. arg min E, (Ix)))) f [arr Bllgg xell
(6)
Teat (y/ xr,)
While computing the / is computationally impractical in the general setting, we neverthless use this pessimistic RL algorithm in Equation 6 for our theoretical analysis.
Early-Ensembling of Losses. As we will demonstrate later, the aforementioned three approaches have different trade-offs. Therefore, it might be advantageous to learn robust policy by optimizing jointly over their losses. We call such an approach an early ensembling. For example, it is quite common in RLHF literature add the supervised learning loss to the preference learning loss [38]:
arg min ((PREF(T) + X
nell
We later empirically investigate the form of early ensembling described in Equation 7.
3.2 Adaptation During the Online Learning Case
During the online learning phase, we encounter a small number of episodes where the main goal is to evaluate the agent. In practice, this will be the deployment phase where the agent interacts with real users. However, this also provides opportunity to perform any additional adaptation.
key challenge of offline approaches is that differrnt methods have different trade-offs. Early- ensembling approaches may not be able to effectively balance between these trade-offs. This is especially true, the user distribution at test time is different from train time, which can happen in practice. Unfortunately, the small number of online episodes makes it challenging to do any effective fine-tuning. Motivated by these two considerations, we employ a very simple approach of training set of policies using the different offline learning methods. During the online learning phase, we then run a bandit algorithm to select which of the learned policy to use for generating the response. We use the user-edit cost input to the bandit algorithm. We call this approach
5

---

late-ensembling of policies. Algorithm 1 gives an example using the Upper Confidence Bound (UCB) approach [5. 31, 11]. however, other bandit approaches can also be used.
Pure Online I Learning Setting. In Appendix A, we consider a pure online learning variant of Pro- tocol with large T and without offline learning phase, and derive no-regret algorithm for it.
4 Theoretical Analysis of Learning from Edits
In the last decade, sigmificant theoretical understanding has been achieved in reinforcement learning where the feedback is reward, and imitation learning where the feedback is action. Can we achieve the same for learning from edits? We initiate the theoretical understanding of learning from edits to provide motivation for our algorithm design and predict experimental phenomena.
4.1 Theoretical Setup and Assumptions.
We start by stating our modeling assumption for the user distribution in Assumption 1. We assume that the user is more likely to edit response to y instead of the reverse depending upon how much more likely is s/ under the optimal policy "' instead of y- This stated formally in Equation which we call the balance equation. This equation can be viewed as serving a similar purpose as the Bradley-Terry distribution RLHF [10, 17] by providing a smooth characterization of user behavior. We also assume that the user distribution has a non-zero probability of generating the optimal response for any input response y. This probability can be small but does not need to scale with the size of the generation space [3%]
Assumption 1 (User Distribution). There exists a (known)
B
0 such that the user distribution
satisfies the balance equation below:
(/V x2
T. (3/ x)
Vx E X, U.V.'' VV,
(8)
40 x. x.,y))
T (y x)
Further, for any E X, the user has at least an Timin (x) > 0 (possibly user dependent) probability of generating the optimal response y² arg maxyey T. (y x): Vx X, y y, g(y" xr. y) > Yimin (ar
In Appendix 1. we provide an example where this assumption holds.
From Balance Equation to Bradley-Terry Distribution. An important consequence of Assump- tion I is that the preference distribution in Equation 4 satisfies the Bradley-Terry assumption. To see this, first realize that our preference pairs will contain two responses (y. v') given context a in precisely two situations: either we generate y ~ Tre((- z) and is edited to y/ ~ q(: x. y), or we generate of ~ T-e((- z) and it is edited to y ~ g(- xr, y). The probability that prefer y/ over (y' 2 y) is then given by the probability of the first situation normalized by the joint probability, i.e.,
Trer(y x) 2)/(v/
2. v)
P(y z y x. v.V.8')=
Tre x))(w/ x. v) + Tret x)a(w 12,5''
Tref (y I)) (y/ xx)
(from Equation 8)
Teer x) (3' x) Teet Tv x))=( (y/ z)
(c(xr,)) y) - c(,,,')))
(using ( x)
Tea (yla) 2(x)
exp( e(z,z)).
This justifies using DPOs even though the joint distribution over the preference pairs (v. y²) is different than IID sampling from Neef which is typically studied in the RLHF literature.
As working with function approximations, we assume policy realizability, i.e., that our function class is expressive enough to contain the desired policies. This is a standard assumption in theoretical analysis of interactive learning algorithms [28, 54].
Assumption 2 (Realizability). We assume that T. II and g o л E where g o π is the policy defined as q [t)(z) = Eyey A8y|,, vf v/)T(y/(z))
Our first result establishes Assumption implies the edits induce a contraction property on the user distribution.

---

Lemma 1. For any E II, the user distribution satisfies the following contraction property:
E X,
l1 (Y x) (Y 1>)l (1 Yimin (2:)
<T
x))
- (Y x))
The proof of Lemma I can be found in A. The contraction property implied by Lemma I indicates that iterative application of the user distribution approaches fixed point.
Finally, we make one final assumption for stating results for DPO. We assume that the data dis- tribution satisfies the following concentrability assumption. Similar to realizability assumptions, concentrability assumptions are typical in RL literature (see for example [28, 54)).
Assumption 3. [Preference Concentrability] For any л E Il we assume:
Gog (V][m 8 log x()]z) x, (ū)lz)
m. (3' m))
CPREF,
E (m.j..0)) (..
x171z) Blgg =(gp=)) Blog x² (171))
-.. (5)la))
where Qr = p(x (ū/ (ū) a:) x "0 . x)).
This assumption states that our data distribution has enough coverage to allow us to evaluate different policies accurately. In practice, can expect CPREF to be small when II is a small constrained class around ref. Alternatively, we can consider pessimistic-versions of preference learning methods [27] which avoids paying for covering all policies.
4.2 Theoretical Results.
We state the results for our different algorithms below and defer their full proof and analysis Appendix A. Our first result is suboptimality bound for the SFT algorithm. Theorem 1 (SFT Result). Let > 0 be sub-optimality target. Under Assumption 1 and Assump-
tion 2 to achieve SubOpt(Asup) & E O min 7max ([Trrtt T&), Mmax DD/22(Tret TT. T with
probability at least 1 - 5 it is enough to set > Ω [1og(111//8) where Mmax 1 minl Vmino (x) and
7max
Ex-p [1 Ymmin
[z] and ((Tref: )T)) En-p Frer(Y [x) m2 (Y xx) m
We state the suboptimality bound for the offline DPO algorithm that treats the edit data set D ((x,, Vi. 10 as a preference data set in Theorem 2.
Theorem 2 (DPO Result). Let >> 0 be a sub-optimality target. Under Assumption 1, Assumption and Assumption 3 to achieve SubOpt (TREE) & e with probability at least 1 8 it is sufficient to set
n 3 Ω
Crane-log([I1:5) where Vmax > 0 satisfies max (xX(xx))xX | log =(g'la) & Vaur
[((o((-Vaax))) +e²
The proof of Theorem inspired by the techniques of works such as [44, 54]. Our final result characterizes the sample complexity of learning from costs.
Theorem 3 (RL Result). Let e O be a sub-optimality target. Under Assumption the policy TRR satisfies SubOpt(#xL) & e with probability at least 1 8 when n >
Ω [og(([]/88) C..dmma(FF)an
(Fgg((())))) where C =Z=Erne
( x (M)a)) (Feat(
2 is the policy concentra-
bility coefficient of Feet and demua (F) is the Eluder dimension -a measure of statistical complexity- of the cost function class (see [45]).
Discussion of Trade-Offs. Theorems 1, 2, and 3 provide some guidance the cases where using the edit data as a preference dataset is better or worse than doing imitation learning on the edits or learning the cost function and doing RL. When the preference coverage coefficient CrnFF is small, the sample complexity of DPO can be much lower than that of SFT and RL. When the weighted approximation
error min Tmmax Trof TT') Tmaax D /2 ( T is smaller than the target error e, SFT is the pre- ferred method. Finally, when the approximation error and preference concentrability are large, but policy concentrability is small and the cost function simple-for for instance, a oow-dimensional linear function-lcaruing learning the cost function and then optimizing is the most sample-effice approach.
7

---

5 Experimental Results and Discussion
We empirically evaluate our theoretical findings in this section to see they apply in complex settings similar to what is encountered in practice.
Task Setup. We evaluate on two tasks: email writing and summarization, from Gao et al., [21]. For each task, there exists a dataset of articles from 4 domains. Corresponding to each task and domain, there is a list of latent user preferences described as a preference string, for example, "structured, straight to the points, respectful, professional greeting and closing". The setup follows Protocol 1. In each round, the agent is given a context describing a task along with a given article. For example. summarize given article. The ageni does not have access to the latent preference string but must generate a response that satisfies this preference. An LLM user generates an edit given the response, context, and latent preference string corresponding to the domain of the article in the context. The use an LLM-based user allows for reproducible experiments which facilitates rapid advancement in algo- rithm development." Finally, we compute edit distance using Levenshtein distance normalized by the number of tokens in the agent response. We use the NLTK word tokenizer to compute the edit distance. We use Qwen 3 32B instruct model as our user but remove think tokens before using the response.
The key challenge of this task is that latent preferences are context-dependent as articles from different domains have different preferences. Further, even for single domain, there are multiple preferences in preference string. For example, the aforementioned preference string contains multiple individual preferences such as "second person narrative" and "show emotions". These two challenges occur in real-world applications. For example, a person may prefer to write informal emails their friends but write formal reviews for their office reports. Note that the context does not state which domain the article in the given context comes from. An LLM agent must implicitly infer the domain from the context, learn the appropriate preference for it, and then use it to generate an appropriate response.
Strong and Weak User. We extend the original setup of Gao et al., [21] to consider two types of users: strong users and weak users. A strong user generates edits based on every individual preference in the article's latent preference string. In contrast, a weak user samples subset of preferences in the article's 's preference string and uses them to generate the edits. For example, in any given interaction a weak user may sample two preferences "second person narrative" and "brief" and perform edits to satisfy these while ignoring the other preferences in the preference string. The weak user models user who may only prefer to perform small edits a time. Conceptually, both the weak and strong users have the same optimal behavior: This is because the optimal response that satisfies all the preference for the strong user, also satisfies any subset of preference that the weak user can sample. However, while strong user perform edits that rapidly converges to the optimal policy (higher Tmin), the weak user's 's edit will slowly take the response towards the optimal behavior (smaller Ymin).
We collect an offline interaction dataset between the agent model and the user. This dataset is supposed to represent deployment logs found in typical LLM agent applications for writing and coding assistants. We use Llama 3.1 8b Instruct model as our agent model. We perform generate responses by doing greedy decoding. We collect dataset of 20,000 examples for the summarization task and 10,000 examples for the email writing task. We collect these datasets separately with both strong and weak users. This gives us 4 separate offline datasets.
Offline Dataset.
We evaluate the model for T = 200 examples for summarization and email writing. We always use the strong user during test time regardless of which user was used to generate the offline dataset. We run each experiment with 3 different seeds.
Online Learning Phase.
Methods and Implementation. We consider the following approaches: (i) Base which generates from the base agent model, (ii) SFT which performs SET on the training data, (iii) DPO which runs the DPO algorithm on the training data, (iv) EarlyEnsemble which performs early ensembling of DPO and SFT losses (Equation 7), and (v) LateEnsemble which performs late ensembling (Algorithm 1) using policies trained by methods (iv)- All generations are performed greedily and with a max num- ber of generation tokens of 1000. We do not evaluate Equation 6 give challenges in implementing it.
We sweep over various hyperparameters including learning rate and epochs for SFT and DPO, as well as 33 for DPO, and B, À for EarlyEnsemble, We pick the best hyperparameters by maximizing log-loss held-out validation set of 200 user-edits. We found in early studies that despite sweeping over the hyperparameters, both DPO and EarlyEnsemble learn policies that tend to repeat text
Gao et al.. [21] took steps validate their LLM user. Please see their paper for details.
8

---

Method
Summarization
Email Writing
Strong User
Weak User
Strong User
Weak User
Max SubOpt
Base
0.9455=0.01
0.9445 0.9304-2.11
0.5108
0.4923+0.001
0.73644u..1
SFT DPO
0.5377 20002
0.4159
0.4539+0.03
0.5772+0.19
1.0790
0.8267
0.3365 +0.30
0.3368+0.0z
0.8698+0.11
EarlyEnsemble
0.2092-0.00
0.3586=0.01
0.3438 20.06
0.486440.01
0.1612+0.01
LateEnsemble
0.2768 1.11
0.4403 1..03
0.4202
0.3739 1 10.04
0.1586 0.004
Table 1: Results on the summarization and email writing domain. We calculate the mean total edit
distance 1
T - et during the online learning phase. We report average performance across 3 seeds.
Method
Summarization
Email Writing
Strong User
Weak User
Strong User
Weak User
Max. SubOpt
Base
1.8498.0.01
0.855810.01
0.4241 10.01
0.4239 0.386610..1
0.665410.01
SFT DPD
0.4890 0.001
0. 0.769822.00
0.3377 10.02
0.5121
0.9650 20.03
0.7800
0.2955
0.304741000
0.780521.02
Ear EarlyEnsemble
0.1845 00.02
0.2577 +0.02
0.240640.00
0.4001+0.03
0.0955+0.03 0.0862+0.02
LateEnsemble
0.2509+0.04
0.3403
0.3123
0.3428
Table 2: Transfer learning results with a Llama-3.3-708-Insttue User at test time.
until their max tokens expire. We, therefore, perform a post-generation trimming strategy where we trim the generations once it starts repeating 200 consecutive characters. For a fair comparison, we apply this post-processing to the output of all methods. An alternative approach could be to use explicit length penalty [39] or more stable preference learning approaches such as Rebel [22]. See Appendix C for full details on experimental setting.
Main Results. We report the main results in Table 1 with the Qwen-3 32B user. We report mean edit cost across rounds (7) and seeds. We also report Max. SubOpt that measures worst-case performance gap across the four settings between a given approach and the best performance. We see that Base method does not perform well showing the need for adaptation. Intuitively, we would expect a strong user have strong convergence to the optimal policy. In our theory, this would be reflected in the value of Ymin in Assumption 1. We see this in our results where SFT is able to learn and performs much better when trained and tested on strong user. Its performance when trained on weak user is weaker consis- tent with Theorem 1. In contrast, DPO's theoretical analysis does not depend on the choice of Ymin and instead only depends on coverage and the validity balance equation. We see that DPO performs bet- ter than SFT when we train weak user and test on strong user. However, when both users are strong, then there is no clear winner between SFT and DPO. Overall, this illustrates the trade-off predicted by our theory where SFT can take advantage whenever the user-edits converges faster to the optimal pol- icy, however, it is also susceptible when this is not the case whereas DPD is more robust to this factor.
We see that Earl EarlyEnsemble can exploit these trade-offs to achieve better performance than both SFT and DPO on the summarization domain, however, it under- sedder-prforms DPO on the email writing domain. One explanation can be sensitivity to the choice of X. While we perform a grid search over this hyperparameter, and use a validation loss over user-edits, it is possible that this selection scheme does not correspond well with test-time performance. Finally, our LateEnsemble approach able to more readily adapt at test time by making suitable trade-offs and achieve the lowest worst case SubOpt value. We emphasize that late ensembling isn't guaranteed to do better in practice. Specifically, if one of the policy in on line in Algorithm 1, is always better than others, then UCB will achieve lower performance than this policy due to exploring other sub-optimal policies.
Transfer Learning to Different User LLM. We investigate how robust these methods are different user L.M. This mimics common real-world setting where the training data may come from one set of users, however, testing maybe done with different set of users. These two sets of users can still have a common preference because these maybe dependent on organization, platform, or other controlled features such as age, or location. We use 1ama--.3-70B--Inttre user model during the online testing phase while using our previously trained models. Table 2 shows the results. We see that the main trends from Table carry over even when we test with a different user LLM. In particular, ensemble approach LateEnsemble achieves the best SubOpt value.
model as our

---

Table 3: Generations from EarlyEnsemble the summarization task with strong
Type
(Latent User Preference String) Agent Response
Success
(question answering style!) **Q: What was the state of the land and people after the calamity?** A: The land exhausted, the towns stripped of their lighting population, and only shadow of kingdom remained. [truncated]
(bullet points, parallel structure, brief) **Updated Mn PBPK model**; Reflects rapid Mn associa- tion/dissociation in tissues, with saturation of binding sites limiting accumulation.
K*y changes**:
Rapid Mn binding and dissociation rates.
High-affiinity binding with rapid exchange between bound and free forms.
Tissuc- specifie binding capacities to account for background Mn levels.
Model performance** Successfully describes Mn kinetics in adult rats exposed to ... [truncated]
Failure
(inquisitive, skillful foreshadowing, with emojis) **Q: What is the traditional approach to drug development, and why is it no longer effective?***
A: The traditional approach focuses single target protein, disease, and drug. However, this paradigm now considered untenable due to promiscuity of drug- like compounds, which can interact with multiple target proteins.
"*Q: How do current target-fishing approaches address the promiscuity issue?**
A: Current approaches, such single-label multi-clasification schemes, [truncated] (question answering style) The article compares ministers, Mr. Mearns and Rev. Mr. Tindall, highlighting their differences style and delivery. Mr. Mcarns described as steady, patient, well-spoken, ... [truncated]
Cummuletive User Edits
LatsEroemble
Cummulativv User Edits
LiteEnsemble
150
100
0
150
Rounds
(a) Summarization Domain with Strong User
(b) Summarization Domain with Weak User
Figure 2: Cumulative User Edits at test time corresponding to Table I for the different setup. For each round, we show mean and standard deviation across seeds.
Sampling Generations and Test-time plot. We provide a few sample generations showing some success and failure cases in Table 3. We also visualize the test-time performance of different methods on the summarization task in Figure 2. As we can see, the LateEnsemble approach converges to the best method in both cases expected. We provide additional details in Appendix C.
6 Conclusion and Limitations.
User edits provide a natural feedback for adapting LLMs. In this work, we initiated the theoretical investigation into learning from user edits. We propose a learning setup (Protocol 1) and analyze different algorithms in this setting. We show that none of these algorithms is always desired and propose simple ensembling approach that empirically works best in two domains. Future works can investigate limitations of our work such as relaxing different assumptions, deriving optimal algorithms, and evaluating these algorithms in human study.
10

---

References
[1] A. Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference Machine Learning. pages
PMLR, 2014.
[2] A. Agarwal, S. Kakade, A. Krishnamurthy, and W. Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances neural information processing systems, 33:20095-2000 2020.
[3] S. Agrawal and M. Carpuat. An imitation learning curriculum for text editing with non- autoregressive models. ArXiv, abs/2203.09486, 2022.
[4] S. Proca-Ouellette, N. Mackraz, B.-J. Theobald, and K. Metcalf. Predict: Preference reasoning by evaluating decomposed preferences inferred from candidate trajectories. ArXiv, 2024.
[5] Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-256, 2002.
[6] Y. Bai, Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05880
2022.
[7] V. Balachandran, H. Hajishirzi, W. Cohen, and Y. Tsvetkov. Correcting diverse factual errors in abstractive summarization via post-editing and language model infilling. ArXiv, abs/2210.12378. 2022.
[8] A. Beygelzimer, J. Langford, L. Li, L.. Reyzin, and R. Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Confer- ence on Artificial Intelligence and Statistics, pages 19-26. JMLR Workshop and Conference Proceedings, 2011.
[9] Bouzenia, Devanbu, and M. Pradel. Repairagent: An autonomous, Ilm-based agent for program repair. ArXiv, 2024.
[10] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-355 1952.
[11] S. Bubeck, N. Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends0 in Machine Learning, 5(1):1-122, 2012.
[12] M. Cao, Y. Dong, J. Wu, and C. K. Cheung. Factual error correction for abstractive summa- rization models. ArXiv, abs/2010.08712, 2020.
[13] S. Cen, J. Mei, K. Goshvadi, H. Dai, Yang, Yang, D. Schuurmans, Y. Chi, and B. Dai. ialueeincentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405.. 19320, 2024.
[14] Chen and N. Jiang. Information-theoredt considerations in batch reinforcement learning. In International conference machine learning, pages 1042-1051 PMLR, 2019.
[15] Z. Chen, M. O. Gul, Y. Chen, G. Geng, A. Wu, and Artzi Retrospective learning from interactions. arXiv preprint arXiv:2410.1385: 2024.
[16] Z. Chen, S. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshyvanyk, and M. Martin. Sequencer: Sequence-to-seqqeenee learning for end-to-end program repair. IEEE Transactions on Software Engineering, 47:1943-1959, 2018.
[17] P. F. Christiano, J. Leike, T. Brown, M. Martie, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[18] M. D' Arcy, A. Ross, Bransom, B. Kuehl, Bragg, Hope, and D. Downey. Aries: A corpus of scientific paper edits made in response peer reviews. ArXiv, abs/2306. 12587, 2023.
11

---

[19] F. Faltings, M. Galley, K. Brantley, B. Peng, W. Cai, Y. Zhang, J. Gao, and B. Dolan. Interactive text generation. In The 2023 Conference on Empirieal Methods Natural Language Processing. 2023.
[20] D. A. Freedman. On tail probabilities for martingales. the Annals of Probability, pages 100-115 1975.
[21] G. Gao, A. Taymanov, E. Salinas, P. Mineiro, and D. Misra. Aligning Ilm agents by learning latent preference from user edits. NeurIPS, 2024.
[22] Z. Gao, J. Chang, W. Zhan, O. Oertell, G. Swamy, K. Brantley, T. Joachims, D. Bagnell, J. D. Lee, and W. Sun. Rebel: Reinforcement learning via regressing relative rewards. Advances in Neural Information Processing Systems, 37:52354-52444 2024.
[23] S. A. Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
[24] K. Guu, T. B. Hashimoto, Y. Oren, and P. Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437 450, 2017.
[25] J. Hong. J. Lin, A. Dragan, and S. Levine. Interactive dialogue agents via reinforcement learning on hindsight regenerations. arXiv preprint arXiv:2411.05194, 2024.
[26] A. Huang, A. Block, D. J. Foster, D. Rohatgi, C. Zhang, M. Simchowitz, J. T. Ash, and A. Krishnamurthy. Self-improvement in language models: The sharpening mechanism. arXiv preprint arXiv:2412.011951 2024.
[27] A. Huang, W. Zhan, T. Xie, J. D. Lee, W. Sun, A. Krishnamurthy, and D. J. Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared pref- erence optimization. In The Thirteenth International Conference Learning Representations.
[28] A. Huang, W. Zhan, T. Xie, D. Lee, W. Sun, Krishnamurthy, and D. J. Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization. arXiv preprint arXiv:2407.13399 2024.
[29] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[30] C. Jiang, W. Xu, and S. Stevens. arxivedits: Understanding the human revision process in scientific writing, ArXiv, 2022.
[31] T. L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4-22, 1985.
[32] Q. Liu, N. Chen, T. Sakai, and X.-M. Wu. A first look at llm-powered generative news recommendation. ArXiv, 2023.
[33] Y. Liu, B. Deb, M. Teruel, A. L. Halfaker, D. Radev, and A. H. Awadallah. On improving summarization factual consistency from natural language feedback. Annual Meeting of the Association for Computational Linguistics, 2022.
[34] J. Mallinson, A. Severyn, E. Malmi, and G. Gariio.. Felix: Flexible text editing through tagging and insertion. ArXiv, abs/2003. 2020.
[35] E. Marrese-Taylor, M. Reid, and Y. Matsuo. Variational inference for learning representations of natural language edits. ArXiv, abs/2004.09143, 2020.
[36] M. Mita, K. Sakaguchi, M. Hagiwara, T. Mizumoto, J. Suzuki, and K. Inui. Towards automated document revision: Grammatical error correction, fluency edits, and beyond. In Workshop Innovative Use of NLP for Building Educational Applications, 2022.
[37] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhhnng S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
12

---

[38] R. Y. Pang, W. Yuan, H. He, K. Cho. S. Sukhbaatar, and J. Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37:116617-11667, 2024.
[39] R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. Findings of the Association for Computational Linguistics ACI. 2024, pages 2024.
[40] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[41] M. Reid, J. Hellendoorn, and G. Neubig. Diffuser: Diffusion via edit-based reconstruction. International Conference on Learning Representations, 2023.
[42] M. Reid and V. Zhong. Lewis: Levenshtein editing for unsupervised text style transfer. In Findings, 2021.
[43] S. Ross, G. Gordon, and D. Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artift- cial intelligence and statistics, pages 627-635, JMLR Workshop and Conference Proceedings, 2011.
[44] C. Rosset, C.-A. Cheng, A. Mitra, M. Santaeroce, A. Awadallah, and T. Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. CoRR, 2024.
[45] D. Russo and B. Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems, 26, 2013.
[46] A. Saha, A. Pacchiano, and Lee. Dueling rl: Reinforcement learning with trajectory prefer- ences. In International conference on artificial intelligence and statistics, pages PMLR, 2023.
[47] A. Salemi, S. Mysore, M. Bendersky, and H. Zamani. Lamp: When large language models meet personalization. ArXiv, 2023.
[48] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
[49] D. Sobania, M. Briesch, C. Hanna, and J. Petke. An analysis of the automatic bug fixing performance of chatgpt. 2023 IEEE/ACM International Workshop on Automated Program Repair (APR), 2023.
[50] F. Stahlberg and S. Kumar. Seq2edits: Sequence transduction using span-level edit operations. Conference on Empirical Methods in Natural Language Processing, 2020.
[51] R. S. Sutton, A. G. Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[52] Z. Tan and M. Jiang. User modeling in the era of large language models: Current research and future directions. ArXiv, 2023.
[53] Y. Wang, Q. Liu, and C. Jin. Is rlhf more difficult than standard r!? a theoretical perspective. Advances in Neural Information Processing Systems, 36:76006-76032 2023.
[54] Xie, D. J. Foster, A. Krishnamurthy, C. Rosset, A. Awadallah, and Rakhlin. Exploratory preference optimization: Harnessing implicit qppaopooximttion for sample-efficient rlhf. arXiv preprint arXiv:2405.2
2024.
[55] Y. Xu, R. Wang. L. Yang, A. Singh, and A. Dubrawski. Preference-based reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 33:18784-18794, 2020.
13

---

[56] T. Yang, J. Mei, H. Dai. Z. Wen, S. Cen, D. Schuurmans, Y. Chi, and B. Dai. Faster wind: Accelerating iterative best-of-n distillation for Ilm alignment. arXiv preprint arXiv:2410.20727 2024.
[57] P. Yin, G. Neubig, M. Allamanis, M. Brockschmidt, and A. L. Gaunt. Learning to represent edits. ArXiv, abs/1810.13337, 2018.
[58] W. Zhan, M. Uehara, N. Kallus, J. D. Lee, and W. Sun. Provable offline preference- basedd reinforcement learning. arXiv preprint wxxv:2305..18866 2023.
[59] Zhang, S. Panthaplackel. P. Nie, J. J. Li, and M. Gligorié. Coditt5: Pretraining for source code and natural language editing. Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022.
[60] Q. Zhang, C. Fang, Ma, W. Sun, and Chen. survey learning- based automated program repair. ACM Transactions on Software Engineering and Methodology, 2023.
[61] B. Zhu, M. Jordan, and J. Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pages 43037-43067. PMLR, 2023.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] " is generally preferable to "[No]" it is perfectly acceptable to answer " "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often
nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide brief list of limitations the end of the main paper in Section 5. We also provide a more detailed discussion a limitation section Appendix A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
15

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specing well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions are listed the main body, and proof along with explanations are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We used an existing public benchmark [21]. We used open-weight LLMs from HuggingFace and we provide details of our experimental setup Section 5 as well as a deeper discussion in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
16

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., . a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We not releasing the code. We adapted existing benchmark of [21] for learning from user edits that available at https : //github. com/ gao- g/ g/prelude, and used four publicly available datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Coo//oooeeseeeeeeeeeeeeee
/ //ipps.c.. wwwpc//ppc//pucecceeeecceeeooooa haabll/c/guiddes/ddeuuiiiiiii cwtuuubmmssiooi wwsssssnnols. insiinncoccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
17

---

6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide information of data splits, hyperparameters, and how they were chosen in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes. We run each experment with 3 seeds and provide standard deviation in tables (e.g., see Table and Table 2).
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details of compute and time of execution in Appendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : //neurrurs..../ cttppppp..pppppppeeppe wwcppbbicc/eaeeeeeeeeeeeeeeeeeee cttideeeeeeeeeeeee
Answer: [Yes]
Our work does not create a new dataset, or releases a model to the public, nor does it include any human study. The main purpose of our paper is theoretical investigation into how to develop principled algorithms for learning from user edits. We provide experiments using publicly available benchmark and datasets.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We list the broader impacts of our paper here suggested in NeurIPS Ethics guidelines. Our work focuses on the foundational aspect of learning from user edits. Even though our work doesn't 't involve human studies or releases any models, the ideas from this paper may influence actual uses cases. Since our learning setup has a humaanin-the-loop performing the user edits, we need be wary when releasing product that learns from user edits. To begin with, we need to ensure that we have the user's ' permission to learn from their edits or even store them. Various privacy and data laws may apply here that we need be mindful of. Another issues is that if we are learning across multiple-users. we need to ensure that we don't leak sensitive information from one user to another. E.g., user may edit an legal documentation to add some identifiable information and an LLM may inadvertently learn edit does to add this information and show this to other users. Even showing this to the same user at a wrong time can be undesirable, e.g., when they are presenting in front of others. Filtering data to remove PII and having guardrails can mitigate this to an extent. Another option is to have users report and be able to delete user-edits whenever they want.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
19

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release any new data or models with this paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite every asset that and provide their details in Appendix C.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
20

---

We are marking it as non-applicable since we do not create any new data or model in this paper and do not release the code with the paper.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not do any human study or involve any human participant in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human study was performed in this paper and no human subjects were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
21

---

Our paper describes ways to fine-tune LLM using user-edits. We used previously released benchmark that includes an LLM user. We trained our LLMs using this simulated LLM user. We did include samples of generations in Appendix C that are generated by LLM. However, we did not use an L.L to generate the main ideas or proofs in this paper.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: / http//eerrpp..ppe htt.://scccccoooeeeneeeee/////// LLLM) for what should or should not be described.
22

---

Appendix
Table of Contents
A Theoretical Analysis with Edit Distance
23
A.1 Properties of the Balance Equation
23
A.2 General Results
25
A.3 Leaning from Supervised Feedback
28
A.4 Theoretical Analysis with Preference Feedback
29
A.5 Learning from Cost
33
A.6 Pure Online Learning Theory
38
A.7 Support Results
41
B Related Work
42
C Additional Experimental Details
42
A
Theoretical Analysis with Edit Distance
We provide table of notations for convenience in Table 4.
Notation
Description
A(U)
Set of all distributions over countable set U. Denotes the set {1, -.. .N} for any N e N. List of contexts which can include text, images, etc.
[N]
X
Y
List of text-based agent response.
PE A(X)
Distribution over context at both train time and test time User distribution, given a context x E and agent response y E y, the user generate an edited response Computes edit distance Acau(y, s/) for editing to y/-
9: x y - A())
y with probability qly x; x,,)).
Aoda: : :2² - [0, Cmax]
We assume A (y, y)
e: X x 3 [0, Cmax
Expected edit cost c(a;, y) for response in context s. general notation for a policy that generates response y given context
w: ::X A())
2 with probability л (3 xr).
A list of policies.
KL-divergence coefficient. It is always non-negative coefficient.
KL-regularized optimal policy
TSUP
Policy trained with supervised fine-tuning (SFT).
TPREF
Policy trained with Direct Preference Optimization (DPO).
FRI.
Policy trained with reinforcement learning (πRI).
fgr
Policy trained with early ensembling (Earl [EarlyEnsemble,
Table 4: Table of important notations and their description.
A.1 Properties of the Balance Equation
Our setup consists of distribution pe e A(X) over context x, user edit distribution q y A(Y), and an edit distance Aedit y x y - [0, cmax]. We make two assumptions on our setup:
q
T. (1/ x)
WI E X.y.V e
q
xr
(y x)
23

---

and that for every E X, there exists y" such that for every y V we have q(y" .r,)) > Ymin min, ( For simplicity we consider constant "/min function this section. We now consider examples of this setup.
Example 1. We start with a singleton context = {xx] and y {y11 U2: UN We define the user edit distribution below:
quN Mis x) Viin
Ymin
q(ys x, 21)
Tinin Vj *j*N
N
We define edit distance as follows Aem (v. 8 * y/} for any and y'. Using this, we get the expected cost as c(x,,) y) 8 8 Nuii for every Un and c(r, UN) - á (7min --س- ). Let č = c(x, yN) then for every y * UN we have c(I, 3) E 07min Hence, the preferred response is yN. We also satisfy the constrained that we must have at least Vmin going the optimal response. That leaves only the balance equation. The optimal policy is given by:
c(sr, (-((x,)) y)
m (ty x))
Zz(x)) exp
For any y. * VN we have g/3// x. y) = gly x,88) = L-nii Further, we also have T x) = T (v/ x) (x) exp
N
0+-77mm Therefore, the balance equation is satisfied in this case. When both are yN then the balance equation is also satisfied. This leaves single case where one of them and other is not. Let the other be
We have:
qUN
x. x.3)
Mmnn
Yain
+ Tinin
Ymin
<(v
J. UNNNN
1 1-Tii
Ymin
and
x (yN x))
c(2, v) c(x, HNN
8 Vmii
exp
exp
n" (y x))
B
Combining them get:
+
Vinin
Ymin
8 3
exp
(10)
Jmin
straightforward to see that for any choice of N and Ymin we can define 3 satisfy this last
equation.
A.1.1
Contraction Property
Lemma (T² as steady-state of user distribution). For any x E X we have q T ()) = T (Y
x).
Proof. Fix x E X, then for any y, y y we have from the balance equation in Assumption the following:
g/y 2, v)
T x)) = g(y x, x,,)) x) = = xx)(( x,21)
(11)
gly x, x.
n (v xx)
Summing over y on both sides gives:
Σ
al x, x))
Σ q(11 x,
yEy
yey
= (v/ 2).
This is equivalent to a xt n."
Lemma 1. For any the user distribution satisfies the following contraction property:
Vx E X, IIgo TY x) - . x (1 (x)) (T(( Y æ) -=² x))
24

---

Proof. Fix x E X. We start by using Lemma 2 to express:
lq x) - (Y x) TV = 1 1 (Y x) go ( TV)
1
Σ
Σ
a' 2. 2.,))((y1
x) Σ q(0' 12,0) xx)
2 y'Ey wey
vEy
We define a quantity w(y', v))) as follows:
fa(v"
x, y) - Miia
yf = y otherwise
w(y's v.
(9 x,
Assumption implies w/y/.,;a) > and
Σ y'Ey
w(y x) = w(y" xr)
(12)
Σ
w(y' = a(v"' xr, v) - (min(()) +
Σ
au 1),
Ymin (z).
Plugging it, we get:
1 (Y x)) - ( T))
(wly" v) + 7imin(x:)) T [x) - Σ
(w('", (w;"", + Ymin (2)) x. ~ x))
2 DEy
wEy
+ 2
1
Σ
Ew(sson (x) Σ
u x; xx)
x +hy
yey
Σ
Σ
w(y' y: x))()(y ()
Σ DEy
wwl. x)) x))
=
2
D'EY' DEy
We view w(y', yi x) as a | |}] matrix W with entries Wy' , > satisfying Ey Wy' 1 (x). We omit x from the W notation it is fixed. Similarly, define [DI-dimensional vectors and n with entries T④ I) and Th (y x). Using these matrix and vector notation we can express:
(Y 1 x) 7' 1v = Σ
Σ
wly' (o..) x)) - wit () x)
2
y'E vEy
yEy
Σ Σ
w v; |T(2 œ) - .. x)
2
yE y'EY
Σ
(1 Mmin (x)) | | x)) m. (y) x)]
2
yEy
- Mmin (x)) - (V|x)
क (Y
where the second step uses the non-negatitivy of w(1 (o0yyy:: x) and triangle inequality and the third step uses Equation 12.
A.2 General Results
Recall that we define optimal policy as a KL-regularized policy:
- arg min Jæ(m), where
(13)
JT(T) = E Ex-py~a [e(x, v) + BDL. (T||
(14)
25

---

B > 0 is a chosen hyperparameter defining the objective. We can establish easily that nT" is given by:
2 I))e (o(a,))
This is standard result but we prove it in the next Lemma for completion.
Lemma 3 (Optimal Policy). The optimal policy satisfies 7. a) = (x) x))-c(z.9)19 for every x E y y E V, where W* = Eyey Treely' =o)e-(l 1/3 is the normalization constant.
Proof. The Lagrangian for the problem is given by:
L = ( + Σ Me (E=w1=)-1)
where fftz E } are our Lagrange multipliers. Taking derivative wrt to ay x) for a fixed value of x X and y )' and setting it to O we get:
dc
pπ x) p(z)c(x))) v) Bp(x) log T(y x)) + 3p(x) - 3p(x) log Trerly x) + p =
Rearranging the terms we get:
π (y x)
Mx
cxx, 3)
log
+ 1
rea (y x))
3p(x)
5
let (x) = cxp (-1- -
ao Sp(z)
) we get:
क (v x) W (x))rer at)e cix.w)
the value of (x) serves as a normalization factor to ensure ΣnEy T x) = and enforced by taking derivative wrt Pz and setting them to O. This gives W*(x)= Eyey Tre x)) (0he-o((,))) o(a. //8
Recall that we define Sub0pt(*) = J=( JT()). In many practical setting, we want to compete with the regularized policy but we only care about the unregularized cost. We capture this by defining the unregularized sub-optimality:
SubOpt = Jo(z) - Jo(
(15)
We also define unregularized regret as
Reg
SubOpty(m)),
(16)
where ne is the policy played by the agent in round t. If we play a fixed policy æ for all rounds then we have Reg+ = -i= SubOpt, (T) = ('SubOpt,,TT).. In these cases, we will use (Tr) to denote the regret of this fixed policy.
useful lemma below relates the total variation between policies to the suboptimality.
Lemma 4 (TV Unregularized Suboptimality). If Ez-p T(Y ()) -
= -
20) |
& e, then
SubOpt,(z) & 26Cmoo and Reg? (T) & 2e T.
Proof.
Σ mly x)((2.)) Σ o m))(a,))
SubOpt = Jo(x) - -o((T') = Exup
v=y
DEY
|Ty x)) - <5(3 x)) c(x, vn
< En-p
Σ ye
= 20max Eemp
T| 2)) - =f x)) $ 2ecmax
Finally, (x) T'Su0Opt, & 2eCmaxx
26

---

Bounding Regularized Suboptimality and Regret. One challenge in bounding regularized subop- timality SubOpt(T)) Jp(m) Jz(x.) and regularized regret Regr
>> SubOpt(TT (1;0 is dealing
with KL divergence. E.g., policy satisty ITY x)) (Y
for some z but still oc if T(y x) = c, + x)) Tref (1 (x) 0.
have (T(, (rel x))
However, in RLHF analysis it is often assumed that densities bounded [44, 54] and this allows us to avoid these cases. In fact, we will later assume this in our proof of DPO. Specifically, we have Assumption (Bounded Density Ratios). We assume there exists Vaaax > 0, such that:
TT x))
V
Vx E 11, EE N.y.
log
Tree (y
3
Under this assumption we can bound the regularized suboptimality and regret.
Lemma 5 (TV to Regularized Suboptimality). If Be-p T x)) T² æ))
& then
TV
under Assumption 4, we have SubOpt(T) & 2E(Cmax + Vimx) and (T) & 2c( Cmax + Vmax)
Proof. We define shorthand Er,
log mo
= Ee p().ppp)))mmmm (vlx) log (a(() (
Then we have:
SubOpt(mT) = J3(TT) -T(
= Jo( - Jo( + A
log
π
HE 2r
log
T
Tz
Jreff
< 2cCmax + FE log π
nT
BEE +
log
TTref
2eCmax + +1Ex log π
π
π
n
BIE
log
+ BE
log
BE
log
iTrof
Wref
Kree
Tre
2eemmax + Flx
log π
BI
π log
BE 3T
log
7
Wref
< 2eGmaxx
+ Blx
log Trref
π
BE
log
π
Wref
where the first inequality follows from Lemma 4 and the last inequality non-negativity of KL divergence:
T log π
- BE
BmE DRL. (# [) x))] 0.
?
Finally, we have:
log π Tree
n
SubOpt(TT) < 2cCmax + BBE
log
nref
( x)
2cCmax + 1BEzpp
Σ
(T( x:) - x x:) log
T (y) (x)
DEE
T ( ())
2eCmax + BEEup
Σ
( x)) ( =)]
log
T. (y
yE
20Cmux + 2V 2V Beup [T (x) ? 1=)[[T]
2c(Cmax + Vmax),
27

---

where the third step uses triangle inequality and fourth step uses Assumption 4. Alternatively we can bound DEg=p
ΣyEr
( x) T.(2 x) log
*(yl))
as follows,
(ula)
T( x)) log T (3) x))
_E
Σ geyy
T( x) - T x)1
E
2V mm Bowp 1| (x) *²(Y [
2c Vmaxs
Finally, Regy(n) -Ei=1 SubOpt(T) (T) SubOpt(T)T 2c(Cmax Vmnx
The advantage of these results is that we can focus on bounding total variation of the learned policy with respect to Th and the result for sub-optimality and regret for either regularized or unregularized objective follows.
A.3 Leaning from Supervised Feedback
We recall that we are given a dataset user-edits D { Di, D/) where for every i [n], we
have 2; p(-), y Ther(- zz), and 1/4 q(- 24, 9:). The supervised fine-tuning (SFT) approach learns the following policy:
Σ log क (y\ m.)).
SUP arg max
TEll
Theorem 4 (SFT Result Full). Under Assumption 1, Assumption 2 and Assumption 4, we have that 4(caur V log(In ml/6) then with probability at least - 6:
when n 3
e²
( & + 2(Cmax + Vimx) (Amle(n, 6) + min (7maxx D n Mmax D1/2 (Tret, xI.
where Mmaax sumption I and
ming Tmin (2))( and Tmax V VTEp [(1 Ymmia (2)) for (min(() defined in As-
DiTret x².) Expp Fref( Y x)) - (f xx) mm)
Proof. The Bayes classifier of this log-likelihood problem is given by g Tref X A()) where Wre x)) Σyey q(v' y, oo x). From Assumption we have realizability and so using standard siniee-saple guarantees for maximum likelihood (e.g., [23]) we have
Ex-p [| x)) - q Feaf (Y x))l Amte 5) = V log In
(17)
with probability at least - 8 for any fixed 8 (0, 1). Using triangle inequality we have: Ez-p
[SUU(( x) +²((Y rv] < Ex-p [| (x) - Tet x)]
E Ex-p [ Tror (Y xx) x!(Y | [x)]
Amte 8) Exzp [(1 - 7min 7min (1)) Troe ()) ==((( [v)
where the last line uses Lemma and Equation 17.
We compute two distinct bounds for the last term and consider the tightest among them:
Egnp p [[1 - 7/ioo (mmi(()) Free (Y x)) - = [x]]
Ezmp [[1 Ymin "min [x Expp Tres(Y (x) . ( x))
TV
(6)
Ez-p [11 Tmiia 2
Tre x)) - : x²) [v]
28

---

where inequality (i) holds because of [0, 1]. Moreover, the following inequality also holds:
and (ii) because the TV distance lies in
Bg-p [(1
Tmmnn Imin( (2)) Fref (Y xr) + x)) rvl $
(1 min Mmiin (z)) Ez-p Tree 1 [x) - - ( x:) [v]
Thus we conclude that,
Ex-p AsSup(Y (Y ax) -Y ( x)) Tv] & (n,
min (1 (1-
-L--iinge min m-inninn ( n.),, V Bus [(1 Ymin (x)i(=)) (Tret; m.)
We can use Lemma or Lemma to get bounds on optimally and regret. In particular, using Lemma 5 we get:
(ubOpt(fsup) & 2(Cmax + Vinx) Amte((2, (n, 8) + min 7mmax DiTrof n.), Amax D222 Trof T.
Reg, (fsup() & 2(Cmax + Vimx) Amee (n1, 5) + min (/hnax (TTrr ) Thnax
D D2/2 Troef T T.
where recall that 7max = 1 - miny Ymin (2) and 7max = VEep [[1 Ymin (min( (x)))J]
Finally, setting n A(con Vina 1 log(In (1)6)) we get the desired result.
A.4 Theoretical Analysis with Preference Feedback
Given the user edit data D = {Tx, Vs. 1/4 1-1 we first create a preference learning dataset D Prre {[z Di. V., 24) > as follows: for every [nz]. we sample 2; E Unf (n and if 2 then define V = Yr and 87 = v/. otherwise, we define y, = 3/ and 874 = ys-
Given the LLM policy class II, induce a binary classifier class C (fr (x,0.0) ((+1))
Vι E II] as:
C= Uf+ (I, - A((+1)) VT I1), where
T x) Teat (5' x))
llog TŪ x) Toat (ū xr)
z log The size of C is the same the size of the policy class II. We define the empirical log-likelihood (TREF(T)) for a policy л E II as:
Vx E II, fa v.
(18)
1 n Σ log Jm (z: 24. Dt. 1k)
ÉPREF(T)
(19)
Using the construction of D pref from DD we can express { as:
TQy xz) Tyet (y/ xx) Trat (y/s xx1)
log T ( x;))
(pREF
log
log
(20)
which is the standard DPO loss without 3 factor. Our theoretical analysis uses this variant of DPO without 3 since in our analysis 3 is given value with respect to which Assumption I holds. In our experiments, we use the standard version DPO with the 3 term. Given the model class C, we maximize the log-likelihood:
MPREF = arg max (PREE
(21)
F,EC
Each datapoint (xa, We, T.. =i) Ppref is sampled IID from distribution and let this be denoted by D preet(2s (x, D. 5/ 2).. From the construction of Dpref we have:
x)q(2' x. V). pp(x) That z)q(ī x, T.
1
D Prnet xr, T.
(22)
==
29

---

Here we show in detail our calculations from Section 4 of how the distribution pper(z= = 1 2, ". satisfies the Bradley- Merry Assumption. As we are using variant of DPO without 5, we will instead use variant of Bradley-Terry Distribution well which has 3 temperature.
Lemma (Satisfying Bradley-Tere Distribution). Under Assumption 1, the probability of preferring j' over y. i.e., < V. in our preference dataset Dyeef satisfies the Bradley- Terry assumption with respect to the cost function c. Formally, for every x E supp Ú, 3/ E y we have:
(xx,)) c(a,gg))
Dpeel 1 1 x.
3
where (t)= = T+exp(-t) is the sigmoid function.
Proof. The probability of the event 1j = z/ is given by Prees= = x, U, 5') which is also the Bayes' classifier of our classification problem. Starting from Bayes's theorem we get:
D Dpor(2 1, x. 0.07
D Ppper(2 = 11| (1,08.8) = Dpre(== 2. 37 + D Dprer(z -,,, ;.,
Trofl (ū x) ( x, 0)
(using Equation 22),
Teer x) I, 5) Tref
[))((
xr ""
Tre (3 x) T
())
(using Assumption 1),
To w)) (5' ar) Teet (i' [x))) (ū (x)
exp( c( 8))
(using Lemma 3)
exp(-c(2: 0)) expl cc, 0)/3)
(1, 5) cx, 0
8
From Assumption 2 we have oh E II which gives
n. (3' x))
8
(ū x)) Tret 17 x))
f- = -. = 3 log
3 log
Tea (j' ())
c(z 5) cz 77) = a
B
This means that we have realizability for the classification problem in Equation 19, We can then use standard MLE guarantees state next result.
Lemma 7 (MLE Guarantee). For any ō (0, 1), under Assumption 2 we have:
2
E( f (Z x. G. Ø¹) Drre( xx, (2..8)) < where Dang(at
(23)
n
(2,, U. 1/) the marginal distribution of Equation 22 and given by D D (x. 9.1/
p(m) (wret x))(7' x. ū) + T x))(ū
x,)
Proof. This is a standard MLE bound under realizability, Proof follows from using Theorem 21 of Agarwal et al. [2] and applying Chernoff's bound EI X < E[[?]]
Using [pp - g1vv~ = 1pp - <ll2 for two finite value distributions p, q we get:
| [ x., 0.00) - D pere 7 x. ū. 0/ lIv
| (Z- 1 T. 5
D Peeet Z 1 x, G. 87)
(24)
For convenience, we define a notion of implicit cost é(æ, y) given by: tPREF(( æ) W(x) x)) exp(-ē(x, w)/
(25)
30

---

where (x) = EVey Tyerly 2)) exp(--(xx v3)/3) is the normalization constant. As - we
have TPREF - TT. Rewriting Equation 25, we get:
((x, - -loog *PREF (y x)) Tee (1) x))
+ 3 log ((r))
(26)
which gives
fkne Z = x. 2..0077 = [( 17) (dz.v)=
c( (7
(27)
B
Using Equations 24 and 27 and Lemma 6 simplify Equation 23 to:
((2,8,8') Dpee [oo (clz.)) - c(2,,0)
[c(z, (7) c(a, 171)
2
11
In n
(28)
5
To further simplify this result, will need certain properties of the sigmoid function (Lemma 13 and Lemma 14). We will also assume Assumption 4.
Let Z1 (E(x.0) (((x.i))ccm.8))// and Z2 (elz.ū)-c(x.""))) We have [Z 2 from Assumption and [Z2l E Cmax since c(x,z) y) E [0, c. Then starting from Equation 28 we get:
lo (á(1, 0) f() or oc((,, 0) x(M)
(o(Zz) [ZZ))l
min(o' (Zz), 0'( (Z2)) I1 Zzl
(using Lemma 13), (using Lemma 14)
min (0 (2 mux/ B), o' (Cmax) I1 Zzl
This gives us:
(( [((x, ()
c(xxx (c(xx)) -od.,,))(...///) (().l/))l] & Gu.
(29)
0 where o (Cmre)
[ 2 In 8.
(emc))
Using Lemma 3 we have:
T. (ū x) Teet ( x))
(x,0) = - log
+ Blog Wx(x).
(30)
Plugging in values of from Equation 30 and from Equation 26 in Equation 29 we get:
tPRRF (ú x) Tret
t 17'
x:)
T x) π (j' x))
E
Blog
3 log
(13 log
- bog
$ 6n
x.)
Tee (j) x))
Teat x)
ne y
x))
which simplifies to
mPREr ( x) n
APREr ( xx) nx (j' x))
E ((r.g-5"-
log
3 log
Cn
(ū x)
We now invoke the concentrability assumption (Assumption which we restate below for conve- nience:
Assumption 3. [Preference Concentrability] For any E II we assume:
B(z,0.81-Q-
Bog
[( 3 log x((iz) x; (t)xz)
":(""" in)
CPREF
E3 (r. 000000000
|logg
x(f(zz) x; (011x)
f log a(jl@)
(((0)))
where Qx }e(x)) (TT
1)) (ū x) n (i/ 2))(ū) z)).
31

---

to get:
PPREF x3)
APrEF (U x 2 i x))
x))
3 log
3 log
CPREF Ga.
(31)
~ (j' x))
where we used the fact that Deef = Qu Concentrability style assumptions common in analysis of RL algorithms [14, 44]. Our assumption is similar to the one in Theorem 1 of Rosset et al., [44]. the main difference is that our assumption is designed for learning from user-edits while that of Rosset et al., is designed for iterative learning with preference feedback.
Simplifying Equation 31 gives:
PREF 7
I) x)
OPREF
(x Q
log n. ( 2 TREF ( x)
(32)
We lower bound the left hand side of this assumption:
PREF
77'
2
log
y
APreF y
Frwerr ( m ())
7.. (7/) (x)) Ayeer xz)
(using log(()) [t-1|
for & from Lemma 15)
Rrwee (8' = (()=)
(z) Aaree xr)
PPRE 0/ ())==" (j a)) n 2T/ x) APREE (ū x))
pPREI- |x)) (ū x)) + T (ū' x)) ATREE xx)
1
= up
Bemp
(T
[x) x)) 2 SPREF (y x))
(using the definition of Q
2
Jyyey
1
Exup
E
A ()) x)) 20 (iprE (5 x))
(triangle inequality)
2
JEy
B'Ey
Epxp
Fref(Y
2))
mf (Y x))l
Combining this all we get:
OPREF
2
II 8
1 x)) [v] & min (a'(2V max/ BB, o² (Cmux)) Ez-p [Iee (Y (x) (
ln
(33)
We can wrap up the proof with the application of Lemma 5.
Theorem 5 (DPO Result). Under Assumption 1, Assumption 2, and Assumption 3, Assumption 4 we &CtngS (Com V.n. ) m
have that when >> min(o'(2V. (022Van/9)... Ea then with probability at least 1 1-8. 8:
1ab00tt(FRRES) t.
(34)
Proof. We set n such that 2(Guu + Y tion 33 and Lemma 5 gives SubOpt A.REF) & and Reg, (TRREF) & cT.
CrnEF
2 In [TI
which together with Equa-
minfo (2Vana / (eau))
We can simplify Theorem and Theorem if define Vmax to the max the quantity defined in Assumption 4 and Cmax-
32

---

A.5 Learning from Cost
Throughout this section, we will assume that the cost functions are bounded. Assumption 5. The cost functions I F satisfy [f(x v)1 & Cmax for some Cmax > 0 for all a E X and y y
Provided the true cost function satisfies E FF where F is a class of functions from the space of pairs X' x ) to the real numbers we can estimate the mean cost function by least squares regression on the observed costs gathered by Tref-
1 Σ (f(es, - c,,²²
arg min feF
The estimator f satisfies,
Lemma 8. Let and 8 e (0, and (( No: aili 1 be dataset of prompt answers and costs. Assumption 5 holds then
Ex-py-matll (f(z. clz, w)) cmax log
|F
E
and defining the confidence set of plausible cost functions as
$= s.t. Σ (F(Z). (a (( & b. c log
[FF]
(35)
for a universal constant 3 1. We have e E F and Ex-ny== [(f(x,)) c(z v))? b e log (15) for all f E f with probability at least - 8/2.
<
Throughout this section we will define as the at least 5/2 probability event defined in Lemma Given cost function f we introduce notation to define the policy optimizing the regularized objective where we add f to the notation.
T x.)
argmin E Be- (x-x)x(((11) Hell
[(xr,)) v) Blog
Tree (y) x))
and satisfies m.. - (y|z)
exp( Mgga) (ur())))
where Z(x) Eny exp (-flaar)) Trer(8/1x)).
2(x)
Pesimism As we have mentioned in the main, we consider the following pessimistic estimator of the cost function,
f(x, v) = max f(x, y)
feF
where F is defined as in equation 35. It follows that गRı defined in Equation 6 satisfies मर.. m! 7.8 Moreover, Assumption implies,
[f x) & Cmax
max x xEX
A.5.1 Regularized Cost Analysis
Proposition 6. For any support(p).
(a) -Cmax & E (1))
n Teef
f.8
(
x))
( - B log( ((x))
f(x. v) + log
Cmar-
(36)
(
33

---

Proof. Equality (6) follows by substitution. We now proceed to prove the inequalities (a) and (c). Now observe that ((x) By--a.-((l))) [exp(-f(as g)/5)] Notice that g(r) _ log (1/y) is convex function. Thus, Jensen's inequality implies,
(1k)) (exp (-
& Ey (-X) 18 log (exp (S(o, w)/3)1
a [f v)]
Gmax
where inequality (1) follows by Jensen and (it) by Assumption derive lower bound we need to upper bound E By [exp fx, v)/ 8)). Assumption 5 implies,
[(x) = (a eexp (- (--f(z,))) < exp (Cmax/3 (3)
Thus,
8 log (1/Z(z)) >> -Cmax.
This concludes the proof.
Proposition 6 implies that
R..3 E [I(f)) = { s.t. B f(x,8) v) + 8- log
[())2)) xz) Tref (ulzz)
< Cmax Vo E support
(37)
For the next few results we condition on the event Because the function f is not in the class of cost functions our results will depend the Eluder dimension of F. Pessimism allows to prove bounds depending on the concentrability coefficient the rref sampling distribution. Assuming access to new prompt dataset ( we analyze the following algorithm,
tRL = argmin
Σ
E Sy)n((12)) [f(z, v) + 8 log (T(v)) (ul z.))))
mell(/)
where
IIf) u(f)= ( T s.t. (y-m(-la)) ((x.)) v) + Blog
[()(2)) Tref(y (z) )}
< Cmax Va e support(p)
}
Equation 37 implies ! E II(f).
In this section we'll write the sample complexity of our algorithms in terms of the Eluder dimension of F. This is a measure of statistical complexity of a function class first introduced by [45] and is based on the notion of e-independence defined as,
Definition 1. (c-dependence) Let G be a scalar function class with domain Z and c > 0. An element is e- dependeen on {z1,...,,2 V (g(2)) - (² & 6 also satisfies g(z) - (z) & 6. Furthermore, E Z 6 independent of (2) )nn) w.E.t. G if is not e- dependent on 21 Zn).
S w.r.t. if any pair of functions g, g' G satisfying
The eluder dimension defined as, Definition 2. (c-eluder) Let G be a function class with support in set Z and values in R. The e-non monotone eluder dimension deluder ( of G is the length of the longest sequence of elements in Z such that every element is inddepenndnnt of its predecessors. Moreover, define the e- e-euuder dimension detuder (9,c) <) as ( = max deluder (9,c).
The following eluder dimension lemmas will also prove useful in proving sample complexity bounds. The first one is a restatement of Proposition from [45].
Lemma 9, Let (ares T- be an arbitrary sequence X x y. When holds,
4b car log
[F]
(1f(2') (c22,6)1 2) &
1+
detuder (F,e).
22
34

---

The result above implies. Lemma 10. Let (2/e 32 be arbitrary sequence in X' V. When & holds,
(f vii c(a'e: 2f2)) & 17b caax log
[
log(n) deluder (F, T).
for the same unversal constant > 1 as in Lemma 8.
Proof. We use the same proof technique as in Lemma 2 from [45].Let wi = (f(2)) 11) cl2'e, 111)
Let 81, in be the permutation of indices to such that wi, >0 ... 2 W/
Let parameter to be specified later on. For t [n]] such that Wi, notice that,
4b c log
[i
(
< 1 (1 cl > wi.)
$
datudor (F, wis)
4b 2 log
(8)
1 1
deluder (F,T)
Wii
2
(ai 8b Caax log
detudor(F,
w
where (i) follows by Lemma 9 and (ii) follows because detuder (F, monotonically decreasing in the second argument. Inequality (1i)) holds because wi, & 4c And therefore, for any such that wi,
max
have,
8b cinu log (155)
w $
detuder (F,
and therefore,
7 8b
2
max log
[F
2 1
> r)
deluder (F,T) & 16b-c log 8
log(n) (r)ddd))deer(FFFF (F,
2-1
8-1
And finally,
w
1 <T) T
And therefore we have,
[F
Σ W & 16b 2 log
log(n) detuder +
Finally, setting 7 Cmax/ nn we get the desired result,
[
w & 17b Caax log
log(n) detuder (F, Cmax/ /n).
8
Additionally our results relies on the following variant of Bernstein inequality for martingales, or Freedman's inequality [20], as stated in e.g., [1, 8].
Lemma 11 (Simplified Freedman's inequality). Let 71,..., Ze be a bounded martingale difference
sequence with [Zel & R. For any 8' E (0, 1), and 1) (0,1/R) R), with probability at least 8'.
12/11 log(1/8")
Z E=1
(38)
17
E=1
where Ee- [:]] is the conditional expectation induced by conditioning on Zi,... Ze Ze-1-
We will use this notation to denote conditional expectations throughout this work.
35

---

The following Lemma will prove useful in our results relating the in-distribution (Tret) enror of the pessimistic cost estimators w.r.t. the true cost function c. Lemma 12. The following inequality holds,
c(z, P({ E [ (f(x,)))c(zz c(z, v)) <o
( (F, 1/m) log (og((FF([]])
} > 1- 1-36/4
P
Proof. Let 96 a fresh set of samples from p x kref and consider the martingale difference sequence Zi = (f(2'e v/ee) - c(z'e v/1))² p. (f(a,
- c(2,, w))" indexed by f E F.
It follows that 12/1 < 4c for all & E [n]] and all f Moreover,
E x /
21
4 max E
[ 8c Bo ma (((o,))-c(o,))))
c(2,
Where the last incquality holds because.
E
| Ex-1((f(a'e v/6)
([a)))][===---aaae Ex-pu-mantll) (f(x. c(2,
2 (. (f(x,)) cc y))
(39)
Using Freedman's 's inequality from Lemma 11 we conclude when & holds,
Σ (f v/) - ((=').1))) & n Po v) c(xx u]
log (og(2)F1/8)
Synee max By Be-
(f(x,)) - c(x, y))²
"
F
log(2 F1
(1 + SrJc max T log
(68)
2bc log
|F
+ Sc log F1
8
(cmmx log( (1/5)) = b2 c log(1F1/0))
(40)
simultaneously for all f ft with probability at least - 8/8.
Where inequality (i) holds because of conditioning on E and Lemma 8 and (ii) follows by setting
Scau
Let 8' be the event where & and 40 hold. In this case,
Σ (lla v/) c(''e v/)) < b' 6l log((F1/8)
for all e [0.... n].
We consider a new martingale difference sequence ( {Ze)²_, defined
Ż, Em.eel((a y) c(z, v)) - (f(2'e vv) cla' 21))
The following bounds hold, İel & 4c for all e [m] and all f F. Moreover,
E--1
4c maa Ee-1 [
Be (F(2. y) c(2, y)
where the last inequality has the same derivation inequality 39. Freedman's inequality (Lemma 11)
implies,
nET... v) ()) Σ (( - c(2(2116)2+
log (8/8)
Synce Er,,d [[F(x,)) v) c(2,,))²1
n
36

---

with probability at least - 8/8. By setting 16c
1
we conclude,
32 max log(8/0)
E [(f c(1, 8)) &
Σ
(f vk) (1))2
(41)
with probability at least 6/8.
Lemma 10 implies that when & holds,
34b 2n log
[F -log(n) deluder (F, Cmax /n) 32c log (8/(8)))
E E [(((2,)) c(xx,)))) v))
7
n
1366 cmx log
log(n) doluder (F, Cmnx/
max
log
+
log(n)
detuder (F, Cmax/ /n)
O
n
Combining & and the conitions for equations 40 and 41, the desired result holds with probability at least 1 5/2 8/8 8/8.
Theorem 7. The policy NRL satisfies,
Jo((TR))= & ()( o (1108(111/0) + C.
log( (n) de)deeeer (F, Cmax/ /2) log (og( (FF
7
n
cuf (v|=))
with probability at least 1 where C.
B
is the optimal policy concentra-
Tinet [y|z]]
bility coefficient.
Proof. Throughout this proof we will condition on the event E. Since T. 7. 11(f) and by definition of fTRL
Eyn (-12.) [f y) log (y)=)) Tre (x))
E n
f (12.))
f(i;,)) v) + B log (ree))//))
(42)
n
Since -mux & Et (m) [f y) + 3 log (T(y)a)//Tedd (v(x,)) & Cmax for all x X and all
([()) (see Equation 37) Hoeffding inequality implies that,
Σ Jo(T, J..i)
Eu- Pa-ppy-alla) [Ja(T, J f.x) < 20max
log(
(43)
n
holds with probability at least 8/4 for all II(e) simultancously where JB(TT, f,z) Bywr((1x)) [f 3) + & log
(B)=)))
Combining equations 42 and 43 we conclude that with probability at least 1 1- 8/4,
log( 2 (1]
E E [J < E
P (v)x) [JoTT fF,z) + 2cmax
log (2) (1/))
Bappoan (wla)) [Jg(T .f. f.z]] 20max
(44)
where inequality (i) holds because f. is achieves better regularized objective value for cost
than c.B
37

---

Moreover, pessimism implies that when & holds,
(tl [c(r, v) log(fRu (ulx))] &
Ep- (112 (f(x,)) (log(/aullll))//me((1///)
Combining these results we conclude that,
1og(2
Egxp p [Jo(FRRL c, 2]] & Ex-p [Jo(= .fxx)] 2cmax
(45)
The final step is to upper bound E ==-ag==? (12) [[(x, w) terms of s~pu~r x (1x) [d v)].
To do this we will again use & and Cauchy Schwartz's 's inequality, in the next few lines we'll use the shorthand Eg [-] to denote (x-))-=x(=))]
n T (ylx) (f(x, y) c(xx,))
155..0 [f y) Exs [e(2, w)] = Esrret
T c. (v)xz)
Exrat
cIz, (f(zz 8) -((,)))))))
Brte
Traf (11l (r)))
c(2,, Finally we can bound the term Ennes [f(a, (f(a,,))c(1z, [
Lemma 12 implies,
1366 max log 8
[F
log(n)
deludee (F, Crnax (1)
E E (f(z, 3) c(xx, w))
n
with probability 35/4. And therefore,
[1
Ez-p [Jo(#tL. & Euup [J ,e, ,,,x))] + 2cmax
[ blog
log(n)
detuder (F, Cmax/ /n)
12 C.
n
with probability at least 1-8. 8.
A.6 Pure Online Learning Theory
In this section we introduce and analyze the regret of the Epoch Supervised Learning Algorithm (see Algorithm 2). This procedure works in epochs, in each epoch the algorithm observes prompts from p. and produces responses from a fixed edits distribution. The algorithm also collects edit data the form of samples q(-1az xx, y). During epoch e the algorithm uses policy He to produce responses and collect me datapoints of the form D. {Te,n Ve,n, 3e where V'.n g( g(-læn,es Mn.e).
The epoch supervised learning algorithm's policy is updated by fitting policy matching the edited responses in De.
Wet1 = arg max
Σ
log T x))
rell (D.D))D
38

---

This way the policy *e+1 is an approximation to q We consider setting where Algorithm interacts in 1, T rounds where each time step the algorithm given prompt 4, and generates response yr- We measure the performance of 2 through its regularized regret Regy as defined in Equation 2. Our main result is the following theorem where we show Algorithm 2 satisfies a (V'T) regret bound.
Algorithm 2 Epoch Supervised Learning
1: Start arbitrary initial policy Tref-
2:
for Epoch 1,2, ... [{(()] do
3: D. Ø
4: for 1....... m. do
5:
Given context Ze,n Agent generates Vee, We User edits it to ve.n
6:
7:
8:
Agent collects edit distance of Aetilye, ve.n)
9:
D. D u {F kenn)
10: Update the policy:
Me+1 arg max sell (a.y")ED
Σ
log TTy x))
Theorem 8. When Assumption and Assumption 4 hold then Algorithm 2 with inputs me
21m 2r, 5e 8/2e² and e(T) be the first index such that E -1 me < T and Ex me 2 T satisfies the regret bound,
Cyaxx + Vimx (1011/1-7mm)) (1-7min) ?mmnn)
T
3/2 Thn
T
Reg, $0
Thn
(T)
with probability at least
Algorithm 2 works over e(T") epochs. In each epoch the algorithm interacts with the "editor" for me steps.
At the start of epoch e we assume access to a response generator (e(y)z). This induces a population version of the target distribution P2+1 y. v') defined Pe+1(2, y, v²) Px(x))T(V x)q(s/ x, y). This induces the following ideal updated generator Te+1(y)x)) defined as,
T (y)z+) Σg Pe.
Px(xx)
Since we admit me samples during epoch e, the MLE guarantee ensures that the computed generator To+1 satisfies,
[T|
Ex-Ex [|("'+ (x) Fetr(Y' [x) &
In
(46)
me
8
with probability at least - Sc. Lemma implies
Ex-Px [ (Y' x) & (1 7min) Ex- Px [T xx) 7 [x)][T]]
(47)
therefore, combining Equations 46 and 47 we obtain,
2
|TI
Ex-Ex [e4'' -x( | & (1 Ymin) ExaEx TY'' =²(X' [x)][[]]]
In
me
de
39

---

we will now unroll this sum.
Let de
Bx-Px TTa(Y' x) - (Y' |xx)
and &. V
2 In
me
Then, unrolling the recursion obtain,
set1 & (1-7imin)ds + Ee & (7((1 Ymin ) + fc-1) + Le = (1 Yimin) 2 (1 Vmin (e-in Ee
so that det1 & Ei=1(1 Ymin)
) "s "min)"d 81. Therefore,
e-1
8e
Tmin)
----i Eo (1 -mmin) ( )0110) 81.
Lemma 5 allows us to bound the regret of epoch by.
Reg,,T
& 2(Cmax + V.nax) me
Σ
Ymin je-1-1g; Ei + (1 Ymin) je-15
2 m
Set me
so that Ee = Y ). Then,
(1 (1-Yai) 20
Reg.,T 2(max + Vina)) me
E: + Mmia) ) 18
6e
4(Cmux + Vima))
(1
Yunin
(1 Pimin)
In
[T|
dc
4(Cmma + Vmax)e.
(1 Yimin c+I
2(Cmax
Vmux
|T 8e
2
me
Timin
21n
where the last equality follows because me (1-7mm] Setting & 5/2e² we conclude that
2(Cmax + Vinx )e
11
Rege,T
2ln
me
1 Timn
de
for all e E N simultaneously with probability at least 6.
To prove regret bound up to time T let e(T) be the necessary epoch index such that ( < (e()) le me. Since the size me is increasing follows that,
me
In
[T]
T
8xcr)+1
2T
T
me
1
In
[II
7min)
8.(T))
the epoch
parameter e(T)
does not need
to
be greater than
e(T)
logg//cc mmm))
(T/2
2
since at this point,
logg/ca
T/2 In
27
T <
me-
40

---

And therefore
2Cma + Vmnx)
c(
2
Regr &
E 2ln
me
Ynin
8
c
O Cax + V Vinax
I
el e(T)
e(T) In
ml
Ymin
Cuux + Vinax O
T
T
(T Ie))
Ymin
1/2
O Crax + Vmna e. 3/2
T
T
Timin )2
8
with probability at least Finally since e(T) &
2log1/(1->mm)
(T/2 In
27 ))
the result
6
follows.
A.7 Support Results
We state useful supporting results below that allow us to prove our main results.
Lemma (Sigmoid Lower Bound), For any 2, y E we have:
[o(x) - avy) min (o'( oz) (w) 1
Proof. We have (1) - for any / R. This gives d((
et (Ite 17
e Observe that
Ite-t d'()) > for all E R. We also have
d²(t) - ta( 20 (o()
-o((t) 2e (())o(()
o'(t) (26 1)
= =((()) (1-e) et) et
Therefore, of(t) > 0 (convexity condition) if and only if 2 et. Therefore, is concave [0, o0) and convex on (-x0, 0].
Fix x, R and assume
y without loss of generality. Note that the result is trivially true for
y. As is differentiable over we have from mean value theorem a(y)-a(a d'(u) for some u E [æ,8].
8-x
v]. As dt()) > for all e R, we have (o(y) - a<x) = o'(t) ly x/. We now consider 3 cases.
1. If & y & 0 in which case we in the convex regime and d'(u) o'(x)
2. If > x> in which case we are in the concave regime and o d'(u) < d'()).
3. If y, then O, then we have d'(u) d'(x). And if > 0, then have
d'(u) (v).
Putting together, we always get o(()) > min (1), giving the desired result. Lemma 14 (Sigmoid Derivative Bound). [f|=| then d' (x) d'c))
Proof. If x > 0, then -c - As is in the concave regime of have o' o'xx) > (c().. Alternatively, i & 0, we have -c -c x & & c, then x is in the convex regime of and we have
('(x) 2 ('(-). As, (-cc) [T+ (++-<p a' (c). we get o'(x) t a'(c) in all cases. I
41

---

Lemma 15 (Log Lower Bound), For any > 0, we have [log(()
R--1
741
Proof. We have log(u) > 2-1 for u > 1. To prove this let g(u) = log(u) (w=1) then g(1) =
u+1
u+1
x++1 and g (u( = [u+1)? = a(u+1) This means g'(u) > 0 for u > 1, and so g(u) e for u > 1.
If & > 1, then [log(0] log(() = - 1 If 1, then [log(()] = log(1) (1/0)
1/t-1 1/2+1 2+1
This completes the proof.
B Related Work
Theory Although there is no theory prior work on edits that we are aware of, there a rich literature studying theoretical guarantees for RLHF objectives from preference feedback. The works of [61] and [53] pioneered the study reinforcement learning from human feedback via offline preference data, while works such as [46] and [55]. These preliminary results were mostly concerned with tabular scenarios. More recently, many works have been dedicated to the study function approximation regimes for learning policies from preference feedback in RLHF scenarios, such as the introduction of the XPO algorithm for exploratory RLHF [54] and [58] for provable guarantees for RLHF from offline data well as unifying frameworks that bridge online and offline preference feedback [13]. Beyond policy optimization, theoretical insights have also been developed for auxiliary phenomena in RLHE, For example, [26] characterized the seff-improvement dynamics of iteratively refined policies, and [56] analyzed the statistical efficiency of Best-of-N mechanisms preference alignment.
Modeling edits. Prior work on text edits has explored various modeling approaches, including generative diffusion models with edit-based corruption and reconstruction [41], latent vector repre- sentations of edits [24, 35], and modeling modular operations the editing process [50, 34, 3, 59]. While these approaches effectively model specific aspects of text editing, our work unifies multiple forms of user feedback - preferences, supervised labels, and cost within single framework. These types of feedback, typically studied in isolation, combined and analyzed both theoretically and empirically this work.
Using edits. Research on text revision often focuses on task-specific improvements, such as enhancing model factuality [12, 33, 7], refining academic writings [36, 30, 18], and enabling style transfer [42]. Similarly, work on code edits has primarily aimed automating program repair [57, 16. 41, 60, 49, 9]. In contrast, we focus on user-driven text edits that reflect expectations and preferences in practical use cases. Building on the setup proposed in prior work [? 4], we systematically investigate the algorithms and learning challenges associated with leveraging such edits.
C Additional Experimental Details
We provide details of our experimental setup here. We used the Prelude framework of Gao et al., [21] which available at https: pttps://gta //tg://itttt thub. com/ gao- s/preludd and has MIT License. We use the following models from HuggingFace: Llama 3.1 8b Instruct, Llama 3.3 70b Instruct and Qwen3-32B.
Grid Search.
We provide hyperparameter values of SFT Table 5 and for DPO and EarlyEnsemble in T. 6. We selected the best hyperparameters by computing log-loss of user-edits given context on a held-out validation set with 200 examples.
Hyperparameter
Values
Pretrained Model
meta- 1itta-1lama/ii L1amaa
ama 3. 1--1--B-IIntuu Instruct
Precision
Bfloat16 betas= 09, 0.95) (1.0e-5, 1.0e-6, 1.0e-7) {lep, 2ep]
Optimizer
Adamii,
Learning Rate
Epoch
Table 5: Hyperparameter search ranges for SFT.
42

---

Hyperparameter
Values
Pretrained Model
meta- beta-1lama/ LaallamaLLaaaaa Llama 1- In--r-nstcctttt
Precision
Bfloat16
Optimizer
AdamW, betas= (0.9, 9, 0.95)
beta
10.1, 0.5)
À in EarlyEnsemble (Bquation 7)
(0, 0.5, 0.75, 1.0)
Learning Rate
[1.0c-6, 3.0e-7, 7.0c-7]
Epoch
[2ep]
Table 6: Hyperparameter search ranges for DPO (A 0) and EarlyEnsemble (A > 0).
For Lateensemble (Algorithm 1) we set a 150 based on the approximate average edit cost the base model. We did not tune
Inference. We do greedy decoding and allow the agent to generate at most 1000 tokens.
Compute and Time. We experiments cluster with H100s and H200s. All experiments took few hours to train. The inference took 10min for each seed for SFT, DPO and Early BarlyEnsemble for each seed since LateEnsemble (Algorithm I) since we cannot evaluate datapoints in parallel for this.
evaluate each datapoint in parallel, while it took 30min for each seed for
Datasets. We do not release any new dataset with this paper. However, used source articles from four datasets provided below. The link to the dataset contains other details such as size of the dataset, samples, and license. We urge readers check the current status of these datasets in context of this work, and recommend them to avoid using any dataset that has been deprecated.
Domain
Link to the Dataset
P
https://github com/ wowmmooggooddeegid////gllll/ https://huggingfe co/ Co//attteee//SShortta//AAAAAA--a
Arxiv
BillSum
https://huggingface co//ataaettt/Faaallltee/bllllll
Elsevier OA CC-By 0
https://huggingfacee ca /o//aaasttt/rrree/eleeeeeee.....
Table 7: List of publicly available datasets that we use for training and evaluation.
We provide additional experimental results below.
Test-time plots. Figure 3 shows the cumulative edit cost at test time for the email writing setting in Table 1. We also show cumulative user-edits at inference time for the transfer learning setting in Table 2 in Figure 4. The trend is similar the trend in Figure with LateEnsemble mostly converging to the performance of the best method. We can see that all domain except email writing with weak user, the performance of LateEnsemble gradually drifts towards the best performing method as we would expect. The anomalous setting of email writing with weak user may need more test rounds before LateEnsemble converges the best method.
43

---

100
EaryErmembee LateCnoemble
EarySnsemmde LnteEnsebble
Curmulative Edit
Cummulativv Edit
60
60
80
40
60-
20-
25
150
Rounds
Rounds
(a) Email Writing with Strong User
(b) Email Writing with Weak User
Figure 3: Cumulative User Edits test time corresponding Table 1 for the different setup. For each round, we show mean and standard deviation across seeds.
200
Bass STT DPD EiryErsembie LateEnsemblee
Base
SET
150
3PO
Ea-tynnsemble LateEnsemble
Cummulative luser Edits
125
75
75
25
50
75
300
125
150
375
200
25
30
75
100
125
130
175
200
Number of Thst Rounds
Number of Text Rounds
(a) Summarization Domain with Strong User
(b) Summarization Domain with Weak User
SFT
DPO
Commnaaltive User Edits
60
80
Cummuuattvve User
40-
60
20-
20-
75
15.
175
25
50
125
150
200
Number
Rounds
Number of Test Rounds
(c) Email Writing with Strong User
(d) Email Writing with Weak User
Figure 4: Cumulative User Edits at test time corresponding to Table 2 for the different setup. For each round, show mean and standard deviation across seeds.
44


Reviews:
[{"review_id": "ssUNemOMf3", "reviewer": "Reviewer_U2M6", "summary": "The paper tackles fine‑tuning LLMs from user‑edit logs—situations where a user edits an LLM’s draft. It formalizes how edits simultaneously provide: (i) supervised targets, (ii) pairwise preferences, and (iii) a reward signal proportional to edit distance. \n\nThe authors derive sample‑complexity bounds for learning from each signal, reveal their trade‑offs, and introduce two ensembling strategies. The LateEnsembl, UCB bandit that picks among separately fine‑tuned policies, which yields the lowest edit cost on simulated email‑writing and summarization tasks.", "questions": "1. Could you clarify how critical Assumption 1 (the contraction with rate $\\gamma_{\\min}$) and other assumptions are in practice? For instance, if real users sometimes do not fully correct every error (violating the contraction property) or if $\\gamma_{\\min}$ varies across contexts, how would that affect the learning outcome?\n\n2. What specific obstacles prevented implementing Eq. 6?", "limitations": "no", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "1. The paper is strong in theoretical quality. And it  tackles a timely and practically significant problem – leveraging naturally occurring user edits to improve LLMs. Unlike expensive hand-labeled data (human-written gold answers or explicit preference ratings), edit logs are abundant and user-generated. The authors are among the first to provide a unified framework for this scenario, identifying that preferences, demonstrations, and reward signals co-occur in user-edit data.\n2. The paper proposes a simple yet clever ensembling strategy to get the best of all worlds. Instead of committing to one feedback type, the approach either (i) blends the training objectives, or (ii) trains separate models for each and then uses an online bandit algorithm (UCB) to choose which policy to deploy at test time.\n3. A notable weakness is the lack of real human user data or human evaluation. All experiments rely on a simulated user (an LLM that performs edits based on preset preferences). While this is a reasonable proxy to validate the concepts, it remains uncertain how well the proposed methods would work with actual users. Real user edits might be noisy, inconsistent, or exhibit biases that a simulated LLM user does not. The paper would be stronger if it at least discussed or tested the approach on a small real-world dataset (if available) or provided a plan for such validation. \n4. The work identifies three feedback modalities (supervision, preference, reward), but in practice the pure reward-based approach (RL with edit distance) was not fully implemented in the experiments. \n5. The theoretical analysis, while insightful, relies on several strong assumptions (explicitly labeled Assumption 1–3 in the paper).", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback. We address their questions below:\n\n## 1. Clarification on assumptions and their validity\n\nWe first provide some important clarifications regarding our assumptions.\n\n> “How is the learning outcome affected if real users sometimes do not fully correct every error (violating the contraction property)”\n\nWe do not assume that users fully correct every error. In fact, this would be a very strong assumption. The contraction lemma states that the user will improve a given policy's response with the rate of improvement determined by $\\gamma_\\min$. This does not mean the user will fix all issues in one round of edits. Instead, what this implies is that if the user keeps on editing a given response to make it better and better, then they will eventually converge to their optimal policy. This is a much weaker assumption and is similar to our lived experience, where constantly editing one’s work (e.g., for a paper deadline) leads to an eventually acceptable version.\n\n> “How is the learning outcome affected if $\\gamma_{\\min}$ varies across contexts”\n\nThis is a great question. Suppose we have a context $x$ dependent value $\\gamma_{\\min}(x)$ where Equation 9 holds. The simplest way will be to define $\\gamma_\\min = \\min_{x \\in X} \\gamma_{\\min}(x)$ as the minimum over all contexts. However, we can do even better as follows:\n\nFirstly, in Lemma 1, for any context $x$, we will get:\n\n$||q \\circ \\pi_{ref}(. | x) - \\pi^\\beta_\\star(. | x) || \\le (1 - \\gamma_\\min(x)) ||\\pi_{ref}(.|x) - \\pi^\\beta_\\star(. | x)||$,\n\nThen, in the proof of Theorem 4, we can use Cauchy-Schwarz’s inequality to get:\n\n$E[||q \\circ \\pi_{ref}(. | x) - \\pi^\\beta_\\star(. \\mid x) ||] \\le E[(1 - \\gamma_\\min(x)) ||\\pi_{ref}(.|x) - \\pi^\\beta_\\star(.|x) ||]$\n\n$\\qquad \\qquad\\qquad \\qquad \\qquad \\qquad \\le \\sqrt{E[(1 - \\gamma_\\min(x))^2]} \\sqrt{E[||\\pi_{ref}(.|x) - \\pi^\\beta_\\star(.|x) ||^2]} $\n\nwhere we now only require $\\sqrt{E_{x}[(1 - \\gamma_\\min(x))^2]}$ to be small. This is a _weaker assumption_ as $\\gamma_\\min(x)$ no longer needs to be close to 1 for all contexts. We will update our proof to use this softer notion of $\\gamma_\\min$.\n\nMore generally, we believe our results are robust to assumptions being violated. The biggest evidence is our experiments, where the assumptions may not necessarily hold as we use complex transformer-based LLMs and perform optimizations that may not necessarily converge to an ERM solution; however, findings from our theory still apply. Finally, assumptions similar to ours are routinely made in RLHF and RL literature. \n\n\n\n## 2. Challenges of Implementing Equation 6\n\nUnfortunately, computing an ``exact’’ pessimistic $\\bar{f}$ estimator for $f$ requires solving a challenging optimization problem (see line 157 for a definition of $\\bar{f}$). This is because enforcing the constraint set $\\gamma(\\mathcal{F}, \\delta)$ is impractical to achieve in real settings. There are some heuristics to implement pessimism using an ensemble, but these tend to be brittle in practice and/or have their own challenges. Our theoretical analysis nevertheless provides insights into the potential of this approach. We will add this clarification in the revision."}, {"review_id": "xMAsuEImnd", "reviewer": "Reviewer_kYmZ", "summary": "This paper studies the problem of post-training LLMs to learn from user edits. The proposed learning setup consists of two phases: the offline phase where a sufficiently large dataset of user edits can be used to fine-tune the LLM, and the online phase where the LLM interacts with users in deployment over a few episodes and can potentially be adapted. The paper considers three different approaches for post-training during the offline phase: supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning with a learned cost function (RL). The paper provides a theoretical analysis of the number of samples required for bounded regret guarantees in the three cases under certain assumptions on the policy class, and the user edits distribution -- further stating the trade-offs between the three methods. An early ensembling of the approaches is also considered. A late ensembling approach is proposed for the online phase: a bandit algorithm is run to select between the the different learned policies from the offline phase based on the cost of user edits.  Finally, the paper presents empirical results on two tasks (summarization and email writing) with strong / weak users simulated using LLMs -- demonstrating the theoretical trade-offs in practice.", "questions": "1. Could the authors provide some insight into why DPO is significantly worse than the base method on summarization with a different strong user (Table 2)?\n2. The empirical results in Section 5 use Llama-3.3-70B-instruct as the user and Llama-3.1-8B-instruct as the agent. I would expect these models to be very similar since they belong to the same model family (similar pre-training data, post-training approaches possibly leading to very similar generation distributions). Would the results in Table 1 be different if the user and agent are from different model families? I would encourage the authors to present some preliminary results on this if possible.\n\nOther comments:\n1. Line 59: \"one can use induce a preference data\" -> this doesn't sound right\n2. Line 70: \"approach perform better\" -> \"approach performs better\"\n3. Line 109: line 6 -> line 4 (incorrect line number from protocol 1)\n4. Line 267: \"The use an LLM\" -> \"The use of an LLM\"\n5. Line 291: \"We perform generate\" -> \"We generate\"\n6. Line 303: \"Equation 6 give challenges\" -> \"Equation 6 given challenges\"\n7. Line 325: \"this affect\" -> \"this effect\"\n8. Line 337: \"these maybe dependent\" -> \"these may be dependent\"", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper studies a well-motivated and timely problem, adapting LLMs to user preferences using naturally available user edit data.\n2. The theoretical analysis of the different post-training approaches (SFT, DPO, RL with a learned cost function) is original in the context of learning from user edits and the four assumptions on the user edit distribution and policy class appear reasonable.\n3. The empirical evaluation with a strong and weak user provides interesting insights (Section 5) about the trade-offs between the different approaches in terms of average regret with respect to the best policy.\n\n**Weaknesses**\n\n1. Protocol 1 is possibly incomplete: there is a mismatch between the line numbers in protocol 1 and the associated discussion on lines 107-113. For example, line 111 mentions line 9 in protocol 1 while the protocol has 7 lines in total.", "was_revised": true, "final_justification": "The paper studies a well-motivated and timely problem, adapting LLMs to user preferences using naturally available user edit data.\nThe theoretical analysis of the different post-training approaches (SFT, DPO, RL with a learned cost function) is original in the context of learning from user edits and the four assumptions on the user edit distribution and policy class appear reasonable.\nThe empirical evaluation with a strong and weak user provides interesting insights (Section 5) about the trade-offs between the different approaches in terms of average regret with respect to the best policy.", "rebuttal": "We thank the reviewer for their helpful feedback. We answer their questions below:\n\n$ $ \n\n## 1. Protocol 1 Line Number Mismatch\n\n**We want to clarify that Protocol 1 is correctly written and complete.** However, there are two typos in the text in the line numbers. On page line 109, the protocol line should be line 4, and on page line 111, the protocol line should be line 7. We appreciate the reviewer for catching this typo, and we will fix these in the revision.\n\n$ $\n\n## 2. Using different LLMs for teacher and user models\n\nWe want to clarify that we experimented with using a different combination of LLMs for user and teacher at test time. In Table 2 in the main paper and in Table 6 in the Appendix, we tested with a setting where the user was either a Qwen or Gemma model while the agent LLM remained Llama 8b. Please also see the discussion on lines 334-342. These results add a new dimension of generalization to our studies by showing the robustness to user LLM. In both cases, the results were generally the same as in Table 1. Notably, our late-ensemble approach performed the best overall.\n\nIf, however, the reviewer meant having different LLM families (e.g., Qwen vs Llama) for the user and teacher during __training time__ and then using the same families at test time, then we are also happy to add those results in our revision. This will be an easier setting since, unlike the transfer learning results, the model will be training and testing on the same combination of LLMs.\n\n$ $\n\n## 3. Why DPO Underperforms the Base Method (Table 2)\n\nWe have noticed some issues with DPO, including the well-known observation that DPO-trained models can be very verbose. This verbosity can be especially hurtful in the summarization task, where the goal is to generate a concise summary. We also noticed that DPO can enter repetition loops, where it repeats the same text, something that we fix with an explicit filter (see discussion around lines 306-308). We found that in this particular case of summarization with a strong user and a different LLM, the DPO was generating either really long summaries or really short ones, both of which are sub-optimal. We believe this is due to the instability in DPO, and testing with a different LLM model results in a higher edit distance penalty for these deficiencies."}, {"review_id": "ZNus3n0EG3", "reviewer": "Reviewer_k1w2", "summary": "This paper pioneers a principled framework for fine-tuning LLMs using user-edits (e.g., from writing assistants or coding agents). Derive bounds are first derived for learning algorithms that learn from each of these feedback types. A simple ensembling procedure is than proposed to jointly learn from these feedback types.\n\nThree feedback types, supervision, preferences, and rewards are integrated into a single theoretical and empirical paradigm. The authors derive sample complexity bounds for each approach, revealing trade-offs influenced by user behavior. \n\nTo balance these, they propose early and late ensemble methods. Experiments on email/summarization tasks show LateEnsemble’s robustness to user distribution shifts, reducing edit costs by a large margin. \n\nStrengths include theoretical grounding and practical algorithms; limitations involve future usage of pessimistic RL and simulated user validation.", "questions": "1. Could prompt-based adaptation (e.g., in-context learning) complement fine-tuning for user personalization? and How? prefer to learn more details.\n2. any discussion on handling users' edit data privacy topics?\n3. can you conclude your contributions at the introduction section?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strong:\n1. the motivation of using user edit feedbacks is relatively interesting and helpful in real-world llm usages. the unifying of perference, supervision and reward learning is a fine-grained method for learning from the users' edition datasets.\n2. three complexity bounds for the three learning approaches are derived under relatively clear assumptions. these mathmatical analysis shall help related researchers to obtain more insights of better leveraging sft/rl algorithms for this direction.\n3. relatively strong experiments on email/summarization tasks with simulated users and showed that the effectiveness of their proposed methods.\n\nWeak:\n1. prefer to learn more connections between the theoretical bounds with more real-world validities, currently the discussion is short.\n2. human evaluation is limitted, and more real-world human usage is preferred.\n3. scalability requires more evidences such as larger datasets. More discussion or experiments on this direction is appreciated.", "was_revised": true, "final_justification": "I upgrade my score by reading the authors' rebuttal contents and comments from other reviewers and appended results/conents from the authors. This paper shall be interested by a relatively wide range of researchers of this field.", "rebuttal": "We thank the reviewer for useful feedback. We address their questions below:\n\n## 1. Prompt-Based Adaptation\n\nOne can easily combine prompt-engineering approaches with fine-tuning approaches. E.g., one can fine-tune an LLM and still prompt it with previously seen relevant in-context learning examples (a popular prompt adaptation approach). While prompt-based adaptation is orthogonal to our main focus on fine-tuning, we will add a short discussion on results with a prompt-adaptation approach. We will try to share preliminary results during the author-reviewer discussion period itself.\n\n$ $\n## 2. Users' edit data privacy topics\n\nWe included a discussion on how to best protect the user’s privacy when handling their user-edit data in the NeurIPS checklist. Please see our response to the broader impact on lines 756-764. If accepted, we will use the extra page to move this discussion to the main paper.\n\n$ $\n\n## 3. Scalability and Human Studies\n\nIn this paper, we focus on laying the algorithmic foundations of learning from user-edits. Expanding our study to a larger scale and doing human studies are natural follow-up work. These studies can refer to our paper to decide what algorithms to run. In terms of scalability, both individual algorithms and their ensembling approach, including Algorithm 1, are easy to run with any standard LLM fine-tuning library. Algorithms such as SFT and DPO are routinely used to train LLMs with big datasets. \n\nOne advantage of using simulated LLMs in our study is that they provide reproducible and quick studies. This is especially useful for algorithm development, which is our main focus. The use of simulated users has increasingly become common due to improvements in LLMs [1, 2, 3, 4]. Further, doing a real human study by collecting enough training data and having to do repeated human evaluations can be prohibitively expensive.\n\nHowever, any industry lab that deploys a writing or coding assistant can easily test our algorithms with a real human study. They can do this by collecting user edits and using them to fine-tune LLMs. They can then do A/B testing to compute total user edits on generations of the trained model (assuming they have all the required permissions to store and train on user edits). We feel our research is particularly suited to make a practical impact in these settings.\n\nReferences:\n1. Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems, Sekulić et al. 2024\n2. Quantifying the Persona Effect in LLM Simulations, Hu and Collier (ACL 2024)\n3. Simulating User Agents for Embodied Conversational-AI, Philipov et al. 2024\n4. USimAgent: Large Language Models for Simulating Search Users, Zhang et al., SIGIR 2024\n\n\n$ $\n\n## 4. Theory and Real World Validity\n\nIn section 4.2, we provide a variety of insights on how our theoretical results relate to practice (lines 248 to 255) where we discuss when SFT, DPO or RL are preferable depending on different user edit ``parameters\" such as $\\gamma_{\\min}$ or the preference coverage $C_{\\mathrm{PREF}}$ parameter. When deployed, these parameters have interpretable descriptions that characterize how thorough user edits are when generating a response ($\\gamma_{\\min}$),  or the richness of the edits training data ($C_{\\mathrm{PREF}}$). If accepted, we will use the extra page to make some of these connections clearer. \n\n$ $\n\n## 5. Contributions in Intro\n\nWe will revise the introduction to explicitly list our contributions (theoretical bounds, unified framework, and ensemble strategies)."}]

Author rebuttal:

Meta-review: Summary: This paper introduces a theoretical framework for fine-tuning LLMs from user-edit data by unifying three feedback types (supervision, preference, and reward).

Strengths: All the reviewers agree upon the paper's primary strengths: its novel and well-motivated theoretical framework that unifies multiple feedback types from user edits, and the principled derivation of algorithms with clear trade-offs.

Weaknesses: The primary point of discussion was the reliance on a simulated user environment for experiments. Evaluating on real human data is a valuable direction for future work but does not detract from the current paper's contributions.

Reasons for Decision: The paper is accepted based on the strong and unanimous consensus from the reviewers. All reviewers agree that this paper's theoretical contribution: providing a principled and unified foundation for learning from user edits. And it is a significant and timely advance for a practical problem. While the reliance on simulated users is a limitation, the conceptual framework and algorithmic insights are of high interest to the community and outweigh this concern.

Discussion Process: The authors successfully clarified technical questions regarding the experimental protocol and theoretical assumptions for the reviewers, and directly led one reviewer to upgrade their score, solidifying a unanimous recommendation for acceptance.
Decision: Accept (poster)