Title: Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM post-training; Learning from interaction; user-edits; reinforcement learning; preference learning
Abstract: We study how to fine-tune LLMs using user-edit deployment data consisting of a set of context, an agent's response, and user edits. This deployment data is naturally generated by users in applications such as LLMs-based writing assistants and coding agents. The _natural_ origin of user edits makes it a desired source for adapting and personalizing of LLMs. In this setup, there emerges a unification of various feedback types namely preferences, supervised labels, and cost that are typically studied separately in the literature. In this paper, we initiate the theoretical investigation of learning from user edits. We first derive bounds for learning algorithms that learn from each of these feedback types. We prove that these algorithms have different trade-offs depending upon the user, data distribution, and model class. We then propose a simple ensembling procedure to jointly learn from these feedback types. On two domains from Gao et al. 2024, we show our ensembling procedure outperforms these methods that learn from individual feedback. Further, we show that our proposed procedure can robustly adapt to different user-edit distributions at test time.

Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward

Abstract

We study how to fine-tune LLMs using user-edit deployment data consisting of a set of context, an agent's response, and user edits. This deployment data is naturally generated by users in applications such as LLMs-based writing assistants and coding agents. The natural origin of user edits makes it a desired source for adapting and personalizing of LLMs. In this setup, there emerges a unification of various feedback types namely preferences, supervised labels, and cost that are typically studied separately in the literature. In this paper, we initiate the theoretical investigation of learning from user edits. We first derive bounds for learning algorithms that learn from each of these feedback types. We prove that these algorithms have different trade-offs depending upon the user, data distribution, and model class. We then propose a simple ensembling procedure to jointly learn from these feedback types. On two domains from Gao et al. [21], we show our ensembling procedure outperforms these methods that learn from individual feedback. Further, we show that our proposed procedure can robustly adapt to different user-edit distributions at test time.

1 Introduction

Post-training of LLMs has become a critical step in developing large language models (LLMs) in recent times. In particular, high-quality data is a valuable artifact for making LLMs useful for domains of interest. Modern approaches typically employ annotators to label a variety of feedback ranging from full human written gold responses to labeling binary preferences [17, 37]. However, these approaches are inherently expensive. In contrast, some recent works have attempted to use deployment logs generated by natural interactions of users with LLM Agents to improve these models [15, 21, 25]. In particular, for a broad class of LLM applications such as coding and writing assistants, user edits [19, 21] form a natural mode of feedback found in deployment logs. Motivated by this, we study how to fine-tune LLMs to a given task using user-edits in deployment logs.

Consider the application in Figure 1 where an LLM agent is being used as a writing assistant. In any given session, a user can query the agent at any time to solve a task such as generating an email given a short description. If the LLM's response is unsatisfactory, the user typically edits the response to fix the deficiencies. For example, the user may prefer a certain writing style that the base LLM agent may not be tailored to generate. We emphasize that these edits are naturally generated by the users making it easy to scale this feedback. Further, since the user has a stake in solving their own problem, we can assume that the edit feedback is of reasonable quality. In contrast, in RLHF applications, feedback such as pairwise preference is collected by hiring annotators to label a pair of

User edits the

response

LLM generates

the response

User queries the

LLM agent

O

ffl

ine Dataset of User Edits

User Edit Cost: 24

LLM receives an

edit cost

1

Preference Feedback

2

Supervised Feedback

Three Feedback Types

3

RL Feedback

Online Learning Phase

User Edit Cost: 10

User Edit Cost: 20

Cumulative User Edit Cost: 1250

Robust Mixing

………………………

………………………

………………………………………..

………………………………………..

………………………………………..

………………………………………..

………………………………………..

………………………………………..

………………………………………..

……………………………………

…………………………………………

………………………………………

……………………………………

…..

………………………………………..

Article: {user-provided article)

Please summarize the above article.

Farming,a part of agriculture,involves growing crops and

rearing animals for food and raw materials. It began thousands

of years ago, likely in the Fertile Crescent, and led to the

Neolithic Revolution as people transitioned from nomadic

hunting to settled farming. This allowed for a significant inc ease

inhumanpopulation.

Farming,asa part of agriculture involves growing crops

cultivation and animal rearing for food and rawmaterials

Originated It began thousands of years ago likely in the Fertile

Crescent,leading to the NeolithicRevolution

Transition as people transitioned from nomadic hunting to

settled farming.resulted in significant human population increase)

Piease summarize the above article.

Article: (user-provided article)

Article: (user-provided article)

Please summarize the above article.

LLM responses and typically requires a quality check. Finally, edit feedback is also convenient to provide since state-of-the-art LLMs are often able to generate a reasonable response that may only need small edits to make it suitable.

There has been significant progress in the development of LLMs over the last few years. However, LLMs are not oracle and cannot apriori adapt to a given user or task preference [47, 52]. A common solution is to prompt these models with the needed preferences and requirement often as a system prompt [32]. However, users in real applications are not prompt-engineers and may not list all their preferences in the prompt, or may even be aware of their preferences. Further, the needs and preferences can be complex, context-dependent, and maybe hard to describe. This necessitates using of user feedback such as edits to improve the agent's response over time.

Our learning protocol shown in Figure 1 consists of an offline learning phase that uses a dataset of deployment logs containing user interactions with the LLM agent. Each data point consists of an original context, along with the agent's response and any user edits. The dataset can be collected from a single user if we are personalizing to a given user's preference, or multiple users if we are learning more general preferences. This dataset is used to adapt the LLM agent. The second phase of our learning consists of an online learning phase during which the agent interacts with a user, or multiple users, to generate a response. We penalize the agent during this phase by the amount of edits, as measured by an edit cost, that the user has to perform to make up for the deficiencies in the agent's response. As in a real-world setting, user(s) during the online learning phase can be misaligned with the user(s) that provided data in the deployment logs. The goal of a learning algorithm is to effectively perform offline and online learning to minimize the cumulative edit cost during the online learning phase.

In machine learning, there is a long line of work on development of optimal algorithms given a set of feedback such as from rewards [51, 48], actions [43, 29], or preferences [17, 37]. As shown in Figure 1, an interesting property of our learning setup is that all three aforementioned feedback types co-occur in this setting. Namely, given the context and user edits, one can simply fine-tune the LLM on the user edits. Alternatively, one can use induce a preference data using the user edits as the preferred response over the agent response. Finally, one can also use the edit distance as a reward to perform reinforcement learning. This provides a rich space to develop algorithms. In this work, we initiate a theoretical investigation into learning from user edits. Our purpose is two-fold: one is to provide principled reasoning for our algorithm choices and compare them to other learning settings, and the second is to predict empirical phenomena that can help in deployment. We first provide bounds for three basic algorithms that use the three separate feedback, and demonstrate that these algorithms have different trade-offs. We then propose an ensembling approach that can balance these different trade-offs.

We evaluate these different methods and our ensembling approach on two domains from Gao et al., [21]. These domains evaluate the ability of an LLM agent to personalize to a simulated user's latent preference based purely on user edit feedback. We show that across a range of different settings, the ensembling approach perform better than algorithms that use a single feedback type.

2 Preliminary

Notation. We use calligraphic letters such as U to denote sets. We define ∆ p U q as the set of all distributions over a countable set U . For any N P N , we define the set r N s ' t 1 , 2 , ¨ ¨ ¨ , N u . We define the set of all texts as Y and the set of all contexts X . The context can include text along with other multimodal data such as images or videos. We define the set of LLM policies as Π ' t π : X Ñ ∆ p Y qu . Given a context x P X , and a policy π P Π , we can sample a response y ' π p¨ | x q . We assume that LLMs can accept multimodal input if the context contains multimodal data.

Learning from user-edits. In our setup (Protocol 1), the agent is repeatedly queried by a user to generate a response y given a context x . For example, in the writing assistant application, the assistant agent may be triggered to generate an email draft given a short prompt. The context x will include this prompt along with the content of the existing document and application metadata such as environment variables and the agent's calendar information. The agent will then generate a response y , which is the email draft. If the user finds the response unsuitable, they may edit it to y 1 . If they decide not to edit, then we have y 1 ' y . We will always state there is a user edit, and if y ' y 1 then we will call this an empty edit . We assume the user edits can be modeled with an unknown distribution q : X ˆ Y Ñ ∆ p Y q such that given a context x and response y , the user generates the edit y 1 ' q p¨ | x, y q .

We emphasize that the user performs the edits as they need the result for their own needs. This contrasts with the more commonly used preference data in the RLHF literature which is collected by asking annotators to label a pair of responses. As this is not a natural task, the annotators are often compensated. In contrast, user edits provide freely generated deployment data for performing never-ending fine-tuning of LLMs .

The necessity for user edits is due to deficiencies in the agent's response y . Further, the more edits the user has to make, the more deficient is the agent's response. We can measure this deficiency with an edit cost c ' ∆ edit p y, y 1 q where ∆ edit : Y 2 Ñr 0 , c max s is a suitable edit distance metric. The edit cost c is higher if the user performs more edits and 0 only if edits are empty ( y ' y 1 ). Following Gao et al., [21], we use Levenhstein edit distance over sub-tokens for defining ∆ edit in our experiments. However, our analysis and algorithm development are independent of the choice of ∆ edit. We are now ready to state our formal setup.

Formal Learning Setup. Protocol 1 states our formal setup. We assume access to a dataset of n user edits D ' tp x i , y i , y 1 i u n i ' 1 and a policy class Π . For every i P r n s , we assume x i ' ρ p¨q , y i ' π ref p¨ | x i q , and y 1 i ' q p¨ | x i , y i q , where ρ is the context distribution and π ref is a reference policy used to generate the agent response. The reference policy can be a pre-trained LLM that is used to warmstart the problem. This is a typical deployment dataset that is encountered in writing and coding assistant applications. This data distribution can be used to perform fine-tune policies in the offline learning phase (line 1).

After this, the agent is evaluated over a series of T episodes during the online learning phase . In each interaction, the world (which includes the user) will provide a context x t ' ρ (line 3). The agent can generate a response y t for this context (line 6). Finally, the user generates an edit y 1 t which leads to an edit cost of c t to the agent (line 5-6). The goal of the agent is to minimize the total edits performed during these T episodes (line 9). We allow the agent to perform online learning during these T episodes. However, T may not be large enough to perform a full fine-tuning. Consequently, any learning algorithm should try to effectively use the edited dataset D .

Our setup can be applied to both settings where edits are performed by an arbitrary set of people or when they are performed by a single person or a particular group. In the first case, we need to learn general preferences, whereas the latter is a personalization setting since we need to learn the preferences of a particular person or group.

We define a few useful concepts to enable us to state our formal objective. We first define the expected cost of a response y in context x as c p x, y q ' E y 1 ' q p¨| x,y q r ∆ edit p y, y 1 qs . Given a policy π P Π , we then define its expected cost as J p π q ' E x ' ρ,y ' π p¨| x q r c p x, y qs . We also define a β -KL regularized objective as:

Protocol 1 Finetuning LLMs from User Edits. An algorithm needs to implement the lines in brown.

Require: Given a dataset D ' tp x i , y i , y 1 i qu n i ' 1 of user edits. Also, given a policy class Π . 1: Initialize agent using D // Offline learning phase 2: for t ' 1 , 2 , 3 , ¨ ¨ ¨ , T do 3: World presents context x t 4: Agent generates a response y t , / / / . / / / -// Online learning and evaluation phase 5: User edits the response to y 1 t 6: Agent receives a cost for edits c t ' ∆ edit p y t , y 1 t q 7: Return ř T t ' 1 c t

It is well-known that optimal solution for objective in Equation 1 is given by π β ‹ p y | x q ' π ref p y 1 | x q Z p x q exp ´ ´ c p x,y q β ¯ π ref p y | x q where Z p x q ' ř y 1 P Y exp ´ ´ c p x,y 1 q β ¯ [40]. We assume π β ‹ P Π for the given β and following RLHF literature [37, 40], we compete with this KL-regularized optimal policy. Formally, we define the sub-optimality ( SubOpt ) of a policy π and the regret ( Reg T ) of an agent during the online learning phase as:

where π t is the policy played by the agent in the t th episode. We also define D p π, π 1 q ' E x ' ρ ' } π p¨ | x q ´ π 1 p¨ | x q} TV ‰ as the expected total-variation distance between two policies π and π 1 . Our goal is to find an interactive learning algorithm that learns from the offline dataset D and minimizes the regret Reg T during the online learning phase with high probability.

3 Principled Learning from User-Edits

There are two stages of learning in Protocol 1 offline and online. We discuss learning in these two stages below where one has access to a moderately large dataset of user edits and a much more limited number of online learning episodes.

3.1 Offline Learning from User-Edits

Our learning protocol (Protocol 1) contains multiple feedback sources that are typically studied separately in the ML literature. We discuss learning algorithms for each feedback source below.

A. Learning from User Edit Supervision. User edits y 1 provide a reasonable sample of the desired behavior. Therefore, one can directly learn a policy ˆ π SUP by fine-tuning on the user-edits:

B. Learning from Preferences. It is possible that users only partially improve the agent's response y when editing it to y 1 and that the optimal response is still far from y 1 . In this case, fine-tuning on y 1 can lead to sub-optimal behavior. In contrast, one can use the observation that as users edited y to y 1 , this implies that y 1 is preferred over y and we can use this to perform learning from preferences. We can use any preference-learning approach with dataset tp x i , y i , y 1 i qu n i ' 1 where y 1 i ľ y i . For example, we can perform Direct Preference Optimization (DPO) [40] to learn policy ˆ π PREF:

We emphasize an important difference between the distribution over preferences in Equation 4 and the typical RLHF literature [17, 6, 37], where the preference pair p y, y 1 q is typically generated by independently sampling from the reference policy. In contrast, in our setting only one response is generated from the reference policy whereas the other is generated by user edits. This distributional change impacts both the theoretical analysis and empirical performance.

Algorithm 1 LateEnsemble of Policies using Upper Confidence Bound.

Require: Given a dataset D ' tp x i , y i , y qu of user edits. Also, given a policy class Π

1: Create a list of policies Ψ ' r ˆ π , ˆ π , ˆ π , ˆ π s by fine-tuning on D // Offline learning

1 i n i ' 1 . SUP PREF RL EF

2: Define total cost C p π q ' 0 and count N p π q ' 0 for each policy π P Ψ

3: for t ' 1 , 2 , 3 , ¨ ¨ ¨ , T do

4: World presents context x t

5: If t ď | Ψ | , then π t ' Ψ r t s else π t ' arg min π P Ψ ! C p π q N p π q ´ α b log p t q N p π q ) , / / / .

6: y t ' π t p¨ | x t q

7: User edits the response to y 1 t

8: Agent receives a cost for edits c t ' ∆ edit p y t , y 1 t q

9: Update cost and counts: C p π t q ' C p π t q ` c t ; N p π t q ' N p π t q ` 1 .

C. Learning from Cost. We observe a cost c i ' ∆ edit p y i , y 1 i q for every datapoint which is a sampled from our cost function c p x, y q ' E y 1 ' q p¨| x,y q r ∆ edit p y, y 1 qs . We can, therefore, use this to train a cost model which can be used to perform reinforcement learning. Formally, given a cost model family F we first learn a cost model via square-loss regression:

A direct empirical approach will be to use this cost function and perform RL on an empirical distribution over the contexts in D . However, p f need not be well-calibrated for all responses. Motivated by this consideration, we define a pessimistic cost function, ¯ f p x, y q ' max f P ˜ F f p x, y q where the set of functions ˜ F ' ! f P F s.t. ř n i ' 1 ´ f p x, y q ´ ˆ f p x, y q ¯ ď γ p F , δ q ) for γ p F , δ q ' O ´ log ´ | F | δ ¯¯ is a confidence set of cost functions that agree with the historical data. We then train the policy ˆ π RL to optimize this cost under KL-constraints on our input contexts.

While computing the ¯ f is computationally impractical in the general setting, we neverthless use this pessimistic RL algorithm in Equation 6 for our theoretical analysis.

Early-Ensembling of Losses. As we will demonstrate later, the aforementioned three approaches have different trade-offs. Therefore, it might be advantageous to learn a robust policy by optimizing jointly over their losses. We call such an approach an early ensembling . For example, it is quite common in RLHF literature to add the supervised learning loss to the preference learning loss [38]:

We later empirically investigate the form of early ensembling described in Equation 7.

3.2 Adaptation During the Online Learning Case

During the online learning phase, we encounter a small number of episodes where the main goal is to evaluate the agent. In practice, this will be the deployment phase where the agent interacts with real users. However, this also provides an opportunity to perform any additional adaptation.

A key challenge of offline approaches is that different methods have different trade-offs. Earlyensembling approaches may not be able to effectively balance between these trade-offs. This is especially true, if the user distribution at test time is different from train time, which can happen in practice. Unfortunately, the small number of online episodes makes it challenging to do any effective fine-tuning. Motivated by these two considerations, we employ a very simple approach of training a set of policies using the different offline learning methods. During the online learning phase, we then run a bandit algorithm to select which of the learned policy to use for generating the response. We use the user-edit cost as input to the bandit algorithm. We call this approach a

/

/

/

-

// Online learning

late-ensembling of policies . Algorithm 1 gives an example using the Upper Confidence Bound (UCB) approach [5, 31, 11], however, other bandit approaches can also be used.

Pure Online Learning Setting. In Appendix A, we consider a pure online learning variant of Protocol 1 with a large T and without an offline learning phase, and derive a no-regret algorithm for it.

4 Theoretical Analysis of Learning from Edits

In the last decade, significant theoretical understanding has been achieved in reinforcement learning where the feedback is reward, and imitation learning where the feedback is action. Can we achieve the same for learning from edits? We initiate the theoretical understanding of learning from edits to provide a motivation for our algorithm design and predict experimental phenomena.

4.1 Theoretical Setup and Assumptions.

We start by stating our modeling assumption for the user distribution in Assumption 1. We assume that the user is more likely to edit a response y to y 1 instead of the reverse depending upon how much more likely is y 1 under the optimal policy π ‹ instead of y . This is stated formally in Equation 8 which we call the balance equation . This equation can be viewed as serving a similar purpose as the Bradley-Terry distribution in RLHF [10, 17] by providing a smooth characterization of user behavior. We also assume that the user distribution has a non-zero probability of generating the optimal response for any input response y . This probability can be small but does not need to scale with the size of the generation space | Y | .

Assumption 1 (User Distribution) . There exists a (known) β ą 0 such that the user distribution satisfies the balance equation below:

Further, for any x P X , the user has at least an γ min p x q ą 0 (possibly user dependent) probability of generating the optimal response y ‹ ' arg max y P Y π β ‹ p y | x q :

In Appendix A.1, we provide an example where this assumption holds.

From Balance Equation to Bradley-Terry Distribution. An important consequence of Assumption 1 is that the preference distribution in Equation 4 satisfies the Bradley-Terry assumption. To see this, we first realize that our preference pairs will contain two responses p y, y 1 q given a context x in precisely two situations: either we generate y ' π ref p¨ | x q and it is edited to y 1 ' q p¨ | x, y q , or we generate y 1 ' π ref p¨ | x q and it is edited to y ' q p¨ | x, y q . The probability that we prefer y 1 over y ( y 1 ľ y ) is then given by the probability of the first situation normalized by the joint probability, i.e.,

This justifies using DPOs even though the joint distribution over the preference pairs p y, y 1 q is different than IID sampling from π ref which is typically studied in the RLHF literature.

As we are working with function approximations, we assume policy realizability, i.e., that our function class is expressive enough to contain the desired policies. This is a standard assumption in theoretical analysis of interactive learning algorithms [28, 54].

Assumption 2 (Realizability) . We assume that π β ‹ P Π and q ˝ π P Π where q ˝ π is the policy defined as q ˝ π p y | x q ' ř y 1 P Y q p y | x, y 1 q π p y 1 | x q .

Our first result establishes Assumption 1 implies the edits induce a contraction property on the user distribution,

Lemma 1. For any π P Π , the user distribution satisfies the following contraction property:

The proof of Lemma 1 can be found in A. The contraction property implied by Lemma 1 indicates that an iterative application of the user distribution approaches a fixed point.

Finally, we make one final assumption for stating results for DPO. We assume that the data distribution satisfies the following concentrability assumption. Similar to realizability assumptions, concentrability assumptions are typical in RL literature (see for example [28, 54]).

Assumption 3. [Preference Concentrability] For any π P Π we assume:

This assumption states that our data distribution has enough coverage to allow us to evaluate different policies accurately. In practice, we can expect C PREF to be small when Π is a small constrained class around π ref . Alternatively, we can consider pessimistic-versions of preference learning methods [27] which avoids paying for covering all policies.

4.2 Theoretical Results.

We state the results for our different algorithms below and defer their full proof and analysis to Appendix A. Our first result is a suboptimality bound for the SFT algorithm.

Theorem 1 (SFT Result) . Let ϵ ą 0 be a sub-optimality target. Under Assumption 1 and Assumption 2 to achieve SubOpt p ˆ π SUP q ď ϵ ` O ´ min ´ η max ¨ D p π ref , π β ‹ q , ¯ η max ¨ D 1 { 2 p π ref , π β ‹ q ¯¯ with ´ ¯

We state the suboptimality bound for the offline DPO algorithm that treats the edit data set D ' tp x i , y i , y 1 i qu n i ' 1 as a preference data set in Theorem 2.

Theorem 2 (DPO Result) . Let ϵ ą 0 be a sub-optimality target. Under Assumption 1, Assumption 2, and Assumption 3 to achieve SubOpt p ˆ π PREF q ď ϵ with probability at least 1 ´ δ it is sufficient to set n ě Ω ´ C PREF ¨ log p| Π |{ δ q β 2 p σ 1 p´ V max qq 2 ϵ 2 ¯ where V max ą 0 satisfies max π P Π max p x,y qP X ˆ Y β log π p y 1 | x q π ref p y 1 | x q ď V max 2 .

The proof of Theorem 2 is inspired by the techniques of works such as [44, 54]. Our final result characterizes the sample complexity of learning from costs.

Theorem 3 (RL Result) . Let ϵ ą 0 be a sub-optimality target. Under Assumption 2 the policy ˆ π RL satisfies SubOpt p ˆ π RL q ď ϵ with probability at least 1 ´ δ when n ě Ω ´ log p| Π |{ δ q ϵ 2 ` ¯ C 2 ‹ ¨ d Elud p F q log p| F |{ δ q ϵ 2 ¯ where ¯ C 2 ‹ ' E π ref ' ´ π β ‹ p y | x q π ref p y | x q ¯ 2 ȷ is the policy concentra- bility coefficient of π ref and d Elud p F q is the Eluder dimension -a measure of statistical complexityof the cost function class F (see [45]).

Discussion of Trade-Offs. Theorems 1, 2, and 3 provide some guidance on the cases where using the edit data as a preference dataset is better or worse than doing imitation learning on the edits or learning the cost function and doing RL. When the preference coverage coefficient C PREF is small, the sample complexity of DPO can be much lower than that of SFT and RL. When the weighted approximation error min ! η max D p π ref , π β ‹ q , ¯ η max D 1 { 2 p π ref , π β ‹ q ) is smaller than the target error ϵ , SFT is the preferred method. Finally, when the approximation error and preference concentrability are large, but policy concentrability is small and the cost function is simple-for instance, a low-dimensional linear function-learning the cost function and then optimizing it is the most sample-efficient approach.

5 Experimental Results and Discussion

We empirically evaluate our theoretical findings in this section to see if they apply in complex settings similar to what is encountered in practice.

Task Setup. We evaluate on two tasks: email writing and summarization, from Gao et al., [21]. For each task, there exists a dataset of articles from 4 domains. Corresponding to each task and domain, there is a list of latent user preferences described as a preference string , for example, ' structured, straight to the points, respectful, professional greeting and closing '. The setup follows Protocol 1. In each round, the agent is given a context describing a task along with a given article. For example, summarize a given article. The agent does not have access to the latent preference string but must generate a response that satisfies this preference. An LLM user generates an edit given the response, context, and latent preference string corresponding to the domain of the article in the context. The use an LLM-based user allows for reproducible experiments which facilitates rapid advancement in algorithm development. 2 Finally, we compute edit distance using Levenshtein distance normalized by the number of tokens in the agent response. We use the NLTK word tokenizer to compute the edit distance. We use Qwen 3 32B instruct model as our user but remove think tokens before using the response.

The key challenge of this task is that latent preferences are context-dependent as articles from different domains have different preferences. Further, even for a single domain, there are multiple preferences in a preference string. For example, the aforementioned preference string contains multiple individual preferences such as ' second person narrative ' and ' show emotions '. These two challenges occur in real-world applications. For example, a person may prefer to write informal emails to their friends but write formal reviews for their office reports. Note that the context does not state which domain the article in the given context comes from. An LLM agent must implicitly infer the domain from the context, learn the appropriate preference for it, and then use it to generate an appropriate response.

Strong and Weak User. We extend the original setup of Gao et al., [21] to consider two types of users: strong users and weak users. A strong user generates edits based on every individual preference in the article's latent preference string. In contrast, a weak user samples a subset of preferences in the article's preference string and uses them to generate the edits. For example, in any given interaction a weak user may sample two preferences ' second person narrative ' and ' brief ' and perform edits to satisfy these while ignoring the other preferences in the preference string. The weak user models user who may only prefer to perform small edits at a time. Conceptually, both the weak and strong users have the same optimal behavior. This is because the optimal response that satisfies all the preference for the strong user, also satisfies any subset of preference that the weak user can sample. However, while strong user perform edits that rapidly converges to the optimal policy (higher γ min), the weak user's edit will slowly take the response towards the optimal behavior (smaller γ min).

Offline Dataset. We collect an offline interaction dataset between the agent model and the user. This dataset is supposed to represent deployment logs found in typical LLM agent applications for writing and coding assistants. We use Llama 3.1 8b Instruct model as our agent model. We perform generate responses by doing greedy decoding. We collect a dataset of 20,000 examples for the summarization task and 10,000 examples for the email writing task. We collect these datasets separately with both strong and weak users. This gives us 4 separate offline datasets.

Online Learning Phase. We evaluate the model for T ' 200 examples for summarization and email writing. We always use the strong user during test time regardless of which user was used to generate the offline dataset. We run each experiment with 3 different seeds.

Methods and Implementation. We consider the following approaches: (i) Base which generates from the base agent model, (ii) SFT which performs SFT on the training data, (iii) DPO which runs the DPO algorithm on the training data, (iv) EarlyEnsemble which performs early ensembling of DPO and SFT losses (Equation 7), and (v) LateEnsemble which performs late ensembling (Algorithm 1) using policies trained by methods (ii)-(iv). All generations are performed greedily and with a max number of generation tokens of 1000. We do not evaluate Equation 6 give challenges in implementing it.

We sweep over various hyperparameters including learning rate and epochs for SFT and DPO , as well as β for DPO , and β, λ for EarlyEnsemble . We pick the best hyperparameters by maximizing log-loss on a held-out validation set of 200 user-edits. We found in early studies that despite sweeping over the hyperparameters, both DPO and EarlyEnsemble learn policies that tend to repeat text

until their max tokens expire. We, therefore, perform a post-generation trimming strategy where we trim the generations once it starts repeating 200 consecutive characters. For a fair comparison, we apply this post-processing to the output of all methods. An alternative approach could be to use explicit length penalty [39] or more stable preference-learning approaches such as Rebel [22]. See Appendix C for full details on experimental setting.

Main Results. We report the main results in Table 1 with the Qwen-3 32B user. We report mean edit cost across rounds ( T ) and seeds. We also report Max. SubOpt that measures worst-case performance gap across the four settings between a given approach and the best performance. We see that Base method does not perform well showing the need for adaptation. Intuitively, we would expect a strong user to have strong convergence to the optimal policy. In our theory, this would be reflected in the value of γ min in Assumption 1. We see this in our results where SFT is able to learn and performs much better when trained and tested on strong user. Its performance when trained on weak user is weaker consistent with Theorem 1. In contrast, DPO 's theoretical analysis does not depend on the choice of γ min and instead only depends on coverage and the validity of balance equation. We see that DPO performs better than SFT when we train on weak user and test on strong user. However, when both users are strong, then there is no clear winner between SFT and DPO . Overall, this illustrates the trade-off predicted by our theory where SFT can take advantage whenever the user-edits converges faster to the optimal policy, however, it is also susceptible when this is not the case whereas DPO is more robust to this factor.

We see that EarlyEnsemble can exploit these trade-offs to achieve better performance than both SFT and DPO on the summarization domain, however, it under-performs DPO on the email writing domain. One explanation can be sensitivity to the choice of λ . While we perform a grid search over this hyperparameter, and use a validation loss over user-edits, it is possible that this selection scheme does not correspond well with test-time performance. Finally, our LateEnsemble approach is able to more readily adapt at test time by making suitable trade-offs and achieve the lowest worst case SubOpt value. We emphasize that late ensembling isn't guaranteed to do better in practice. Specifically, if one of the policy in Ψ on line 1 in Algorithm 1, is always better than others, then UCB will achieve lower performance than this policy due to exploring other sub-optimal policies.

Transfer Learning to Different User LLM. We investigate how robust these methods are to a different user LLM. This mimics a common real-world setting where the training data may come from one set of users, however, testing maybe done with a different set of users. These two sets of users can still have a common preference because these maybe dependent on organization, platform, or other controlled features such as age, or location. We use Llama-3.3-70B-Instruct model as our user model during the online testing phase while using our previously trained models. Table 2 shows the results. We see that the main trends from Table 1 carry over even when we test with a different user LLM. In particular, our ensemble approach LateEnsemble achieves the best SubOpt value.

Type

Success

(Latent User Preference String) Agent Response

( question answering style! ) **Q: What was the state of the land and people after the calamity?** A: The land was exhausted, the towns were stripped of their fighting population, and only a shadow of a kingdom remained. ¨ ¨ ¨ [truncated]

( bullet points, parallel structure, brief ) **Updated Mn PBPK model**: Reflects rapid Mn association/dissociation in tissues, with saturation of binding sites limiting accumulation.

**Key changes**:

Rapid Mn binding and dissociation rates.

High-affinity binding with rapid exchange between bound and free forms.

Tissue-specific binding capacities to account for background Mn levels.

**Model performance**: Successfully describes Mn kinetics in adult rats exposed to ¨ ¨ ¨ [truncated]

Failure ( inquisitive, skillful foreshadowing, with emojis ) **Q: What is the traditional approach to drug development, and why is it no longer effective?**

A: The traditional approach focuses on a single target protein, disease, and drug. However, this paradigm is now considered untenable due to the promiscuity of drug-like compounds, which can interact with multiple target proteins.

**Q: How do current target-fishing approaches address the promiscuity issue?**

A: Current approaches, such as single-label multi-classification schemes, ¨ ¨ ¨ [truncated] ( question answering style ) The article compares two ministers, Mr. Mearns and Rev. Mr. Tindall, highlighting their differences in style and delivery. Mr. Mearns is described as steady, patient, and well-spoken, ¨ ¨ ¨ [truncated]

0

25

50

75

100

125

150

175

200

Number of Test Rounds

0

50

100

150

200

Cummulative User Edits

Base

SFT

DPO

EarlyEnsemble

LateEnsemble

0

25

50

75

100

125

150

175

200

Number of Test Rounds

0

50

100

150

200

Cummulative User Edits

Base

SFT

DPO

EarlyEnsemble

LateEnsemble

(a) Summarization Domain with Strong User

(b) Summarization Domain with Weak User

Figure 2: Cumulative User Edits at test time corresponding to Table 1 for the different setup. For each round, we show mean and standard deviation across 3 seeds.

Sampling Generations and Test-time plot . We provide a few sample generations showing some success and failure cases in Table 3. We also visualize the test-time performance of different methods on the summarization task in Figure 2. As we can see, the LateEnsemble approach converges to the best method in both cases as expected. We provide additional details in Appendix C.

6 Conclusion and Limitations.

User edits provide a natural feedback for adapting LLMs. In this work, we initiated the theoretical investigation into learning from user edits. We propose a learning setup (Protocol 1) and analyze different algorithms in this setting. We show that none of these algorithms is always desired and propose a simple ensembling approach that empirically works best in two domains. Future works can investigate limitations of our work such as relaxing different assumptions, deriving optimal algorithms, and evaluating these algorithms in a human study.

References

A. Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , pages 1638-1646. PMLR, 2014.

A. Agarwal, S. Kakade, A. Krishnamurthy, and W. Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems , 33:20095-20107, 2020.

S. Agrawal and M. Carpuat. An imitation learning curriculum for text editing with nonautoregressive models. ArXiv , abs/2203.09486, 2022.

S. Aroca-Ouellette, N. Mackraz, B.-J. Theobald, and K. Metcalf. Predict: Preference reasoning by evaluating decomposed preferences inferred from candidate trajectories. ArXiv , 2024.

P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002.

Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

V. Balachandran, H. Hajishirzi, W. Cohen, and Y. Tsvetkov. Correcting diverse factual errors in abstractive summarization via post-editing and language model infilling. ArXiv , abs/2210.12378, 2022.

A. Beygelzimer, J. Langford, L. Li, L. Reyzin, and R. Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 19-26. JMLR Workshop and Conference Proceedings, 2011.

I. Bouzenia, P. Devanbu, and M. Pradel. Repairagent: An autonomous, llm-based agent for program repair. ArXiv , 2024.

R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

S. Bubeck, N. Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1-122, 2012.

M. Cao, Y. Dong, J. Wu, and J. C. K. Cheung. Factual error correction for abstractive summarization models. ArXiv , abs/2010.08712, 2020.

S. Cen, J. Mei, K. Goshvadi, H. Dai, T. Yang, S. Yang, D. Schuurmans, Y. Chi, and B. Dai. Value-incentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405.19320 , 2024.

J. Chen and N. Jiang. Information-theoretic considerations in batch reinforcement learning. In International conference on machine learning , pages 1042-1051. PMLR, 2019.

Z. Chen, M. O. Gul, Y. Chen, G. Geng, A. Wu, and Y. Artzi. Retrospective learning from interactions. arXiv preprint arXiv:2410.13852 , 2024.

Z. Chen, S. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshyvanyk, and M. Martin. Sequencer: Sequence-to-sequence learning for end-to-end program repair. IEEE Transactions on Software Engineering , 47:1943-1959, 2018.

P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

M. D'Arcy, A. Ross, E. Bransom, B. Kuehl, J. Bragg, T. Hope, and D. Downey. Aries: A corpus of scientific paper edits made in response to peer reviews. ArXiv , abs/2306.12587, 2023.

F. Faltings, M. Galley, K. Brantley, B. Peng, W. Cai, Y . Zhang, J. Gao, and B. Dolan. Interactive text generation. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

D. A. Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

G. Gao, A. Taymanov, E. Salinas, P. Mineiro, and D. Misra. Aligning llm agents by learning latent preference from user edits. NeurIPS , 2024.

Z. Gao, J. Chang, W. Zhan, O. Oertell, G. Swamy, K. Brantley, T. Joachims, D. Bagnell, J. D. Lee, and W. Sun. Rebel: Reinforcement learning via regressing relative rewards. Advances in Neural Information Processing Systems , 37:52354-52400, 2024.

S. A. Geer. Empirical Processes in M-estimation , volume 6. Cambridge university press, 2000.

K. Guu, T. B. Hashimoto, Y. Oren, and P. Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics , 6:437-450, 2017.

J. Hong, J. Lin, A. Dragan, and S. Levine. Interactive dialogue agents via reinforcement learning on hindsight regenerations. arXiv preprint arXiv:2411.05194 , 2024.

A. Huang, A. Block, D. J. Foster, D. Rohatgi, C. Zhang, M. Simchowitz, J. T. Ash, and A. Krishnamurthy. Self-improvement in language models: The sharpening mechanism. arXiv preprint arXiv:2412.01951 , 2024.

A. Huang, W. Zhan, T. Xie, J. D. Lee, W. Sun, A. Krishnamurthy, and D. J. Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization. In The Thirteenth International Conference on Learning Representations .

A. Huang, W. Zhan, T. Xie, J. D. Lee, W. Sun, A. Krishnamurthy, and D. J. Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization. arXiv preprint arXiv:2407.13399 , 2024.

A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR) , 50(2):1-35, 2017.

C. Jiang, W. Xu, and S. Stevens. arxivedits: Understanding the human revision process in scientific writing. ArXiv , 2022.

T. L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Q. Liu, N. Chen, T. Sakai, and X.-M. Wu. A first look at llm-powered generative news recommendation. ArXiv , 2023.

Y. Liu, B. Deb, M. Teruel, A. L. Halfaker, D. R. Radev, and A. H. Awadallah. On improving summarization factual consistency from natural language feedback. Annual Meeting of the Association for Computational Linguistics , 2022.

J. Mallinson, A. Severyn, E. Malmi, and G. Garrido. Felix: Flexible text editing through tagging and insertion. ArXiv , abs/2003.10687, 2020.

E. Marrese-Taylor, M. Reid, and Y. Matsuo. Variational inference for learning representations of natural language edits. ArXiv , abs/2004.09143, 2020.

M. Mita, K. Sakaguchi, M. Hagiwara, T. Mizumoto, J. Suzuki, and K. Inui. Towards automated document revision: Grammatical error correction, fluency edits, and beyond. In Workshop on Innovative Use of NLP for Building Educational Applications , 2022.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

R. Y. Pang, W. Yuan, H. He, K. Cho, S. Sukhbaatar, and J. Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems , 37:116617-116637, 2024.

R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024 , pages 4998-5017, 2024.

R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

M. Reid, V. J. Hellendoorn, and G. Neubig. Diffuser: Diffusion via edit-based reconstruction. International Conference on Learning Representations , 2023.

M. Reid and V. Zhong. Lewis: Levenshtein editing for unsupervised text style transfer. In Findings , 2021.

S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

C. Rosset, C.-A. Cheng, A. Mitra, M. Santacroce, A. Awadallah, and T. Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. CoRR , 2024.

D. Russo and B. Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

A. Saha, A. Pacchiano, and J. Lee. Dueling rl: Reinforcement learning with trajectory preferences. In International conference on artificial intelligence and statistics , pages 6263-6289. PMLR, 2023.

A. Salemi, S. Mysore, M. Bendersky, and H. Zamani. Lamp: When large language models meet personalization. ArXiv , 2023.

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

D. Sobania, M. Briesch, C. Hanna, and J. Petke. An analysis of the automatic bug fixing performance of chatgpt. 2023 IEEE/ACM International Workshop on Automated Program Repair (APR) , 2023.

F. Stahlberg and S. Kumar. Seq2edits: Sequence transduction using span-level edit operations. Conference on Empirical Methods in Natural Language Processing , 2020.

R. S. Sutton, A. G. Barto, et al. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998.

Z. Tan and M. Jiang. User modeling in the era of large language models: Current research and future directions. ArXiv , 2023.

Y. Wang, Q. Liu, and C. Jin. Is rlhf more difficult than standard rl? a theoretical perspective. Advances in Neural Information Processing Systems , 36:76006-76032, 2023.

T. Xie, D. J. Foster, A. Krishnamurthy, C. Rosset, A. Awadallah, and A. Rakhlin. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf. arXiv preprint arXiv:2405.21046 , 2024.

Y. Xu, R. Wang, L. Yang, A. Singh, and A. Dubrawski. Preference-based reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems , 33:18784-18794, 2020.

T. Yang, J. Mei, H. Dai, Z. Wen, S. Cen, D. Schuurmans, Y. Chi, and B. Dai. Faster wind: Accelerating iterative best-ofn distillation for llm alignment. arXiv preprint arXiv:2410.20727 , 2024.

P. Yin, G. Neubig, M. Allamanis, M. Brockschmidt, and A. L. Gaunt. Learning to represent edits. ArXiv , abs/1810.13337, 2018.

W. Zhan, M. Uehara, N. Kallus, J. D. Lee, and W. Sun. Provable offline preference-based reinforcement learning. arXiv preprint arXiv:2305.14816 , 2023.

J. Zhang, S. Panthaplackel, P. Nie, J. J. Li, and M. Gligori´ c. Coditt5: Pretraining for source code and natural language editing. Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , 2022.

Q. Zhang, C. Fang, Y. Ma, W. Sun, and Z. Chen. A survey of learning-based automated program repair. ACM Transactions on Software Engineering and Methodology , 2023.

B. Zhu, M. Jordan, and J. Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning , pages 43037-43067. PMLR, 2023.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: NA

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a brief list of limitations at the end of the main paper in Section 5. We also provide a more detailed discussion in a limitation section Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are listed in the main body, and proof along with explanations are provided in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We used an existing public benchmark [21]. We used open-weight LLMs from HuggingFace and we provide details of our experimental setup in Section 5 as well as a deeper discussion in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We are not releasing the code. We adapted an existing benchmark of [21] for learning from user edits that is available at https://github.com/gao-g/prelude , and used four publicly available datasets.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide information of data splits, hyperparameters, and how they were chosen in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes. We run each experment with 3 seeds and provide standard deviation in tables (e.g., see Table 1 and Table 2).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide details of compute and time of execution in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work does not create a new dataset, or releases a model to the public, nor does it include any human study. The main purpose of our paper is a theoretical investigation into how to develop principled algorithms for learning from user edits. We provide experiments using publicly available benchmark and datasets.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We list the broader impacts of our paper here as suggested in NeurIPS Ethics guidelines. Our work focuses on the foundational aspect of learning from user edits. Even though our work doesn't involve human studies or releases any models, the ideas from this paper may influence actual uses cases. Since our learning setup has a human-in-the-loop performing the user edits, we need to be wary when releasing a product that learns from user edits. To begin with, we need to ensure that we have the user's permission to learn from their edits or to even store them. Various privacy and data laws may apply here that we need to be mindful of. Another issues is that if we are learning across multiple-users, we need to ensure that we don't leak sensitive information from one user to another. E.g., a user may edit an legal documentation to add some identifiable information and an LLM may inadvertently learn to edit docs to add this information and show this to other users. Even showing this to the same user at a wrong time can be undesirable, e.g., when they are presenting in front of others. Filtering data to remove PII and having guardrails can mitigate this to an extent. Another option is to have users report and be able to delete user-edits whenever they want.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any new data or models with this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite every asset that we use and provide their details in Appendix C.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We are marking it as non-applicable since we do not create any new data or model in this paper and do not release the code with the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We did not do any human study or involve any human participant in this paper. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human study was performed in this paper and no human subjects were involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: Our paper describes ways to fine-tune LLM using user-edits. We used a previously released benchmark that includes an LLM user. We trained our LLMs using this simulated LLM user. We did include samples of generations in Appendix C that are generated by LLM. However, we did not use an LLM to generate the main ideas or proofs in this paper.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

Table of Contents

A Theoretical Analysis with Edit Distance

We provide a table of notations for convenience in Table 4.

A.1 Properties of the Balance Equation

Our setup consists of a distribution ρ P ∆ p X q over context x , a user edit distribution q : X ˆ Y Ñ ∆ p Y q , and an edit distance ∆ edit : Y ˆ Y Ñr 0 , c max s . We make two assumptions on our setup:

and that for every x P X , there exists y ‹ such that for every y P Y , we have q p y ‹ | x, y q ě γ min ' min x γ min p x q . For simplicity we consider a constant γ min function in this section. We now consider examples of this setup.

Example 1. We start with a singleton context set X ' t x u and Y ' t y 1 , y 2 , ¨ ¨ ¨ , y N u . We define the user edit distribution below:

We define edit distance as follows ∆ edit p y, y 1 q ' δ t y ‰ y 1 u for any y and y 1 . Using this, we get the expected cost as c p x, y q ' δ ´ δ 1 ´ γ min N for every y ‰ y N and c p x, y N q ' δ ´ δ ` γ min ` 1 ´ γ min N ˘ . Let ¯ c ' c p x, y N q then for every y ‰ y N we have c p x, y q ' ¯ c ` δγ min. Hence, the preferred response is y N . We also satisfy the constrained that we must have at least γ min going to the optimal response. That leaves only the balance equation. The optimal policy is given by:

For any y, y 1 ‰ y N we have q p y 1 | x, y q ' q p y | x, y 1 q ' 1 ´ γ min N . Further, we also have π β ‹ p y | x q ' π β ‹ p y 1 | x q ' 1 Z β p x q exp ´ ´ ¯ c ` δγ min β ¯ . Therefore, the balance equation is satisfied in this case. When both are y N then the balance equation is also satisfied. This leaves a single case where one of them is y N and other is not. Let the other be y .

We have:

and

Combining them we get:

It is straightforward to see that for any choice of N and γ min, we can define β to satisfy this last equation.

A.1.1 Contraction Property

Lemma 2 ( π ‹ as steady-state of user distribution) . For any x P X we have q ˝ π β ‹ p Y | x q ' π β ‹ p Y | x q .

Proof. Fix x P X , then for any y, y 1 P Y we have from the balance equation in Assumption 1 the following:

Summing over y on both sides gives:

This is equivalent to q ˝ π ‹ ' π ‹ .

Lemma 1. For any π P Π , the user distribution satisfies the following contraction property:

Proof. Fix x P X . We start by using Lemma 2 to express:

We define a quantity w p y 1 , y ; x q as follows:

Assumption 1 implies w p y 1 , y ; x q ě 0 and

Plugging it, we get:

We view w p y 1 , y ; x q as a | Y | ˆ | Y | matrix W with entries W y 1 y ' w p y 1 , y ; x q ě 0 satisfying ř y 1 W y 1 ,y ' 1 ´ γ min p x q . We omit x from the W notation as it is fixed. Similarly, we define | Y | -dimensional vectors π and π ‹ with entries π p y | x q and π β ‹ p y | x q . Using these matrix and vector notation we can express:

where the second step uses the non-negatitivy of w p y 1 y ; x q and triangle inequality and the third step uses Equation 12.

A.2 General Results

Recall that we define optimal policy as a KL-regularized policy:

β ą 0 is a chosen hyperparameter defining the objective. We can establish easily that π ‹ is given by:

This is a standard result but we prove it in the next Lemma for completion.

Lemma 3 (Optimal Policy) . The optimal policy satisfies π β ‹ p y | x q ' W ‹ p x q π ref p y | x q e ´ c p x,y q{ β for every x P X , y P Y , where W ‹ p x q ' ř y 1 P Y π ref p y 1 | x q e ´ c p x,y 1 q{ β is the normalization constant.

Proof. The Lagrangian for the problem is given by:

where t µ x | x P X u are our Lagrange multipliers. Taking derivative wrt to π p y | x q for a fixed value of x P X and y P Y and setting it to 0 we get:

Rearranging the terms we get:

let W ‹ p x q ' exp ´ ´ 1 ´ µ x βρ p x q ¯ , we get:

the value of W ‹ p x q serves as a normalization factor to ensure ř y P Y π p y | x q ' 1 and is enforced by taking derivative wrt to µ x and setting them to 0. This gives W ‹ p x q ' ř y 1 P Y π ref p y 1 | x q e ´ c p x,y 1 q{ β .

Recall that we define SubOpt p π q ' J β p π β ‹ q ´ J β p π q . In many practical setting, we want to compete with the regularized policy but we only care about the unregularized cost. We capture this by defining the unregularized sub-optimality :

We also define unregularized regret as

where π t is the policy played by the agent in round t . If we play a fixed policy π for all T rounds then we have Reg 0 T ' ř T t ' 1 SubOpt 0 p π q ' T SubOpt 0 p π q . In these cases, we will use Reg 0 T p π q to denote the regret of this fixed policy.

A useful lemma below relates the total variation between policies to the suboptimality.

Lemma 4 (TV to Unregularized Suboptimality) . If E x ' ρ '› › › π p Y | x q ´ π β ‹ p Y | x q › › › TV ı ď ϵ , then SubOpt 0 p π q ď 2 ϵc max and Reg 0 T p π q ď 2 ϵc max T .

Proof.

Finally, Reg 0 T p π q ' T SubOpt 0 p π q ď 2 ϵc max T

Bounding Regularized Suboptimality and Regret. One challenge in bounding regularized suboptimality SubOpt p π q ' J β p π q ´ J β p π β ‹ q and regularized regret Reg T ' ř T t ' 1 SubOpt p π t q is dealing with KL divergence. E.g., a policy π can satisfy › › › π p Y | x q ´ π β ‹ p Y | x q › › › TV ď ϵ for some x but still have D KL p π p Y | x q|| π ref p Y | x qq ' 8 if π p y | x q ' ϵ , π β ‹ p y | x q ' π ref p y | x q ' 0 .

However, in RLHF analysis it is often assumed that densities are bounded [44, 54] and this allows us to avoid these cases. In fact, we will later assume this in our proof of DPO. Specifically, we have Assumption 4 (Bounded Density Ratios) . We assume there exists V max ą 0 , such that:

Under this assumption we can bound the regularized suboptimality and regret.

Lemma 5 (TV to Regularized Suboptimality) . If E x ' ρ '› › › π p Y | x q ´ π β ‹ p Y | x q › › › TV ı ď ϵ , then under Assumption 4, we have SubOpt p π q ď 2 ϵ p c max ` V max q and Reg T p π q ď 2 ϵ p c max ` V max q T .

where the first inequality follows from Lemma 4 and the last inequality non-negativity of KL divergence:

Finally, we have:

where the third step uses triangle inequality and fourth step uses Assumption 4. Alternatively we can bound β E x ' ρ ' ř y P Y ˇ ˇ ˇ π p y | x q ´ π β ‹ p y | x q ˇ ˇ ˇ ˇ ˇ ˇ log π p y | x q π β ‹ p y | x q ˇ ˇ ˇ ı as follows,

The advantage of these results is that we can focus on bounding total variation of the learned policy with respect to π β ‹ and the result for sub-optimality and regret for either regularized or unregularized objective follows.

A.3 Leaning from Supervised Feedback

We recall that we are given a dataset of user-edits D ' tp x i , y i , y 1 i qu n i ' 1 where for every i P r n s , we have x i ' ρ p¨q , y i ' π ref p¨ | x i q , and y 1 i ' q p¨ | x i , y i q . The supervised fine-tuning ( SFT ) approach learns the following policy:

Theorem 4 (SFT Result Full) . Under Assumption 1, Assumption 2 and Assumption 4, we have that when n ě 4 p c max ` V max q 2 log p ln | Π |{ δ q ϵ 2 then with probability at least 1 ´ δ :

where η max ' 1 ´ min x γ min p x q and ¯ η max ' a E x ' ρ rp 1 ´ γ min p x qq 2 s for γ min p x q defined in Assumption 1 and

Proof. The Bayes classifier of this log-likelihood problem is given by q ˝ π ref : X Ñ ∆ p Y q where q ˝ π ref p y 1 | x q ' ř y P Y q p y 1 | y, x q π ref p y | x q . From Assumption 2 we have realizability and so using standard finite-sample guarantees for maximum likelihood (e.g., [23]) we have

with probability at least 1 ´ δ for any fixed δ P p 0 , 1 q . Using triangle inequality we have:

where the last line uses Lemma 1 and Equation 17.

We compute two distinct bounds for the last term and consider the tightest among them:

where inequality p i q holds because of Cauchy-Schwartz and p ii q because the TV distance lies in r 0 , 1 s . Moreover, the following inequality also holds:

Thus we conclude that,

We can use Lemma 4 or Lemma 5 to get bounds on sub-optimality and regret. In particular, using Lemma 5 we get:

Reg T p ˆ π SUP q ď 2 p c max ` V max q ´ ∆ mle p n, δ q ` min ´ η max ¨ D p π ref , π β ‹ q , ¯ η max ¨ D 1 { 2 p π ref , π β ‹ q ¯¯ T. where recall that η max ' 1 ´ min x γ min p x q and ¯ η max ' a E x ' ρ rp 1 ´ γ min p x qq 2 s . Finally, setting n ' 4 p c max ` V max q 2 log p ln | Π |{ δ q 2 we get the desired result.

A.4 Theoretical Analysis with Preference Feedback

Given the user edit data D ' tp x i , y i , y 1 i qu n i ' 1 , we first create a preference learning dataset D pref ' tp x i , ˜ y i , ˜ y 1 i , z i qu n i ' 1 as follows: for every i P r n s , we sample z i P Unf pt˘ 1 uq and if z i ' 1 then we define ˜ y i ' y i and ˜ y 1 i ' y 1 i , otherwise, we define ˜ y i ' y 1 i and ˜ y 1 i ' y i .

Given the LLM policy class Π , we induce a binary classifier class C ' t f π : p x, ˜ y, ˜ y 1 q Ñ ∆ pt˘ 1 uq | @ π P Π u as:

The size of C is the same as the size of the policy class Π .

We define the empirical log-likelihood ℓ PREF p π q for a policy π P Π as:

Using the construction of D pref from D we can express ℓ PREF p π q as:

which is the standard DPO loss without β factor. Our theoretical analysis uses this variant of DPO without β since in our analysis β is a given value with respect to which Assumption 1 holds. In our experiments, we use the standard version of DPO with the β term. Given the model class C , we maximize the log-likelihood:

Each datapoint p x i , ˜ y i , ˜ y 1 i , z i q P D pref is sampled IID from a distribution and let this be denoted by D pref p x, ˜ y, ˜ y 1 , z q . From the construction of D pref we have:

Here we show in detail our calculations from Section 4 of how the distribution D pref p z ' 1 | x, ˜ y, ˜ y 1 q satisfies the Bradley-Terry Assumption. As we are using a variant of DPO without β , we will instead use a variant of Bradley-Terry Distribution as well which has a β temperature.

Lemma 6 (Satisfying Bradley-Terry Distribution) . Under Assumption 1, the probability of preferring ˜ y 1 over ˜ y , i.e., ˜ y ĺ ˜ y 1 , in our preference dataset D pref satisfies the Bradley-Terry assumption with respect to the cost function c . Formally, for every x P supp ρ, ˜ y, ˜ y 1 P Y we have:

where σ p t q ' 1 1 ` exp p´ t q is the sigmoid function.

Proof. The probability of the event ˜ y ĺ ˜ y 1 is given by D pref p z ' 1 | x, ˜ y, ˜ y 1 q which is also the Bayes' classifier of our classification problem. Starting from Bayes's theorem we get:

From Assumption 2 we have π β ‹ P Π which gives

This means that we have realizability for the classification problem in Equation 19. We can then use standard MLE guarantees to state our next result.

Lemma 7 (MLE Guarantee) . For any δ P p 0 , 1 q , under Assumption 2 we have:

where D pref p x, ˜ y, ˜ y 1 q is the marginal distribution of Equation 22 and given by D pref p x, ˜ y, ˜ y 1 q ' ρ p x q 2 p π ref p ˜ y | x q q p ˜ y 1 | x, ˜ y q ` π ref p ˜ y 1 | x q q p ˜ y | x, ˜ y 1 qq .

Proof. This is a standard MLE bound under realizability. Proof follows from using Theorem 21 of Agarwal et al. [2] and applying Chernoff's bound E r ? X 2 s ď a E r X 2 s .

Using } p ´ q } TV ' 1 2 } p ´ q } 1 for two finite value distributions p, q we get:

For convenience, we define a notion of implicit cost ˆ c p x, y q given by:

where W p x q ' ř y 1 P Y π ref p y 1 | x q exp p´ ˆ c p x, y 1 q{ β q is the normalization constant. As ˆ c Ñ c , we have ˆ π PREF Ñ π β ‹ . Rewriting Equation 25, we get:

Using Equations 24 and 27 and Lemma 6 we can simplify Equation 23 to:

To further simplify this result, we will need certain properties of the sigmoid function (Lemma 13 and Lemma 14). We will also assume Assumption 4.

Let Z 1 ' p ˆ c p x, ˜ y q´ ˆ c p x, ˜ y 1 qq { β and Z 2 ' p c p x, ˜ y q´ c p x, ˜ y 1 qq { β . We have | Z 1 | ď 2 V max { β from Assumption 4 and | Z 2 | ď c max since c p x, y q P r 0 , c max s . Then starting from Equation 28 we get:

This gives us:

Using Lemma 3 we have:

which gives

Plugging in values of c from Equation 30 and ˆ c from Equation 26 in Equation 29 we get:

which simplifies to

We now invoke the concentrability assumption (Assumption 3) which we restate below for convenience:

Assumption 3. [Preference Concentrability] For any π P Π we assume:

where Q π p x, ˜ y, ˜ y 1 q ' 1 2 ρ p x q p π p ˜ y 1 | x q π ‹ p ˜ y | x q ` π ‹ p ˜ y 1 | x q π p ˜ y | x qq .

to get:

where we used the fact that D pref ' Q π ref . Concentrability style assumptions are common in analysis of RL algorithms [14, 44]. Our assumption is similar to the one in Theorem 1 of Rosset et al., [44]. the main difference is that our assumption is designed for learning from user-edits while that of Rosset et al., is designed for iterative learning with preference feedback.

Simplifying Equation 31 gives:

We lower bound the left hand side of this assumption:

Combining this all we get:

We can wrap up the proof with the application of Lemma 5.

Theorem 5 (DPO Result) . Under Assumption 1, Assumption 2, and Assumption 3, Assumption 4 we have that when n ě 8 C 2 PREF p c max ` V max q 2 ln | Π | δ min t σ 1 p 2 V max { β q ,σ 1 p´ c max qu 2 ϵ 2 then with probability at least 1 ´ δ :

Proof. Weset n such that ϵ 2 p c max ` V max q ' C PREF min t σ 1 p 2 V max { β q ,σ 1 p c max qu b 2 n ln | Π | δ which together with Equation 33 and Lemma 5 gives us SubOpt p ˆ π PREF q ď ϵ and Reg T p ˆ π PREF q ď ϵT .

We can simplify Theorem 4 and Theorem 5 if we define V max to be the max of the quantity defined in Assumption 4 and c max.

A.5 Learning from Cost

Throughout this section, we will assume that the cost functions are bounded.

Assumption 5. The cost functions f P F satisfy | f p x, y q| ď c max for some c max ą 0 for all x P X and y P Y .

Provided the true cost function satisfies c P F where F is a class of functions from the space of pairs X ˆ Y to the real numbers we can estimate the mean cost function by least squares regression on the observed costs gathered by π ref .

The estimator ˆ f satisfies,

Lemma 8. Let and δ P p 0 , 1 q and tp x i , y i , c i qu n i ' 1 be a dataset of prompt answers and costs. If Assumption 5 holds then

and defining the confidence set of plausible cost functions as

for a universal constant b ě 1 . We have c P ˆ F and E x ' ρ,y ' π ref p¨| x q ' p f p x, y q ´ c p x, y qq 2 ‰ ď b ¨ c 2 max log ´ | F | δ ¯ for all f P ˆ F with probability at least 1 ´ δ { 2 .

Throughout this section we will define E as the at least 1 ´ δ { 2 probability event defined in Lemma 8. Given cost function f we introduce notation to define the policy optimizing the regularized objective where we add f to the notation.

and satisfies π f,β ‹ p y | x q ' exp p ´ f p x,y q β q π ref p y | x q Z p x q where Z p x q ' ř y 1 exp ´ ´ f p x,y 1 q β ¯ ¨ π ref p y 1 | x q .

Pesimism As we have mentioned in the main, we consider the following pessimistic estimator of the cost function,

where ˆ F is defined as in equation 35. It follows that ˆ π RL as defined in Equation 6 satisfies ˆ π RL ' π ¯ f,β ‹ . Moreover, Assumption 5 implies,

A.5.1 Regularized Cost Analysis

Proposition 6. For any x P support p ρ q ,

Proof. Equality p b q follows by substitution. We now proceed to prove the inequalities p a q and p c q . Now observe that Z p x q ' E y ' π ref p¨| x q r exp p´ f p x, y q{ β qs . Notice that g p γ q ' β ¨ log p 1 { γ q is a convex function. Thus, Jensen's inequality implies,

where inequality p i q follows by Jensen and p ii q by Assumption 5. To derive a lower bound we need to upper bound E y ' π ref p¨| x q r exp p´ f p x, y q{ β qs . Assumption 5 implies,

Thus,

This concludes the proof.

Proposition 6 implies that

For the next few results we condition on the event E . Because the function ¯ f is not in the class of cost functions our results will depend on the Eluder dimension of F . Pessimism allows us to prove bounds depending on the concentrability coefficient of the π ref sampling distribution. Assuming access to a new prompt dataset t ¯ x i u n i ' 1 we analyze the following algorithm,

where

Equation 37 implies π ¯ f,β ‹ P Π p ¯ f q .

In this section we'll write the sample complexity of our algorithms in terms of the Eluder dimension of F . This is a measure of statistical complexity of a function class first introduced by [45] and is based on the notion of ϵ -independence defined as,

Definition 1. ( ϵ ´ dependence) Let G be a scalar function class with domain Z and ϵ ą 0 . An element z P Z is ϵ ´ dependent on t z 1 , ¨ ¨ ¨ , z n u Ď Z w.r.t. G if any pair of functions g, g 1 P G satisfying ař n i ' 1 p g p z i q ´ g 1 p z i qq 2 ď ϵ also satisfies g p z q ´ g 1 p z q ď ϵ . Furthermore, z P Z is ϵ ´ independent of t z 1 , ¨ ¨ ¨ , z n u w.r.t. G if it is not ϵ ´ dependent on t z 1 , ¨ ¨ ¨ , z n u .

The eluder dimension is defined as,

Definition 2. ( ϵ -eluder) Let G be a function class with support in a set Z and values in R . The ϵ ´ non monotone eluder dimension Ğ d eluder p G , ϵ q of G is the length of the longest sequence of elements in Z such that every element is ϵ ´ independent of its predecessors. Moreover, we define the ϵ ´ eluder dimension d eluder p G , ϵ q as d eluder p G , ϵ q ' max ϵ 1 ě ϵ Ğ d eluder p G , ϵ q .

The following eluder dimension lemmas will also prove useful in proving our sample complexity bounds. The first one is a restatement of Proposition 3 from [45].

Lemma 9. Let t x 1 ℓ , y 1 ℓ u n ℓ ' 1 be an arbitrary sequence in X ˆ Y . When E holds ,

The result above implies,

Lemma 10. Let t x 1 ℓ , y 1 ℓ u n ℓ ' 1 be an arbitrary sequence in X ˆ Y . When E holds ,

for the same unversal constant b ě 1 as in Lemma 8.

Proof. We use the same proof technique as in Lemma 2 from [45].Let w i ' p ¯ f p x 1 ℓ , y 1 ℓ q ´ c p x 1 ℓ , y 1 ℓ qq 2 . Let i 1 , ¨ ¨ ¨ , i n be the permutation of indices 1 to n such that w i 1 ě ¨ ¨ ¨ ě w i n .

Let τ ą 0 be a parameter to be specified later on. For t P r n s such that w i t ě τ notice that,

where p i q follows by Lemma 9 and p ii q follows because d eluder p F , ¨q is monotonically decreasing in the second argument. Inequality p iii q holds because w i 1 ď 4 c 2 max .

And therefore, for any t such that w i t ě τ we have,

and therefore, n ÿ ℓ ' 1 w 2 i ℓ ¨ 1 p w i ℓ ě τ q ď n ÿ ℓ ' 1 8 b ¨ c 2 max log ´ | F | δ ¯ ℓ ¨ d eluder p F , τ q ď 16 b ¨ c 2 max log ˆ | F | δ ˙ ¨ log p n q¨ d eluder p F , τ q .

And finally,

And therefore we have,

Finally, setting τ ' c max { n we get the desired result,

Additionally our results relies on the following variant of Bernstein inequality for martingales, or Freedman's inequality [20], as stated in e.g., [1, 8].

Lemma 11 (Simplified Freedman's inequality) . Let Z 1 , ..., Z T be a bounded martingale difference sequence with | Z ℓ | ď R . For any δ 1 P p 0 , 1 q , and η P p 0 , 1 { R q , with probability at least 1 ´ δ 1 ,

where E ℓ ´ 1 r¨s is the conditional expectation 3 induced by conditioning on Z 1 , ¨ ¨ ¨ , Z ℓ ´ 1 .

The following Lemma will prove useful in our results relating the in-distribution ( π ref ) error of the pessimistic cost estimators w.r.t. the true cost function c .

Lemma 12. The following inequality holds,

Proof. Let tp x 1 ℓ , y 1 ℓ qu n ℓ ' 1 a fresh set of samples from ρ ˆ π ref and consider the martingale difference sequence Z f ℓ ' p f p x 1 ℓ , y 1 ℓ q´ c p x 1 ℓ , y 1 ℓ qq 2 ´ E x ' ρ,y ' π ref p¨| x q ' p f p x, y q ´ c p x, y qq 2 ı indexed by f P ˆ F .

It follows that | Z f ℓ | ď 4 c 2 max for all ℓ P r n s and all f P ˆ F . Moreover,

Where the last inequality holds because,

Using Freedman's inequality from Lemma 11 we conclude when E holds,

simultaneously for all f P ˆ F with probability at least 1 ´ δ { 8 .

Where inequality p i q holds because of conditioning on E and Lemma 8 and p ii q follows by setting η ' 1 8 c 2 max .

Let E 1 be the event where E and 40 hold. In this case,

for all t P r 0 , ¨ ¨ ¨ , n s .

We consider a new martingale difference sequence t ˜ Z ℓ u n ℓ ' 1 defined as

The following bounds hold, | ˜ Z ℓ | ď 4 c 2 max for all ℓ P r n s and all f P ˆ F . Moreover,

where the last inequality has the same derivation as inequality 39. Freedman's inequality (Lemma 11) implies,

with probability at least 1 ´ δ { 8 . By setting η ' 1 16 c 2 max we conclude,

with probability at least 1 ´ δ { 8 .

Lemma 10 implies that when E holds,

Combining E and the conitions for equations 40 and 41, the desired result holds with probability at least 1 ´ δ { 2 ´ δ { 8 ´ δ { 8 .

Theorem 7. The policy ˆ π RL satisfies,

with probability at least 1 ´ δ where ¯ C ‹ ' d E π ref ' ´ π c,β RL p y | x q π ref p y | x q ¯ 2 ȷ is the optimal policy concentrability coefficient.

Proof. Throughout this proof we will condition on the event E . Since π ¯ f,β ‹ P Π p ¯ f q and by definition of ˆ π RL,

Since ´ c max ď E y ' π p¨| x q ' ¯ f p x, y q ` β log p π p y | x q{ π ref p y | ¯ x i qq ‰ ď c max for all x P X and all π P Π p ¯ c q (see Equation 37) Hoeffding inequality implies that,

holds with probability at least 1 ´ δ { 4 for all π P Π p ¯ c q simultaneously where J β p π, f, x q ' E y ' π p¨| x q r f p x, y q ` β log p π p y | x q{ π ref p y | x qqs .

Combining equations 42 and 43 we conclude that with probability at least 1 ´ δ { 4 ,

where inequality p i q holds because π ¯ f,β ‹ is achieves a better regularized objective value for cost ¯ f than π c,β ‹ .

Moreover, pessimism implies that when E holds,

Combining these results we conclude that,

The final step is to upper bound E x ' ρ,y ' π c,β ‹ p¨| x q ' ¯ f p x, y q ‰ in terms of E x ' ρ,y ' π c,β ‹ p¨| x q r c p x, y qs .

To do this we will again use E and Cauchy Schwartz's inequality, in the next few lines we'll use the shorthand E π r¨s to denote E x ' ρ,y ' π p¨| x q r¨s .

Finally we can bound the term E π ref ' ` ¯ f p x, y q ´ c p x, y q ˘ 2 ı . Lemma 12 implies,

with probability 1 ´ 3 δ { 4 . And therefore,

with probability at least 1 ´ δ .

A.6 Pure Online Learning Theory

In this section we introduce and analyze the regret of the Epoch Supervised Learning Algorithm (see Algorithm 2). This procedure works in epochs, in each epoch the algorithm observes prompts from ρ , and produces responses y from a fixed edits distribution. The algorithm also collects edit data in the form of a samples y 1 ' q p¨| x, y q . During epoch e the algorithm uses policy π e to produce responses and collect m e datapoints of the form D e ' t x e,n , y e,n , y 1 e,n u m e n ' 1 where y 1 e,n ' q p¨| x n,e , y n,e q .

The epoch supervised learning algorithm's policy is updated by fitting a policy matching the edited responses in D e ,

This way the policy π e ` 1 is an approximation to q ˝ π e . We consider a setting where Algorithm 2 interacts in t ' 1 , ¨ ¨ ¨ , T rounds where in each time step the algorithm is given a prompt x t and generates a response y t . We measure the performance of 2 through its regularized regret Reg T as defined in Equation 2. Our main result is the following theorem where we show Algorithm 2 satisfies a O p ? T q regret bound.

Algorithm 2 Epoch Supervised Learning

10:

1: Start arbitrary initial policy π 1 ' π ref . 2: for Epoch e ' 1 , 2 , ¨ ¨ ¨ , t e p T q u do 3: D e ' H 4: for n ' 1 , 2 , ¨ ¨ ¨ , m e do 5: Given context x e,n 6: Agent generates y e,n ' π e 7: User edits it to y 1 e,n 8: Agent collects edit distance of ∆ edit p y e,n , y 1 e,n q 9: D e Ð D Y ␣ p x e,n , y 1 e,n q ( Update the policy:

Theorem 8. When Assumption 1 and Assumption 4 hold then Algorithm 2 with inputs m e ' 2 ln p | Π | δe q p 1 ´ γ min q 2 e , δ e ' δ { 2 e 2 and e p T q be the first index such that ř e p T q´ 1 e ' 1 m e ă T and ř e p T q e ' 1 m e ě T satisfies the regret bound,

with probability at least 1 ´ δ .

Algorithm 2 works over e p T q epochs. In each epoch the algorithm interacts with the 'editor' for m e steps.

At the start of epoch e we assume access to a response generator π e p y | x q . This induces a population version of the target distribution p e ` 1 p x, y, y 1 q defined as p e ` 1 p x, y, y 1 q ' P X p x q π e p y | x q q p y 1 | x, y q . This induces the following ideal updated generator ¯ π e ` 1 p y | x q defined as,

Since we admit m e samples during epoch e , the MLE guarantee ensures that the computed generator π e ` 1 satisfies,

with probability at least 1 ´ δ e . Lemma 1 implies

therefore, combining Equations 46 and 47 we obtain,

we will now unroll this sum.

Then, unrolling the recursion we obtain,

δ e ` 1 ď p 1 ´ γ min q δ e ` ξ e ď η pp 1 ´ γ min q δ e ´ 1 ` ξ e ´ 1 q ` ξ e ' p 1 ´ γ min q 2 δ e ´ 1 `p 1 ´ γ min q ξ e ´ 1 ` ξ e ď ¨ ¨ ¨ so that δ e ` 1 ď ř e i ' 1 p 1 ´ γ min q e ´ i ξ i `p 1 ´ γ min q e δ 1 . Therefore,

Lemma 5 allows us to bound the regret of epoch e by,

Set m e ' 2 ln p | Π | δe q p 1 ´ γ min q 2 e so that ξ e ' p 1 ´ γ min q e . Then,

where the last equality follows because m e ' 2 ln p | Π | δe q p 1 ´ γ min q 2 e .

Setting δ e ' δ { 2 e 2 we conclude that

for all e P N simultaneously with probability at least 1 ´ δ .

To prove a regret bound up to time T let e p T q be the necessary epoch index such that ř e p T q´ 1 e ' 1 m e ă T ď ř e p T q e ' 1 m e . Since the size of m e is increasing it follows that,

the epoch parameter e p T q does not need to be greater than e p T q ď Q 2 log 1 {p 1 ´ γ min q ´ T { 2 ln ´ 2 T 2 | Π | δ ¯¯U since at this point,

And therefore

with probability at least 1 ´ δ . Finally since e p T q ď Q 2 log 1 {p 1 ´ γ min q ´ T { 2 ln ´ 2 T 2 | Π | δ ¯¯U the result follows.

A.7 Support Results

We state useful supporting results below that allow us to prove our main results.

Lemma 13 (Sigmoid Lower Bound) . For any x, y P R we have:

Proof. We have σ p t q ' 1 1 ` e ´ t for any t P R . This gives σ 1 p t q ' e ´ t p 1 ` e ´ t q 2 ' e ´ t σ p t q 2 . Observe that σ 1 p t q ě 0 for all t P R . We also have

Therefore, σ 2 p t q ě 0 (convexity condition) if and only if 1 ě e t . Therefore, σ is concave on r 0 , 8q and convex on p´8 , 0 s .

Fix x, y P R and assume x ă y without loss of generality. Note that the result is trivially true for x ' y . As σ is differentiable over R , we have from mean value theorem σ p y q´ σ p x q y ´ x ' σ 1 p u q for some u P r x, y s . As σ 1 p t q ě 0 for all t P R , we have | σ p y q ´ σ p x q| ' σ 1 p t q | y ´ x | .

We now consider 3 cases.

If x ď y ď 0 in which case we are in the convex regime and σ 1 p u q ě σ 1 p x q

If y ě x ě 0 in which case we are in the concave regime and σ 1 p u q ď σ 1 p y q .

If x ď 0 ď y , then if u ď 0 , then we have σ 1 p u q ě σ 1 p x q . And if u ě 0 , then we have σ 1 p u q ě σ 1 p y q .

Putting it together, we always get σ 1 p t q ě min t σ 1 p y q , σ 1 p x qu giving us the desired result.

Lemma 14 (Sigmoid Derivative Bound) . If | x | ď c then σ 1 p x q ě σ 1 p c q .

Proof. If x ě 0 , then ´ c ď 0 ď x ď c . As x is in the concave regime of σ we have σ 1 p x q ě σ 1 p c q . Alternatively, if x ď 0 , we have ´ c ď x ď 0 ď c , then x is in the convex regime of σ and we have σ 1 p x q ě σ 1 p´ c q . As, σ 1 p´ c q ' e c p 1 ` e c q 2 ' e ´ c p 1 ` e ´ c q 2 ' σ 1 p c q , we get σ 1 p x q ě σ 1 p c q in all cases.

Lemma 15 (Log Lower Bound) . For any t ą 0 , we have | log p t q| ě | t ´ 1 | t ` 1 .

B Related Work

Theory Although there is no theory prior work on edits that we are aware of, there is a rich literature studying theoretical guarantees for RLHF objectives from preference feedback. The works of [61] and [53] pioneered the study of reinforcement learning from human feedback via offline preference data, while works such as [46] and [55]. These preliminary results were mostly concerned with tabular scenarios. More recently, many works have been dedicated to the study of function approximation regimes for learning policies from preference feedback in RLHF scenarios, such as the introduction of the XPO algorithm for exploratory RLHF [54] and [58] for provable guarantees for RLHF from offline data as well as unifying frameworks that bridge online and offline preference feedback [13]. Beyond policy optimization, theoretical insights have also been developed for auxiliary phenomena in RLHF. For example, [26] characterized the self-improvement dynamics of iteratively refined policies, and [56] analyzed the statistical efficiency of Best-of-N mechanisms in preference alignment.

Modeling edits. Prior work on text edits has explored various modeling approaches, including generative diffusion models with edit-based corruption and reconstruction [41], latent vector representations of edits [24, 35], and modeling modular operations in the editing process [50, 34, 3, 59]. While these approaches effectively model specific aspects of text editing, our work unifies multiple forms of user feedback - preferences, supervised labels, and cost - within a single framework. These types of feedback, typically studied in isolation, are combined and analyzed both theoretically and empirically in this work.

Using edits. Research on text revision often focuses on task-specific improvements, such as enhancing model factuality [12, 33, 7], refining academic writings [36, 30, 18], and enabling style transfer [42]. Similarly, work on code edits has primarily aimed at automating program repair [57, 16, 41, 60, 49, 9]. In contrast, we focus on user-driven text edits that reflect expectations and preferences in practical use cases. Building on the setup proposed in prior work [ ? 4], we systematically investigate the algorithms and learning challenges associated with leveraging such edits.

C Additional Experimental Details

We provide details of our experimental setup here. We used the Prelude framework of Gao et al., [21] which is available at https://github.com/gao-g/prelude and has MIT License. We use the following models from HuggingFace: Llama 3.1 8b Instruct, Llama 3.3 70b Instruct and Qwen3-32B.

Grid Search. We provide hyperparameter values of SFT in Table 5 and for DPO and EarlyEnsemble in Table 6. We selected the best hyperparameters by computing log-loss of user-edits given context on a held-out validation set with 200 examples.

For LateEnsemble (Algorithm 1) we set α ' 150 based on the approximate average edit cost of the base model. We did not tune α .

Inference. We do greedy decoding and allow the agent to generate at most 1000 tokens.

Compute and Time. We ran our experiments on a cluster with H100s and H200s. All experiments took a few hours to train. The inference took 10min for each seed for SFT , DPO and EarlyEnsemble for each seed since we can evaluate each datapoint in parallel, while it took 30min for each seed for LateEnsemble (Algorithm 1) since we cannot evaluate datapoints in parallel for this.

Datasets. We do not release any new dataset with this paper. However, we used source articles from four datasets provided below. The link to the dataset contains other details such as size of the dataset, samples, and license. We urge readers to check the current status of these datasets in context of this work, and recommend them to avoid using any dataset that has been deprecated.

We provide additional experimental results below.

Test-time plots. Figure 3 shows the cumulative edit cost at test time for the email writing setting in Table 1. We also show cumulative user-edits at inference time for the transfer learning setting in Table 2 in Figure 4. The trend is similar to the trend in Figure 3 with LateEnsemble mostly converging to the performance of the best method. We can see that in all domain except email writing with a weak user, the performance of LateEnsemble gradually drifts towards the best performing method as we would expect. The anomalous setting of email writing with weak user may need more test rounds before LateEnsemble converges to the best method.

0

25

50

75

100

125

150

175

200

Number of Test Rounds

0

20

40

60

80

100

Cummulative User Edits

Base

SFT

DPO

EarlyEnsemble

LateEnsemble

0

25

50

75

100

125

150

175

200

Number of Test Rounds

0

20

40

60

80

100

Cummulative User Edits

Base

SFT

DPO

EarlyEnsemble

LateEnsemble

0

25

50

75

100

125

150

175

200

Number of Test Rounds

0

25

50

75

100

125

150

175

200

Cummulative User Edits

Base

SFT

DPO

EarlyEnsemble

LateEnsemble

(a) Summarization Domain with Strong User

0

25

50

75

100

125

150

175

200

Number of Test Rounds

0

25

50

75

100

125

150

175

Cummulative User Edits

Base

SFT

DPO

EarlyEnsemble

LateEnsemble

(b) Summarization Domain with Weak User

0

25

50

75

100

125

150

175

200

Number of Test Rounds

0

20

40

60

80

Cummulative User Edits

Base

SFT

DPO

EarlyEnsemble

LateEnsemble

0

25

50

75

100

125

150

175

200

Number of Test Rounds

0

20

40

60

80

Cummulative User Edits

Base

SFT

DPO

EarlyEnsemble

LateEnsemble

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper pioneers a principled framework for fine-tuning LLMs using user-edits (e.g., from writing assistants or coding agents). Derive bounds are first derived for learning algorithms that learn from each of these feedback types. A simple ensembling procedure is than proposed to jointly learn from these feedback types.\n\nThree feedback types, supervision, preferences, and rewards are integrated into a single theoretical and empirical paradigm. The authors derive sample complexity bounds for each approach, revealing trade-offs influenced by user behavior. \n\nTo balance these, they propose early and late ensemble methods. Experiments on email/summarization tasks show LateEnsemble’s robustness to user distribution shifts, reducing edit costs by a large margin. \n\nStrengths include theoretical grounding and practical algorithms; limitations involve future usage of pessimistic RL and simulated user validation.\n\nSTRENGTHS AND WEAKNESSES:\nStrong:\n1. the motivation of using user edit feedbacks is relatively interesting and helpful in real-world llm usages. the unifying of perference, supervision and reward learning is a fine-grained method for learning from the users' edition datasets.\n2. three complexity bounds for the three learning approaches are derived under relatively clear assumptions. these mathmatical analysis shall help related researchers to obtain more insights of better leveraging sft/rl algorithms for this direction.\n3. relatively strong experiments on email/summarization tasks with simulated users and showed that the effectiveness of their proposed methods.\n\nWeak:\n1. prefer to learn more connections between the theoretical bounds with more real-world validities, currently the discussion is short.\n2. human evaluation is limitted, and more real-world human usage is preferred.\n3. scalability requires more evidences such as larger datasets. More discussion or experiments on this direction is appreciated.\n\nQUESTIONS:\n1. Could prompt-based adaptation (e.g., in-context learning) complement fine-tuning for user personalization? and How? prefer to learn more details.\n2. any discussion on handling users' edit data privacy topics?\n3. can you conclude your contributions at the introduction section?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of post-training LLMs to learn from user edits. The proposed learning setup consists of two phases: the offline phase where a sufficiently large dataset of user edits can be used to fine-tune the LLM, and the online phase where the LLM interacts with users in deployment over a few episodes and can potentially be adapted. The paper considers three different approaches for post-training during the offline phase: supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning with a learned cost function (RL). The paper provides a theoretical analysis of the number of samples required for bounded regret guarantees in the three cases under certain assumptions on the policy class, and the user edits distribution -- further stating the trade-offs between the three methods. An early ensembling of the approaches is also considered. A late ensembling approach is proposed for the online phase: a bandit algorithm is run to select between the the different learned policies from the offline phase based on the cost of user edits.  Finally, the paper presents empirical results on two tasks (summarization and email writing) with strong / weak users simulated using LLMs -- demonstrating the theoretical trade-offs in practice.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper studies a well-motivated and timely problem, adapting LLMs to user preferences using naturally available user edit data.\n2. The theoretical analysis of the different post-training approaches (SFT, DPO, RL with a learned cost function) is original in the context of learning from user edits and the four assumptions on the user edit distribution and policy class appear reasonable.\n3. The empirical evaluation with a strong and weak user provides interesting insights (Section 5) about the trade-offs between the different approaches in terms of average regret with respect to the best policy.\n\n**Weaknesses**\n\n1. Protocol 1 is possibly incomplete: there is a mismatch between the line numbers in protocol 1 and the associated discussion on lines 107-113. For example, line 111 mentions line 9 in protocol 1 while the protocol has 7 lines in total.\n\nQUESTIONS:\n1. Could the authors provide some insight into why DPO is significantly worse than the base method on summarization with a different strong user (Table 2)?\n2. The empirical results in Section 5 use Llama-3.3-70B-instruct as the user and Llama-3.1-8B-instruct as the agent. I would expect these models to be very similar since they belong to the same model family (similar pre-training data, post-training approaches possibly leading to very similar generation distributions). Would the results in Table 1 be different if the user and agent are from different model families? I would encourage the authors to present some preliminary results on this if possible.\n\nOther comments:\n1. Line 59: \"one can use induce a preference data\" -> this doesn't sound right\n2. Line 70: \"approach perform better\" -> \"approach performs better\"\n3. Line 109: line 6 -> line 4 (incorrect line number from protocol 1)\n4. Line 267: \"The use an LLM\" -> \"The use of an LLM\"\n5. Line 291: \"We perform generate\" -> \"We generate\"\n6. Line 303: \"Equation 6 give challenges\" -> \"Equation 6 given challenges\"\n7. Line 325: \"this affect\" -> \"this effect\"\n8. Line 337: \"these maybe dependent\" -> \"these may be dependent\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles fine‑tuning LLMs from user‑edit logs—situations where a user edits an LLM’s draft. It formalizes how edits simultaneously provide: (i) supervised targets, (ii) pairwise preferences, and (iii) a reward signal proportional to edit distance. \n\nThe authors derive sample‑complexity bounds for learning from each signal, reveal their trade‑offs, and introduce two ensembling strategies. The LateEnsembl, UCB bandit that picks among separately fine‑tuned policies, which yields the lowest edit cost on simulated email‑writing and summarization tasks.\n\nSTRENGTHS AND WEAKNESSES:\n1. The paper is strong in theoretical quality. And it  tackles a timely and practically significant problem – leveraging naturally occurring user edits to improve LLMs. Unlike expensive hand-labeled data (human-written gold answers or explicit preference ratings), edit logs are abundant and user-generated. The authors are among the first to provide a unified framework for this scenario, identifying that preferences, demonstrations, and reward signals co-occur in user-edit data.\n2. The paper proposes a simple yet clever ensembling strategy to get the best of all worlds. Instead of committing to one feedback type, the approach either (i) blends the training objectives, or (ii) trains separate models for each and then uses an online bandit algorithm (UCB) to choose which policy to deploy at test time.\n3. A notable weakness is the lack of real human user data or human evaluation. All experiments rely on a simulated user (an LLM that performs edits based on preset preferences). While this is a reasonable proxy to validate the concepts, it remains uncertain how well the proposed methods would work with actual users. Real user edits might be noisy, inconsistent, or exhibit biases that a simulated LLM user does not. The paper would be stronger if it at least discussed or tested the approach on a small real-world dataset (if available) or provided a plan for such validation. \n4. The work identifies three feedback modalities (supervision, preference, reward), but in practice the pure reward-based approach (RL with edit distance) was not fully implemented in the experiments. \n5. The theoretical analysis, while insightful, relies on several strong assumptions (explicitly labeled Assumption 1–3 in the paper).\n\nQUESTIONS:\n1. Could you clarify how critical Assumption 1 (the contraction with rate $\\gamma_{\\min}$) and other assumptions are in practice? For instance, if real users sometimes do not fully correct every error (violating the contraction property) or if $\\gamma_{\\min}$ varies across contexts, how would that affect the learning outcome?\n\n2. What specific obstacles prevented implementing Eq. 6?\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-03"}