Aligning LLM Agents by Learning Latent Preference from User Edits
Authors: Ge Gao
Alexey Taymanov
Eduardo Salinas
Paul Mineiro
Dipendra Misra
Venue: NeurIPS 2024 poster (2024)
Keywords: NLP
LLM
preference learning
user feedback
user edits
Primary area: natural_language_processing

Abstract: We study interactive learning of language agents based on user edits made to the agent's output. In a typical setting such as writing assistants, the user interacts with a language agent to generate a response given a context, and may optionally edit the agent response to personalize it based on their latent preference, in addition to improving the correctness. The edit feedback is naturally generated, making it a suitable candidate for improving the agent's alignment with the user's preference, and for reducing the cost of user edits over time. We propose a learning framework, PRELUDE that infers a description of the user's latent preference based on historic edit data and using it to define a prompt policy that drives future response generation. This avoids fine-tuning the agent, which is costly, challenging to scale with the number of users, and may even degrade its performance on other tasks. Furthermore, learning descriptive preference improves interpretability, allowing the user to view and modify the learned preference. However, user preference can be complex and vary based on context, making it challenging to learn. To address this, we propose a simple yet effective algorithm named CIPHER that leverages a large language model (LLM) to infer the user preference for a given context based on user edits. In the future, CIPHER retrieves inferred preferences from the k-closest contexts in the history, and forms an aggregate preference for response generation. We introduce two interactive environments -- summarization and email writing, for evaluation using a GPT-4 simulated user. We compare with algorithms that directly retrieve user edits but do not learn descriptive preference, and algorithms that learn context-agnostic preference. On both tasks, CIPHER outperforms baselines by achieving the lowest edit distance cost. Meanwhile, CIPHER has a lower computational expense, as using learned preference results in a shorter prompt than directly using user edits. Our further analysis reports that the user preference learned by CIPHER shows significant similarity to the ground truth latent preference.

Aligning LLM Agents by Learning Latent Preference from User Edits
Ge Gao Alexey Taymanov 0+ Eduardo Salinas Paul Mineiro
Dipendra Misra c
Department of Computer Science, Comell University + ggao@cs.c cornell. edu fataymano, edus, pmineiro, dimisra) @mi crosoft. com
Microsoft Rescarch New York o
Abstract
We study interactive learning of LLM-based language agents based on user edits made to the agent's output. In a typical setting such as writing assistants, the user interacts with language agent to generate a response given context, and may optionally edit the agent response to personalize it based on their laten! preference, in addition improving the correctness. The edit feedback is naturally generated, making it a suitable candidate for improving the agent's 's alignment with the user's preference, and for reducing the cost of user edits over time. We propose a learning framework, PRELUDE that infers a description of the user's latent preference based on historic edit data. The inferred user preference descriptions are used to define prompts for generating responses in the future. This avoids fine-tuning the agent, which is costly, challenging to scale with the number of users, and may even degrade its performance on other tasks. Furthermore, learning descriptive preference improves interpretability, allowing the user to view and modify the learned preference. However, user preference can be complex, subtle, and vary based on context, making challenging to learn. To address this, we propose a simple yet effective algorithm named CIPHER that leverages the L.M to infer the user preference for a given context based on user edits. In the future, CIPHER re- trieves inferred preferences from the k-closest contexts in the history, and forms an aggregate preference for response generation. We introduce two interactive environments summarization and email writing, and use GPT-4 simulated user for evaluation. On both tasks, CIPHER outperforms several baselines by achieving the lowest edit distance cost while only having small overhead in LLM query cost over the base agent. Our analysis reports that user preferences learned by CIPHER show significant similarity the ground truth latent preferences!!
1 Introduction
Language agents based on large language models (LLMs) have been developed for a variety of applications (Dohmke) 2022 Brynjolfsson al. 2023). following recent breakthroughs in improving LLMs (Achiam et al. 2023 Ouyang et al. 2022b Team et al.) (023). However, despite their impressive zero-shot performance, LLMS still need to align given user and task (Mysore et al.)
2023 Li et al. 2023). In many applications, a natural feedback for LLM-based agents is user edits, where user queries the agent and edits the agent's response before their own final use. In contrast, typical feedback used for fine-tuning such as the comparison-busee
preference feedback in RLHF explicitly collected by providing annotators with model responses and asking them to rank (Ziegler ct al. 2019: Stiennon et al.) 2020: Nakano et al., 2021]: Ouyang et al., 2022a, inter alia), making such feedback an expensive choice for improving alignment. Motivated by this observation, we focus on interactive learning of LLM-based based language agents using user edits as feedback.
"quual contribution.
Our code and data are publicly available at https: com/ gao- g/prelude
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Interactive Learning from User Edits
Round
Step 1: User (and the world) provides context Tyt the LLM agent.
Article: (user-provided article)
Wt
t
Please summarize the above article.
Farming, a part of agriculture, involves growing crops rearing animals for food and raw materials. It began thousands Crescent, and led
Step 2: LLM Agent generates a response 3/+ given the context Ztt.
years ago, likely in the Neolithic Revolution as people transitioned from nomadic hunting to settied farming. This allowad significant human population.
Y/
FFrming, as part of agriculture involves growing crops cultivation and animal rearing for food and raw materials, Originated began thousands of years ago, likely the Fertile 1/
Minimize
Step 3: User edits response 1/: b/ before using it.
cumulative
Crescent, leading to the Neolihic Revolution
cost
Transition as people transitineed from nomadic hunting setlled farming, resulted in significant human population increase
Agent incurs a cost Cr = Aodit (Me, 366
Illustration of interactive learning from user edits. Color coding in edits is for visualization only - our agent takes the plain revised text as feedback.
Figure 1:
Consider the scenario in Figure I where a user interacts with an LLM-based writing assistant (agent) to complete their task. The interaction starts with the user (and the world) providing a context to the agent. This context may include query prompt provided by the user, along with additional information provided by the world, such as the content on the screen, current time, and the user's s calendar information. The agent generates textual response given the context.
In the beginning, the agent's response may not be optimal for the user, as it is not personalized to this user's individual needs and preference. As most users are not familiar with prompt engineering, and LLMs are often able to generate a reasonable response for the task, therefore, users may find it the most convenient to simply edit this response when it is not ideal, rather than trying different prompts to get new responses. The example in Figure Iillustrates that the user directly edits the summary generated by the agent to satisfy their preference for bullet point format. It takes time and effort for the user to make edits which can be measured using metrics such as the edit distance between the agent's 's response and the user edits. Our goal is to minimize the cumulative user edit cost over time using feedback from user edits. Notably, there is no distinction between training and testing in our setting as every natural use of the agent yields an edit feedback for learning.
We conjecture that user edits are driven by user's hidden preference which can be described in natural language. These preference descriptions are different from the notion of comparison-basee preference used in RLHF In this paper, we use the word preference to mean preference descriptions. For instance, preference of the user in Figure Il can be described as bullet points. In practice, user preference can be compound, such as preferring bullet point, informal, with emojis at the same time, and also context-dependent, e.g., informal tone when writing an email to a family member, and formal tone when writing to a colleague. In more complex settings, user preference can evolve with time (non- stationary), or depend on information unavailable the context (partially observed). Further, users may not be fully aware of all their preferences, or may fail to express these preferences in their query prompt. These considerations imply that user preference latent to the language agent. If the agent could learn the latent preference correctly, it can significantly improve its performance by generating satisfactory responses. Furthermore, preference learned by the agent can be shown to the user to enhance interpretability, and can even be modified by the user to improve correctness. Motivated by this, we propose learning framework, PRELUDE (PREference Learning from User's Direct Edits), where we seek to learn user preference description for a given context using the history of user edits.
In typical real-world scenario such as writing assistants, one has to potentially update the LLM-based agent for every user. Efficient approaches, therefore, must scale with the number of users. This makes approaches that fine-tune LLM parameters expensive to scale. Furthermore, LLMs typically undergo rigorous evaluation on a variety of safety tests before being released, and fine-tuning them can result in loosing the safety guarantees offered by these tests. For example, fine-tuning GPT-4 for millions of users can quickly turn very expensive. Approaches such as adding LORA and Adapter layers and only updating them, or using federated learning, can reduce the expense to some extent, but the loss of safety guarantees remains a concern. In this work, we focus
2

---

PRELUDE Framework. We describe our PRELUDE framework in Protocol 2 which is a specializa- tion of Protocol 1. In PRELUDE, in the 8th round, the agent infers the preference of the user as ft, and uses it to generate a response. We assume that in this round and for the given context Jiz the user has a latent preference fč that drives the user to perform all edits. Furthermore, agent was able to infer this latent preference (fe ft), then it will lead to minimal possible edits To remove the dependence on performance due the choice of the base LLM agent, compare on leveraging a frozen, black-box LLM, and instead learning prompr policy that can infer user preference description for given context, and then use directly drive the response generation.
We introduce a simple yet effective algorithm CIPHER that implements the PRELUDE framework. CIPHER infers user preference for every context in the history with the aid of L.M.. In the future, given a context, it retrieves inferred preferences of similar contexts from the history and uses them to generate response. CIPHER is computationally efficient and only slightly increases the LLM query cost compared to the base agent.
We introduce two interactive environments that evaluate the agent's 's ability to summarize documents and compose emails from a given notes. These tasks are inspired by writing assistant applications.Fu both tasks, simulate GPT-4 user that can generate edits based on a pre-designed latent preference that can vary based on the context. We evaluate CIPHER against several baselines and show that it achieves the lowest user edit cost. Additionally, CIPHER results in lower L.M query cost than other retrieval-based -aasd baselines. Finally, we analyze preferences learned by our agents, and find that they show significant similarity to the ground truth latent preferences in our setup.
Interactive Learning from User Edits and the PRELUDE Framework
We first describe LLM agents and the general learning framework from user edits and then discuss our PRELUDE framework and associated learning challenges.
LLM and I Language Agents. We assume access to a language agent that internally relies on an LLM. We make no assumption on the agent except that it can take as input piece of context which can include both texts and images and an additional prompt (which can be in-context learning examples or learned preferences) and generates text response. The language agent may simply perform greedy decoding of the LLM given the input or may perform complex planning to generate response.
Protocol Interactive Learning from User Edits.
1: for = 1, 2, ..., T do
2: User and the world provide context Zt
3: Agent generates a response y/ given the context
4: User edits the response to y/4
5: Agent receives cost of = Aeda (yt, 161
6: Evaluate the agent and learning algorithm on E Ce
Interactive Learning from User Edits. In an application such as writing assistant, a user interacts with the language agent over I' rounds. Protocol shows such learning protocol. In the tth round, the user and the world provide a context 2; E X where A' is the space of all possible contexts. This context will include the user prompt in text, along with additional information provided by the user or the world, and may include multimodal data as well such as images. Given the context the language agent generates a response y/ E y in text, where ) the space all texts. The user edits the response y/ to y/- the user does not perform any edits, we treat this as setting s/ ye- The agent receives a cost of G. = Dedia (Jt: 3/) for this round, which measures the user's efforts on making edits. The goal of the agent is to minimize the sum of costs across all rounds Σ Ct. In our experiments, we use Aedit as Levenshtein edit distance (Levenshtein 1965) in the token space which computes the minimum number of token insertion, deletion, and substitution necessary to convert to y/- In general, a higher edit distance implies that the user has made more edits and spent more efforts.
assume that if the
with an oracle agent that has access to fī at the start of each round. We assume that the LLM remains frozen across all methods in this work.
"The edit cost in practice may not always be 0, as the language agent could be incapable of adeptly using the correct preference, or the user may perform edits that inconsistent with their preference.
3

---

Protocol 2 PRELUDE: PREference Learning from User's Direct Edits
1: for tort= 1,2, , do
2: User presents text context 2t
3: Agent infers a preference ft using the history ((re, Me, 1/ 1/)
and context iy
4:
Agent uses Jt and Fy to generate response 1/2 User edits the response to V/ using their latent preference
5:
6: Agent incurs cost (y v()
7: Return Σ Ct
Challenges of Learning User Preference. Learning user preference from edits is challenging. In practice, user preference are multifaceted and complex. Furthermore, user's preference can also signif- icantly vary based on the context. The feedback in the form of user edits emerges naturally but is inher- ently implicit, lacking direct expressions of the actual preference and carrying subtleties that may lead to diverse interpretations. The combination of preference variability and the implicit nature of feed- back poses considerable challenges for agents in accurately learning and integrating these preferences.
3 Learning User Preference through Retrieval and Aggregation
In this section, we present our method, CIPHER (Consolidates Induced Preferences hased on Historical Edits with Retrieval), that learns user preference based on user edits.
Algorithm 1 CI 6). context representation function & : X + Rdd the retrieval hyperparameter k, and tolerance hyperparameter 5' 2 0. We initialize history D = 0.
1: for 6 == 1,2, ... T do
2: User (and the world) presents context 2,
3: Retrieve the top-k examples { (d(x=,; (1 in D with maximum cosine similarity to ((x;))
4:
A > 1, then query the L.M to aggregate these preferences (f=,) into ft, else ft = Szn
5:
Agent generates text response Ye based dt and
6:
User edits the response to y/ using their latent preference fü
7:
Agent incurs cost & A (Vs,, v()
8:
if c, < 6 then
9:
Jt f
10:
else
Query the LLM to generate preference ft that best explains user edits in (yt, v/)
12:
D + DU ((d(x)),
Return T Σt=1 Cz
13:
Algorithm shows CIPHER which implements the PRELUDE framework. CIPHER maintains preference history D. agent. CIPHER assumes access context representation function & X' .Rd that can map context to a vector representation. For a given round with context 2t, the agent first retrieves the k-closest contexts from the interaction history D, We use cosine similarity for computing proximity, although other metrics such as Euclidean distance, or Hamming distance when & outputs a binary vector, can be used. Given the retrieved contexts and their inferred preferences
of past contexts 2ạ along with preference fe inferred by the
we query the underlying LLM to summarize the inferred preferences (fz, 14= into single preference ft. In the beginning, when t I k, we retrieve all the past contexts. particular, for we have fi as an empty string as the agent has no prior knowledge of this user's preference
f2,
The agent the inferred preference S. generate the response. This is done by concatenating the context xe with an agent prompt such as "This user has a preference of <ʃt> which must be used when generating the response", where <fi> indicates where we insert the inferred preference fr. We list the actual template used in our experiments in Table in Anpenndii B
Given the user edits vl. if the user edits minimal, i.c., Aedie(Vt. v() 8 for hyperparameter 8, then we the inferred preference for this round as f, Je using fa for generating response
In practice initialize with publicly available preference history.

---

Table Latent user preference design, specific to the document source.
Doc Source
Latent User Preference
Scenario
Summarization
News article
targeted to young children, storytelling. short sentences, playful language, interactive, positive second person narrative, brief, show emotions, invoke personal reflection, immersive bullet points, parallel structure, brief
introduce political news to kids
(See et al. 2017)
Reddit post
for character development in cre- ative writing take notes for key knowledge
(Stiemnon ct al
2020
Wikipedía page
[Foundation 2022
Paper abstraci
tweet style, simple English. inquisitive, skillful forcshadowing, with emojis question answering style, direct, concise
promote a paper to invoke more attention and interests quickly get main opinions
(Clement ct al
2019
Movic reVIeW
MMass et al.
2011
Email Writing
Personal problem
informal, conversational, short, closing
share life with friends
(Stiennon et al.
2020
Paper review
casual tone, positive, clear, call to action
peer review to colleague
(uua et al. 2019)
Paper tweet
engaging. personalized, professional tone, thank- ful closing
networking emails for researchers
Har 2022
Paper summary
structured, straight to points, respectful, pro- fessional greeting and closing
milestone report to superiors
(Kershaw & Koeling
2020)
resulted in minimal edits. However, if Aumiur vf) 6, then we query the LLM third time to generate the inferred preference f. that explains why the user edited y/s to y/- We call this the Latent Preference Induction (LPI) step. In both cases, we append (Iz, ft) to the preference history.
Note that cannot query the LLM for the inferred preference in the first case where the user edit cost e is small, i.c., C, S 6. In this case, querying the LLM to infer the preference to explain the edits in v/ given yt, will result in the LLM outputting that the agent has preference which is incorrect.
Computational Cost of CIPHER. In a given round, CIPHER adds maximum of LLM calls on top of the cost of calling the underlying inference algorithm of the agent in line 5| CIPHER further reduces the memory storage by only storing the representation of contexts in the preference string instead of the input itself. Finally, CIPHER only adds small prompt to the context zz, before calling the agent's inference algorithm. This only slightly increases the length of the prompt, thereby, reducing the query cost associated with LLMs that scales with the number of input tokens.
4 Experiment
We first introduce two interactive tasks for learning from user edits, and then describe our results.
4.1 Two Interactive Writing Assistant Environments for Learning from User Edits
Task. We introduce two tasks inspired by the use of LLMs as writing assistants (Mysore et al. 2023 Shen et al., 2023 Wang al., (023). In the first task, we evaluate the agent's 's ability to summarize a given document. notes. For both tasks, we use documents from several existing sources listed in Table These sources represent a diverse category of documents that a writing assistant would typically encounter (see Table in Appendix for examples). In any given round, the user is provided context that is document from one the sources for the given task. Importantly, the agent is unaware of the source of the given document which as we discuss later, will determine the user preference. For both tasks, we run an experiment for T = 200 rounds. We sample an equal number of documents from each source and mix them remove any temporal correlation in document sources.
the second task, we evaluate the agent's s ability to compose an email given
Two-Stage GPT-4 Simulated User. We simulate a user that can edit a given response. We define a set of latent user preferences for the user that vary based on the document source. Table lists the preference for every source. This captures the context-dependent nature user preferences as the document source influences the type of context. For example, the Personal problem document
5

---

source contains documents pertaining to discussions with friend, and a user may have a different preference when writing an email friend compared writing an email to a colleague. We assume that our user is aware of the document source de of a given context Jt. This implies, that we can express the true user preference for Te as Si = F(de) where maps given document source to the user preference. Recall that the agent is never provided the document source of any context.
We model our user using GPT-4 with a two-stage approach. Given an agent response yr. and the context F,, we first query GPT-4 to check if yr satisfies the preference in ft If the answer is yes, then the user preforms no edits and returns 1/: = ye. If the answer no, then we use GPT-4 to generate the edited response v given yt and fr. We found that our two-stage GPT-4 user can generate high-quality edits, consistent with observations in prior work that LLM-written feedback is high- quality and useful to learn from (Bai et al. 2022 Saunders et al. (0022. We adopted two-stage process since using GPT-4 to directly edit the response Yt always resulted in edits even when the response satisfied the preference f We provide GPT-4 user prompt template and user edit examples in Apppeddix B]
Evaluation Metric. We propose three metrics for evaluating agents learning from user edits. Our main metric is the cumulative user edit cost Σ Dedie (3h: y') over T rounds where Aedi (ye, v/) is the Levenshtein edit distance between agent response Y/t and user edits v/ computed in the token space using Tiktoken tokenizer. For methods that learn an interpretable preference, we additionally evaluate the quality of the inferred user preference ft- We do so by evaluating if fo is closer to the true preference ft- = F(d)), where de is the document source of context in round t, compared to preference of any other document source. Formally, we compute + Σ/= [d, = arg max BERTSscore ft, F(d)), where BERTScore (Zhang* et al., 2020) is a text similarity metric and S is the set of all document sources. Finally, we report the total number of input and output BPE tokens to the LLM across all rounds. This measures the expense associated with using LLM, used by popular LLM providers to charge their customers.
4.2 Details of CIPHER and Comparison Systems
We use GPT-4 as our base LLM for CIPHER and all baselines. We do not perform fine-tuning of the GPT-4 and do not add any additional parameters to the model. We use a prompt-based GPT-4 agent for all methods that uses a single prompt with greedy decoding to generate the response. Our main method CIPHER and the baselines, can be extended to more complex language agents that perform multiple steps of reasoning on top of the base LLM before generating response.
CIPHER Details. We use a simple agent that uses GPT-4 with a prompt template to generate the response ye given the context 2; and preference ft- We list templates in Table 7 in Appendix B, We experiment with MPNET (Song et al.) 2020) and BERT (Devlin et al. (019)) as our two context representation functions Φ. and use cosine similarity for retrieval. We experiment with two different values of the number of retrieved examples A E (1,
Baselines. We evaluate CIPHER against baselines that either perform no learning, or learn context- agnostic preferences, or directly use past edits to generate a response:
1. No learning: The agent performs no learning based on interaction with the user.
2. Explore-then-exploit (E-then-e) LPI: This baseline is based on the classic explore-then-exploii strategy in interactive learning Garivverr et al. 2016). The agent first generates responses for the first T. rounds without performing any learning (exploration stage). It then infers a single user preference fe using the user edits in the first Te rounds by applying the LPI step (Algorithm line 11I. which is used to generate responses for remaining rounds (exploitation step).
3. Continual LPI: This baseline is similar to E-then-e LPI except that it never stops exploring and avoids overfitting to the first T. rounds. In any given round t, it uses the data of all past edits {(ve, o M=1 to learn preference fa by performing the LPI step. It then generates a response using this preference. Similar to E-then-e LPI, this approach learn context-agnostic preferences.
4. ICL-edit: This is standard surievallbased in-context learning (ICL) baseline (Brown et al.) 2020). In a given round I, the agent first retrieves the closest examples {(Ve, (( 1/ to the given context d, using the representation function Φ. These examples are provided in an ICL prompt and use to generate the response yt- This approach does not learn preferences but unlike E-then-e LPI and Continual LPI it can perform context-dependent learning.

---

5. CoT-edit: This is a standard retrieval-bars -aaed chain-of-thoupe ohotht (CoT) baseline (Wei et al. 2022). This baseline is similar to ICL-edit except the prompt for generation requires the agent to infer a user preference Sa based on retrieved k examples, and generate an output according to f
Oracle Method. We also evaluate an oracle approach which uses the true user preference in each round to generate the response. This provides an upper bound on performance and helps to evaluate setup is desl-des,gn,d, i.e, whether learning the true user preference indeed leads to low edit costs.
4.3 Main Result and Discussion.
Main Results. Table 2 reports the performance of all methods on the two tasks three metries. We report the mean and standard deviation across 3 different random seeds $
Table 2: Performance of baselines and our methods in terms of cumulative edit distance cost and classification accuracy. Pe denotes the mean j and standard deviation a across 3 runs over different seeds. Expense column shows budget as the average number of input and output BPE tokens across 3 runs (unit 10%). We use -k in method names to denote that we use retrieved examples. Numbers in bold are the best performance in each column excluding oracle preference method, underline for the second best, and dotted underline for the third best.
Method
Summarization Accuracy!
Email Writing
Edit Distance.
Expense
Edit Distancel
Accuracyt
Expense
Oracle Preference
6,5731.451
1.000
1.67
1,851
1.000
1.62
No Learning E-then-e LPI
48,269us7
1.50
31,103
1.65
0.21 80.0s
1.99
24,562
0.2633.00
1.73
Continual LPI
57,9152,10
0.233
8.89
26,8521.464
0,243a.19
8.63
(CL-edit-S-MPNEES
38.560
8.00
32,405 1307
12.12
ICL-edit-S-BERR
39.7341.90
7.96
30, 40,9499.50
11.55
CoT-edit-SMPPETT
40,747
0.230m.026
6.82
24,2923.503
0.300
8.74
CoT-edit-5-BERT BBET
41,0881,846
0.230
6.92
24,301 1,32
0.263aas2
8.26
CIPHER-I-MPNET
33,9264,000
0.520b.o22
2.74
10,781
0.4355..84
1.94
CIPHER-5 --MER-
32,974195
0.478%.010
3.00
10,0581.709
0.4677.01
2.09
CIPHER-I-BERE
37,637
0.565
2.81
12,6344x68
0.4877.125
1.99
CIPHER-S-BERT 35.8113384
0.478p.02s
3.03
8,391
0.3630.075
2.22
Discussion of Main Result. We observe that not performing learning results in a high edit cost, whereas using the oracle preferences achieves significantly smaller edit cost. This shows that our environments are sound and well-conditioned. E-then-e LPI and Continual LPI learn context-agnostic preferences which cannot capture the context-dependent preferences in the environments and end up doing poorly. For the summarization task, they end up with a higher edit distance than even performing no learning. One possible explanation is that using context-agnosse the model to specialize to given preference much more than the base model, resulting in more edits when that preference is incorrect. We see this in preference accuracy, which is low for both of these baselines, and lower for the summarization task than the email writing task where they outperform no learning baselines. Further, Continual LPI has a higher expense cost due to constantly querying the LLM to infer the user preference.
preferences can push
ICL-edit baselines perform significantly better on the summarization task. However, using list of user edits the prompt results higher token expense cost, as the responses and their edits can be significantly long in practice. Further, the ICL-edit baselines provide no interpretable explanation for their response or for explaining user behavior. Although CoT-edit baselines provide an interpretable preference, they still result in relatively high expense and low classification accuracy.
CIPHER achieves the smallest edit distance cost reducing edits by 31% in the summarization task and 73% in the email writing task. We observe that retrieving k = 5 preferences and aggregating them achieves lower edit distance, however, the choice of ideal representation seems lask-dependent.
Appendix B reports additional details of our baselines, such as hyperparameters and prompt templates. We randomize the context sampling from source datasets, so experiments on different seeds contain different sets of input contexts. On the same seed, experiments across different methods are strictly comparable, as both the of input contexts and the order of input context
the same in our implementation.
7

---

Figure 2: Learning curves of different methods based cumulative cost time (average across seeds). In the legend, -k means with top retrieved examples, -B for BERT, and -M for MPNET.
-10
Summarization
-10*
Email Writing
Oracle
Learning
6
B-then-e
Cumulative Cost
Continual
Cumulative Cost
ICL-edit-B
4
ICL-edit-M
2
CoT-edit-B
Cot-edit-M
CIPEER-I-B
2
CIPHER-S-B
1
CIPHER-I-M
CIPHER-
0
40
80
120
160
200
40
80
120
160
200
Round
Round
Figure 3:
Percentage of zero-cost examples of CIPHER over time, binned per 20 rounds to show the trend (average across 3 seeds). the legend, -k means with top k retrieved examples, -B for BERT, and -M for MPNET.
Summarization
Email Writing
0.6
0.8
CIPHER-S-B
% Zero-coss cost
per Bin
CIPHER-I-M
Zero-cost
per Bin
0.6
CIPHER-S-M
0.4
0..4
Oracle
Ex.
0.2
0.2
40
80
120
160
200
40
80
120
160
200
Round
Round
Further, CIPHER achieves the highest preference accuracy showing that CIPHER can learn preferences that correlate more with the ground truth preference than preferences of other document sources. Note that the performance of random preference classifier is only 20% for summarization and 25% for email writing. Further, CIPHER achieves a smaller cost than ICI.-edit and Continual IPI baselines, as it doesn't use long user edits in the prompt for generating response. In summary, CIPHER provides cheap, more effective, and interpretable method than our baselines.
Learning Curves. We plot mean cumulative user edit costs over rounds in Figure 2] The cumulative user edit costs in Figure 2 show that the angle of the learning curves decreases for CIPHER after an initial number of rounds, showing that learning helps decrease the rate at which user edits are accu- mulated. In contrast, the angle of the learning curve for the no-leanning baseline remains unchanged.
Evaluating Fraction of Non-Edited Responses. Recall that the first stage our GPT-4 user checks if the agent response satisfies the latent user preference f*. If it does, then edits performed, otherwise, the user edits the response. We plot the percentage of examples with zero edit cost per 20 rounds bin in Figure 3] We notice small increase in the number of examples with edit cost. This indicates that gains come not just by increasing the number of examples that avoid getting edited in stage of our user but more generally across examples.
Qualitative Analysis Learned Preferences. We evaluate the quality of preferences learned by CIPHER the harder summarization task. Table 3 lists 3 learned preferences per document source for CIPHER-51 5-PHER-5-MPNET which are randomly sampled at the beginning, middle, and end of the interaction history. We that overall the agent can learn reasonable description of the latent preference. For example, it can learn bullet points preference for Wikipedia articles, and second person narrative for Reddit posts, and QA style for Movie reviews. CIPHER pick some preferences fairly early such as bullet points for Wikipedia and emojis for Paper abstract, whereas some learned only later such as Structured Q&& for Movie reviews. This shows using CIPHER can quickly learn useful preferences, but further interaction continues to help
'We present more additional analysis in Appendix C including detailed expense report, normalized edit distance cost, failure analysis, and retrieval accuracy.

---

Table 3: Examples of learned preferences on summarization task with CIPHER-5-MPNEL grouped
based on the document source and corresponding latent preference. We randomly sample 3 examples per type at the beginning, middle, and end of the interaction history.
Latent User Preference (Round) Learned Preference
News article. targeted to (22) Fairy tale narrative style, informal and conversational tone, use of rhetorical young children, storytelling, questions, simplified language.
short sentences, playful lan- (115) Simplified, childlike storytelling with playful language and imagery
guage, interactive, positive
(192) Simplified and playful storytelling language
Reddit post, second person (14) Concise and coherent storytelling
second-person narrative, brief, show emo- (102) The user prefers a second person narrative and more direct, personal tone
tions, invoke personal reflec- (194) Poetic and descriptive language, narrative perspective shift to second person tion, immersive
Wikipedia page. bullet (19) Concise, Bullet-Pointe's Structured Summaries with a Narrative Q&A Style
points, parallel structure, (124) Concise and factual writing style. bullet-point formatting brief
(197) Concise and streamlined formatting, with bullet points and clear subhead- ings for casy scanning
Paper abstract. tweet style, (20) Concise, conversational summarics with bullct points and emojis.
simple English, inquisitive. (111) Concise, conversational, whimsical bullet-point summaries with emojis. skillful foreshadowing, with
emojis
(193) Concise, conversational, and whimsical bullet-point summaries with emojis.
Movic review. question an- swering style
(12) The user prefers a straightforward, clear, and concise writing style with factual formatting.
(123) The user prefers a clear and concise question and answer format with straightforward language. (199) Concise, Structured Q&A with Whimsical Clarity
4.4 Human Evaluation
We conduct two types of evaluation with human users to further understand the performance of our methods on summarization. We focus on our best-performing method CIPHER-S-MPNE:7]
Win Rate Evaluation. We conduct win rate evaluation where evaluators are given a pair of text and choose which one has higher quality. We compare the output of CIPHER-S-MPNET against the output of the best-performing baseline /CL-edii-S-MPNET, and against the generation of the oracle method. Each evaluation covers 15 text pairs, with three random samples from each scenario in the last 50 rounds of interaction. We conduct these CIPHER vs. ICL and CIPHER vs. Oracle evaluations with 7 human evaluators recruited through our personal network. For each text pair, we consider the output receiving the majority vote as a win. We find that the win rate of CIPHER-5-MPNEZ against JCI-edii-S-MPNET is 73.3%. This confirms that our method outperforms the best-performing baseline for human users. In CIPHER vs. Oracle evaluation, the win rate of CIPHER-S-MPNET is 23.7%, which reflects the performance gap we reported in previous sections
Edits by Human Users. We study the edit feedback from human users the generation of CIPHER- 5-MPNET and the oracle method. We instruct human users to edit the output based on the given latent preference, and to leave no edits when the output aligns with the given preference. We mix 20 outputs from CIPHER-5 5-MEET and the oracle method so that human users cannot tell the source of each out- put. The total edit distance, averaged across human users, 211 for CIPHER, and 98 for the oracle method. The averaged percentage of zero-edit examples is 60% for CIPHER and 76.7% for oracle.
5 Conclusion
We study aligning L.L.Mbaseed agents using user edits that arise naturally in applications such as writ- ing assistants. We introduce the PRELUDE framework that seeks to learn the latent user preferences that drive these edits, and uses them to generate a response. We propose a practical algorithm CI- PHER that implements PRELUDEand outperforms baselines two interactive tasks with GPT-4 simulated user. Evaluating CIPHER with human-in-the-loop as well as developing algorithms that can fine-tune LLMs using user edit where fine-tuning is feasible, are interesting future work directions.
"Examples in each evaluation have no overlap. which are sampled from experiments different seeds.

---

Acknowledgments
Gao was a research intern in MSR NYC, and later partially supported by NSF project #1901030. All content represents the opinion of the authors, which not necessarily shared or endorsed by their respective employers and/or sponsors. We thank MSR NYC research community, Jonathan D. Chang. Daniel D. Lee, Claire Cardie, and Sasha Rush for helpful discussions and support. We also thank Stéphane Aroca-Ouelleee Kyunghyun Cho, and Columbia NLP community for their valuable feedback.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.0877: 2023.
Sweta Agrawal and Marine Carpuat. An imitation learning curriculum for text editing with non- autoregressive models. ArXiv, abs/2203 099466 2022.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer. Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.
Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov. Correcting diverse factual errors in abstractive summarization via post-editing and language model infilling. ArXiv, abs/2210. 10.13780 2022.
Nitsan Bar. Papertweet.
2022.
Jan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das. Learning split and rephrase from wikipedia edit history. ArXiv, abs/1 88880988666 2018.
Shaked Brody, Uri Alon, and Eran Yahav. A structural model for contextual code changes. Proceed- ings of the ACM on Programming Languages, 4:1 -88, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Erik Brynjolfsson, Danielle Li, and Lindsey R Raymond. Generative ai at work. Technical report, National Bureau of Economic Research, 2023.
Mengyao Cao, Yue Dong. Jiapeng Wu, and Jackie Chi Kit Cheung. Factual emror correction for abstractive summarization models. ArXiv, abs/2010.08712, 2020.
Jonathan Chang. Kiante Brantley, Rajkumar Ramamurthy, Dipendra Misra, and Wen Sun. Learning to generate better than your Ilm. arXiv preprinl arXiv:2306.1 1816, 2023.
Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and learning for end-to-end program repair.
Monperrus Martin. Sequencer: Sequence-to-sequuee IEEE Transactions on Software Engineering, 47:1943-1959 2018.
Ching-An Cheng, Andrey Kolobov, Dipendra Misra, Allen Nie, and Adith Swaminathan. Lif-bench: Benchmark for interactive learning from language feedback. arXiv preprint arXiv:2312. 068653,, 2023.
10

---

Ruining Chong, Cunliang Kong, Liu Wu, Zhenghao Liu, Ziye Jin, Liner Yang, Yange Fan, Hanghang Fan, and Erhong Yang. Leveraging prefi transfer for multi-intent text revision. Annual Meeting of the Association for Computational Linguistics, 2023.
Colin B. Clement, Matthew Bierbaum, Kevin O'Keeffe, and Alexander Alemi. On the use of arxiv as a dataset, 2019.
Mike D' Arey, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope, and Doug Downey. Aries: A corpus of scientific paper edits made in response to peer reviews. ArXiv, abs/2306.12587, 2023.
Mingkai Deng, Jianyu Wang. Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint 1XXi:22005.1248, 2022.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics, 2019.
Thomas Dohmke. Github copilot is generally available to all developers. https://githubb blog/ copilot- si-githuub-coploo--ie
2022-06- 21 -a--gthhub--o Accessed: 2pr12-20-2024.
generally- btg.appoooooolo--ooooooooooaoaaa tteelaablleeleeeeeee all deve-leeeeerrrs///// 2022.
Wanyu Du, Vipul Raheja, Dhruv Kumar, Zae Myung Kim, Melissa Lopez, and Dongyeop Kang. Understanding iterative revision from human-written text. ArXiv, abs/2203.03802, 2022.
Felix Faltings, Michel Galley, Gerold Hintz, Chris Brockett, Chris Quirk, Jianfeng Gao, and Bill Dolan. Text editing by command. ArXiv, abs/2010. 128266 2020.
Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Sherry Wu, Graham Neubig, and André F. T. Martins. Bridging the gap: A survey on integrating (human) feedback for natural language generation. ArXiv, foss/3355000555 2023.
Wikimedia Foundation. Wikimedia downloads. https: btttps:/ddupss http://ddmmss.iiiiiieeeee wikimedia. org 2022.
Aurélien Garivier, Tor Lattimore, and Emilie Kaufmann. On explore-then-commi strategies. Ad-
vances in Neural Information Processing Systems, 29, 2016.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024.
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450, 2017.
Braden Hancock, Antoine Bordes, dialogue after deployment: Feed yourself, chatbot! Annual Meeting of the Association for Computational Linguisties, 2019.
Mazaré, and Jason Weston. Learning from
Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. Argument mining for understanding peer reviews. Proceedings of the 2019 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers),
June 2019.
Daniel James Kershaw and R. Koeling. Elsevier oa ec-by corpus. ArXiv, abw.200000070074, 2020. doi: https///oo..og//0045555/rr.... 0077744 URL.itttps://eeeevirr diggeerrer..gggiaaalooosooss.. com homm/dataasetsszm33cddnddd
Zae Myung Kim, Wanyu Du, Vipul Raheja, Dhruv Kumar, and Dongyeop Kang. Improving iterative text revision by learning where to edit from other revision tasks. ArXiv, abs/2212.01350, 2022.
11

---

Philippe Laban. Jesse Vig, Wojciech Kryscinski, Shafiq R. Joty, Caiming Xiong, and Chien-Sheng Wu. Swipe: A dataset for document-level simplification of wikipedia pages. Annual Meeting of the Association for Computational Linguistics, 2023.
Mina Lee, Katy Ilonka Gero, John Joon Young Chung, Simon Buckingham Shum, Vipul Raheja, Hua Shen, Subhashini Venugopalan, Thiemo Wambsganss, David Zhou, Emad A. Alghamdi, Tal August, Avinash Bhat, Madiha Zahrah Choksi, Senjuti Dutta, Jin L.C. Guo, Md. Naimul Hoque, Yewon Kim, Seyed Parsa Neshaei, Agnia Sergeyuk, Antonette Shibani, Disha Shrivastava, Lila Shroff, Jessi Stark, S. Sterman, Sitong Wang, Antoine Bosselut, Daniel Buschek, Joseph Chee Chang, Sherol Chen, Max Kreminski, Joonsuk Park, Roy Pea, Eugenia H. Rho, Shannon Zejiang Shen, and Pao Siangliulue. A design space for intelligent and interactive writing assistants. Conference on Human Factors in Computing Systems, abs/2403. 11124403.4117,, 2024.
Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics. Dokludy, 10:707-710, 1965.
Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, and Michael Bendersky. Automatic prompt rewriting for personalized text generation. arXiv preprini arXiv:2310.00/522 2023.
Jiwei Li, Alexander Miller, Sumit Chopra, Marc' Aurelio Ranzato, and Jason Weston. Dialogue learning with human-in-the-loop. ArXiv, abs/161 1.09223, 2016.
Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie Chi Kit Cheung, and Siva Reddy. Using in- teractive feedback to improve the accuracy and explainability of question answering systems post-deployment. ArXiv, abs/2204.03025, 2022.
Ruibo Liu, Chenyan Jia, Ge Zhang. Ziyu Zhuang, Tony X. Liu, and Soroush Vosoughi. Second thoughts are best: Learning to re-align with human values from text edits. ArXiv, abs//301.00000 103055, 2023.
Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron L Halfaker, Dragomir R. Radev, and Ahmed Has- san Awadallah. On improving summarization factual consistency from natural language feedback. Annual Meeting of the Association for Computational Linguisties, 2022.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, June 2011.
Chaitanya Malaviya, Subin Lee, Dan Roth, and Mark Yatskar. Pachinko: Patching interpretable qa models through natural language feedback. ArXiv, abs/231 0tss/2311.0965 1.09558, 2023.
Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. Felix: Flexible text editing through tagging and insertion. ArXiv, abs/2003.10687 2020.
Edison Marrese- Taylor, Machel Reid, and Yutaka Matsuo. Variational inference for learning repre- sentations of natural language edits. ArXiv, abs/2004.09143 2020.
Edison Marrese-Taylor Machel Reid, and Alfredo Solano. Edit aware representation learning via levenshtein prediction. The Fourth Workshop on Insights from Negative Results in NLP, 2023.
Dipendra Misra, Aldo Pacchiano, and Robert E Schapire, Provable interactive learning with hindsight instruction feedback. arXiv preprint 4xXiv:2404.0912. 2024.
Masato Mita, Keisuke Sakaguchi, Masato Hagiwara, Tomoya Mizumoto, Jun Suzuki, and Kentaro Inui. Towards automated document revision: Grammatical error correction, fluency edits, and beyond. ArXiv, abs/2205. 1484, 2022.
Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longqi Yang, Steve Menezes, Tina Baghaee, Em- manuel Barajas Gonzalez, Jennifer Neville, and Tara Safavi. Pearl: Personalizing large language model writing assistants with generation-calibratee retrievers. arXiv preprint a 1,09180, 2023.
12

---

Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou. Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback. ArXiv, 2021.
Khanh X Nguyen, Dipendra Misra, Robert Schapire, Miroslav Dudík, and Patrick Shafto. Interactive learning from activity description. International Conference on Machine Learning, pp. 8096-8108, 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, 2022a.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022b.
Dominic Petrak, Nafise Sadat Moosavi, Ye Tian, Nikolai Rozanov, and Iryna Gurevych. Learning from free-text human feedback collect new datasets or extend existing ones? ArXiv, abs/2310.15752 2023.
Dheeraj Rajagopal, Xuchao Zhang, Michael Gamon, Sujay Kumar Jauhar, Diyi Yang, and Eduard H. Hovy. One document, many revisions: A dataset for classification and description of edit intents. International Conference on Language Resources and Evaluation, 2022.
Machel Reid and Graham Neubig. Learning to model editing processes. Conference on Empirical Methods in Natural Language Processing, 2022.
Machel Reid, Vincent J. Hellendoom, and Graham Neubig. Diffuser: Diffusion via edit-based reconstruction. International Conference on Leurning Reprexentations, 2023.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Ouyang Long, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. ArXiv, abs/2206.05802 2022.
J' J'er'emy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale. ArXiv, abs/2303. 16755, 2023.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer- generator networks. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), July 2017.
Zejiang Shen, Tal August, Pao Siangliulue, Kyle Lo. Jonathan Bragg, Jeff Hammerbacher, Doug Downey, Joseph Chee Chang, and David Sontag. Beyond summarization: Designing support for real-world expository writing tasks. arXiv preprint arXiv:2304.02623, 2023.
Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels. ArXiv, 1589233 2022.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. ArXiv, abs/2004.09297, 2020.
Felix Stahlberg and Shankar Kumar. Seq2edits: Sequence transduction using span-level edit opera- tions. Conference on Empirical Methods in Natural Language Processing, 2020.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. ArXiv, abs/2009.01 01355, 2020.
13

---

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Sitong Wang, Lydia B Chilton, and Jeffrey V Nickerson. Writing with generative ai: Multi-modal and multi mulei-ddimensoonl tools for journalists. The Second Workshop on Intelligent and Interactive Writing Assistants at ACM CHI, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201. 1112001.1190.. 2022.
Jason Weston. Dialog-based language learning. ArXiv, abs/1604.06045, 2016.
Jing Xu, Megan Ung. Mojtaha Komeili, Kushal Arora, Y-L.n Boureau, and Jason Weston. Learn- ing new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. Annual Meeting of the Association for Computational Linguistics, 2022.
Ziyu Yao, Frank F. Xu, Pengcheng Yin, Huan Sun, and Graham Neubig. Learning structural edits via incremental tree transformations. ArXiv, abs/2101.12085 2021.
Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L. Gaunt. Learning to represent edits. ArXiv, abs/ 8 $10.13331 2018.
Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li, and Miloš Gligorié. Coditt5: Pretraining for source code and natural language editing. Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. International Conference on Learning Representations, 2020.
Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, 2019.
14

---

Appendix
A
Related Work
We describe related work in this area grouped by main themes in this work.
Learning from Feedback. Besides pair-wise comparison feedback from annotators used in Rein- forcement Learning from Human Feedback (RLHF) research (Ziegler et al. 2019 Stiennon et al. 2020 Nakano et al. 2021] Ouyang et al. [022a]] inter alia). prior work has also studied free-form text feedback provided by annotators (Fernandes et al. (023), such as on the task of dialog (We- ston 2016 Li ct al., 2016, Hancock al. 2019; Xu et al. 2022; Petrak et al. 2023), question answering (i et al. |022| et al. 2023), summarization (Saunders et al. 2022), and general decision making (Cheng et al.] 2023). This feedback, tailored to each example, is often utilized to rank candidate outputs, thereby improving task performance. Some work studies learning from
text feedback to generate outputs directly (Scheurer et al.) 2023] Bai et al.] 2022: Shi al. (022), by generating multiple refinements of the original output based on the feedback and fine- tuning the original model to maximize the likelihood of the best refinement. In grounded settings such as instruction-basse
navigation, one line of work has also used hindsight feedback that explicitly provides text instruction for the generated trajectory, to train policies (Nguyen et al. 2021: Misra et al., 2024). Moving beyond the conventional focus on text feedback that explicitly articulates human intent, we investigate feedback in the form of direct edits on the original model output. Such revisions by users occur naturally during model deployment in practice. Additionally, we examine the learning of user preferences through historical interactions, aiming to surpass the constraints of example-specife feedback.
Language Agents and Personalization. LLMs have enabled the development of language agents for a variety of tasks from writing assistants (Lee et al. 2024), coding assistants (Dohmke) 2022). and customer service assistants (Brynjolfssonn et al. 2023). Since these I.L.-based assistants are often used by individuals, natural question has arisen on how to personalize these agents for each user. Straightforward approaches for fine-tuning LLMS includes supervised learning, online DPO (Guo et al. 2024), learning-to-searde (Chang et al. 2023), and reinforcement learning (Ouyang) itt al. 2022b). These approaches can be directly applied to our setting. For example, one can use (Jt, v/) in Protocol 1 as the preference data where yí is preferred over Yt, or use y/ as the ground truth for supervised
However, fine-tuning is expensive and hard to scale with the number of users. Therefore, a line of work has explored improving the alignment of frozen LLMs by prompt engineering, such as learning a personalized retrieval model (Mysore et al. 2023). learning a prompt policy given a reward function (Deng et al. 2022), , more generally, learning to rewrite the entire prompt (Li et al. 2023). We focus on learning prompt policy by learning from user edits, and specifically, using them to extract textural descriptions of user preference.
Edits and Revisions. Many prior work on editing model output focuses on error correction, such
as fixing source code (Yin et al. 2018 Chen al., 2018 Reid et al. 2023) and improving the factual
consistency of model summaries (Cao et al. 2020; Liu et al. 2022 Balachandran et al. (022). A
line of work has explored understanding human edits based
edit history of Wikipedia (Botha
et al. 2018: Faltings et al., 2020 Rajagopal et al. 2022 Reid & Neubig 2022) Laban et al.
2023).
or revisions of academic writings (Mita et al. [202 Du et al. al.] 2022 D' Arcy et al. 2023
Prior
work explores predicting text revisions with edit intents (Brody et al. 2020 Kim et al.
2022
Chong
al. 2023), and modeling edits with various approaches, including latent vectors (Guu al.
2017
Marrese- Taylor et al. 2020 2023), structured trees (ao et al. (022). discrete diffusion process (Reid
et al. 2023), or a series singular edit operations (Stahlberg & Kumar [020)] [aalinson al.
2020 Agrawal & Carpuat 2022 Zhang et al. [2022] Liu et al. (023). However, these methodologies
predominantly target generic improvements in model performance, overlooking the intricacies of
individual user satisfaction and preference. Our research takes distinct direction, focusing on
understanding edits across a variety of examples to study user-level preferences, with a practical goal of aligning the agent to individual preferences.
15

---

B
Additional Details
Dataset Examples.
We list links to dataset sources for our user-provided context in Table 4
GPT-4 User's Edits on summarization in Table 5
We list examples of OUR GPT-4 user's edits with different latent preference
GPT-4 User Templates.
Prompt templates used by our GPT-4 user are provided in Table 6
Baseline Hyperparameters. For E-then-e LPI and Continual LPI we set Te = 5. For ICL-edit baselines, we experimented with different values of k, and report our best results with 5.
CIPHER Templates.
Prompt templates used by CIPHER are provided in Table 7
ICL-edit Templates.
Prompt templates used by ICL-edit baseline provided in Table
CoT-edit Templates.
Prompt templates used by CoT-edit baseline are provided in Table 9
C Additional Analysis
Evaluating Normalized Edit Cost. The cumulative user edit cost measures the total effort of the user but is susceptible to outlier examples, the edit distance for given round is potentially unbounded. Therefore, we also compute a normalized edit distance Dedit (W), v/)/lye by dividing the edit distance by max [b]]]. i.e. the max length of the agent output user revised text. As Levenshtein distance (3tt y/) is upper bounded by max [yo]. ///]- therefore, the normalized cost is most 1. Figure 4 reports normalized cost over rounds for the top 3 methods. We notice that for all variants of CIPHER for the summarization task, and for CIPHER-5- --- for the email writing task, the normalized cost decreases notably as training progresses indicating learning. As the cost is normalized by the response length, even a small decrease can lead to a significant reduction in the number of tokens edited.
Detailed Expense.
We list detailed computational expense of different methods in Table 10
Failure Case Analysis. CIPHER notably reduces the edit cost and learns useful preference, however, significant gaps to the oracle method remain, especially in the summarization task. We manually analyze failure cases summarization task with the best performing method CIPHER-5- MPNET. Table iin the Appendix reports the summary and example our findings, categorized as preference inference from output-revision pair, consolidation of inferred preferences, and retrieval. In brief, the most common type of failure on the preference inference step given the agent output and user revision. For example, the agent often misses the exact keyword for brief short sentences, and sometimes struggles with inferring the second-persse persoo narrative aspect.
Retrieval Accuracy. We calculate retrieval accuracy for CIPHER as the fraction of all retrieved contexts that are of the same document type the currently given context across all seeds and time steps. We report the results in Table 12] We find that the retrieval accuracy is higher on the summarization task than email writing. and using MPNET typically performs better than using Bert to encode context.
Survey Details. We did a small survey with several participants recruited from our personal network. The instructions for the two tasks are as follows:
1. Task 1 instruction:
"You' re asked to compare 2 pieces of writing in terms of satisfaction towards certain preference. There are 15 pairs to compare in total.". For specific example, we ask "Assume that the writing style you prefer is <preference>, (e.g., you want to quickly gel main opinions from a movie review) Which piece of writing below you like better". We replace <preference> with the given preference.
16

---

Table 4: Link each source dataset, from which we randomly sample examples as the user-provte user-provided context in our tasks.
Data Source
Link Example
CNN Daily Mail (See et al. 2017)
SLF5K (Stiennon et al. 2020)
Wikidump (Foundation 2022
Arxiv (Clement ar. 2019)
MMDb (Maas
2011)
Paper Tweet [ar 2022)
htte:///mmggiiiiferrrr//sttttttt tweet
Elsevier OA CC-By (Kershaw &
ttpp://iingaggeeee.o///atttttt/p ott.://eeerorreeeeoeeeeeeeeeee oa- Cc-
Koeling 2020)
Figure 4: Normalized cost of CIPHER over time, binned per 20 rounds show the trend (average across 3 seeds). In the legend, -k means with top k retrieved examples, -B for BERT, and -M for
MPNET.
Summarization
Email Writing
Normalized Cost
Normalized Cost
0.6
CIPHER-5-B
0.4
CIPHER-I-M
0.4
CIPHER-S-M
Oraele
0.2
0.2
40
80
120 Round
160
200
40
80
120
160
200
Round
2. Task instruction: "This study is simulation of how you use AI writing assistants. There are 20 pieces of text that you need to review. You will be given an assumption of the preferred writing style, and you can edit the piece of writing it doesn't satisfy the given preference. If the writing aligns with the specified style, please leave no edits! Please treat each writing as independent piece, even though some writings are based on the same article. For example, when reviewing the writing and making edits, do not bring in the knowledge you learned from another piece of writing.
D Broader Impact Statement
Our work provides way to develop language agents that learn from user edits. Our work is part of broader effort to release language-basee based agents and consequently shares all risks associated with deploying language agents such as hallucination bias innate in these models. We advocate for caution and thorough testing in releasing language agents. Further, we strongly emphasize seeking the permission of users before releasing language agents that learn from their edits. Specially, users should have the option to take back their consent at any time.
17

---

Table Examples of user edits with different latent preference on summarization. The agent output column shows the agent generation without conditioning on any preference. The latent preference for the user edits is bold and parenthesis. We truncate some examples due to limited space.
Agent Output
User Edits
Leon Fenner, the suspected gunman in a $2 million diamond heist at Watch Standard Jewelry in New York City, was arrested in Maryland. The robbery occurred on November 11 and resulted in the in- jury of the store owner's 's elderly father. Fenner's accomplice, Rondu Frisby, was a friend of the store owner and was apprehended week after the heist. The robbery took place during the city's 's Veteran's Day Parade, leading to a block shutdown. Fenner and Frisby stole numerous jewels and 20 luxury watches, but were caught surveillance cameras. Fenner is set to appear before a foderal court in Maryland.
(targeted to young children, storyielling, short sentences, playful language, interactive, posi- tive)
Once upon a time in the big, bustling city of New York, during parade filled with music and march- ing on Veteran's 's Day, something sneaky happened at place called Watch Standard Jewelry. This wasn't I just any store; was a treasure chest full sparkling diamonds and shiny watches that could make your eyes twinkle like stars.
Now, two not-so-nice men, Lcon Fenner and his buddy Rondu Frisby, thought they could be sneaky and take these treasures without asking. On November 11, while everyone was watching the parade, they tiptoed into the store. But oh no! They accidentally hurt the store owner's dear old dad, which wasn't I very kind at all. (TRUNCATED]
The article describes situation where neigh- bor's cat, one of seven rescued by a "cat lady," has started frequenting the author's 's front yard after leaving its original home due to the presence of other cats. Over the past few months, the cat has become more consistent in its visits, meowing at the author and their family members for attention. The cat allows itself to be petted and shows af- fectionate behavior, such as rubbing against legs and headbutting hands, until is fed, After cating, the cat typically leaves but returns the evening. The author is seeking advice on how to handle the situation, expressing concern about feeding the cat because is not legally theirs and there could be potential ownership and legal issues.
(second person narrative, brief, show emotions, invoke personal reflection, immersive) Imagine yourself in a quaint neighhorhood, where the gentle purrs of feline friend have become a familiar part of your daily routine. This isn't 't just any cat-it's it's one of seven, charming escapee from the home of a local ""cat lady," "" seeking solace in the tranquility of your front yard. Over recent months, this whiskered wanderer has turned your doorstep into its personal stage, meowing for attention with persistence that tugs at your heartstrings.
Each encounter with this affectionate creature is dance of mutual admiration. You find yourself captivated by its soft fur and the way it eagerly rubs against your legs, headbutting your hands in a silent plea for affection. /TRUNCATED]
The Puerto Plata Province is a northern province in
(bullet points, parallel structure, brief)
the Dominican Republic. known for its beautiful
Location: Northern Dominican Republic
beaches and popular tourist destination since
Capital: San Felipe de Puerto Plata
the late 1990s. Its capital is San Felipe de Puerto Plata, named after the silver clouds Columbus saw on a mountain nearby in 1493. The city of Puerto Plata was founded in 1502.
Founding: 1502 by Europcans
Historical site: La Isabela (1493, first European town in the Americas)
1605: Destruction by Governor Antonio de Oso- rio to stop illegal trade
Historically, the first European town in the Ameri- cas, La Isabela, was established by Columbus near Puerto Plata in 1493. In 1605, the governor An- tonio de Osorio destroyed Puerto Plata and other towns to curb illegal trade, leading to the formation of new settlements.
Status: Became Maritime District in 1865, recog- nized as province in 1907
Expansion: Added more municipalities, latest in 2007
Borders: Atlantic Ocean, Espaillat, Santiago, Valverde, Monte Cristi provinces [TRUNCATED]
Puerto Plata became a Maritime District in 1865 and was later recognized as province in the 1907 constitution. Over time, it expanded to include more municipalities and municipal districts, with the latest additions in 2007. [TRUNCATED]
18

---

The article discusses the challenge of detecting and classifying stuttered speech, which affects many people but has been subject to limited research. The authors present new model that differs from most existing methods, which typically use lan- guage models for stutter identification. Instead, their model is based on acoustic features, which allows it identify various kinds of stuttering dis- fluencies without relying on speech recognition technology. The proposed model combines a deep residual network with bidirectional long short-term memory (LSTM) layers to classify different stutter types. It has been shown to significantly outper- form current leading methods, reducing aver- age miss rate to 10.03%, which is a 27% improve- ment over the state-of of-the-art.
(tweet style, simple English, inquisitive, skillful foreshadowing, with emojis)
Ever struggled with stuttering or know someone who does? Research detecting stutters hasn't been huge, but guess what? new study is changing the game! Instead of just looking at words, this smart model listens to how things are said. @ It's like detective for sounds, using cool tech called deep learning and memory networks
spot different stutter types. And it's crushing it, beating old methods by 27%! Missed stutters?
Down to just 10%! This could be big deal for speech therapy! Innovation
#Stuttering # SpeechTech #
The article discusses movie that begins promis- ingly but later becomes confusing due to poorly depieted portrayal of ancient black magic rituals. The male lead delivers strong performance de- spite his character's s unfavorable outcome in the climax. The movie is compared to a mix of "Boomerang" and "Extremities."
(question answering style, direct, concise) What is the general sentiment of the article about the movie?
The article suggests that the movie starts well but eventually becomes muddled. The depiction of ancient black magic rituals is criticized for being confusing.
How does the article deseribe the male lead's per- formance?
The male lead's performance is described as strong, even though his character does not have a favorable outcome in the climax.
What films the movie compared to in article? The movie is compared to "" "Boomerang" and ""Extremities"" in the article.
Table 6: Prompt templates for the AI user. The first step is to prompt the user for yes/no answer regarding satisfaction. If the answer is no, the second step is to ask the user edit the agent output according the latent preference. the answer is yes, the agent output receives edits.
Summarization
Email Writing
Step 1 Article: (user-provided article)
Notes: (user-providee notes)
Summary: (agent-generated summary) Is the above summary of the above article good for person who would love to use following style: (latent user preference]? Please answer yes or no.
Email:
email)
Is the above email based the above notes good for a user who wants the following style: (latent
user preference]? ? Please answer yes or no.
Step 2
summary) Please revise the above summary of an article
Summary: (agent-generall
Email: aagent-generated email)
Assume that you prefer (latent user preference). Please revise the above email to meet your style.
to meet your style: [latent user preference).
19

---

Table 7: Prompt templates for CIPHER.
Summarization
Email Writing
Task prompt
Article: (user-provided article)
Notes: (user-provided notes)
conditioned
Assume that you need to summarize the above who prefers the following style: [inferred user preference). Please write summary of ahove article to address those specified preferences.
These notes are written by user who prefers the following style of emails: (inferred user preference). Please write short email based on the above notes address those specified preferences.
inferred
article for
preference
[line 5| in AH
gorithm
Prompt to in- for user pref-
Original summary an article: (agent- generated summary)
Original email: (agent-generated email) Revised email: {user revision]
erence based on revision
Revised summary user: {user revision) Based on the edits and revision by this user on the original summary in the above examples, what do you find about this user's generic pref- erence in terms of writing style and formatting? ? Please answer in short phrase and only mend those preferences that are widely used.
Based the edits and revision by this user the original email in the above examples, what do you find about this user's generic preference in terms of writing style and formatting? Please answer in a short phrase and only recommend those preferences that are widely used.
(line 11 AP
gorithm
Prompt
List of user preferences successfully being used to generate summaries similar documents: finferred preference in a retrieved example) (inferred preference in retrieved example] ...
List of user preferences successfully heing used to generate emails a similar kind: (inferred preference in a retrieved example) (inferred preference in a retrieved example)
consolidate
inferred
preferences
from history
[line in Al gorithm
Based on the the above examples, please come up with short phrase with the most represented summarization preferences of the user.
Based on above examples, please come up with short phrase with the most represented writing preferences of this user.
Table 8: Prompt templates for the ICL-edit baseline.
Summarization
Email Writing
Prompt with retrieved edit examples
Original summary of an article: (agent- generated summary in retrieved example)
Original summary of an article: (agent- generated summary in retrieved example) Revised summary by user: (user revision in retrieved example)
Revised summary
(user revision in retrieved example]
Original summary an article: (agent- generated summary in retrieved example) Revised summary by user: (user revision in retrieved example)
Original summary of an article: (agent- generated summary in refrieved example) Revised summary by a user: (user revision in retrieved example)
Article: (user-provided article)
Notes: (user-provided notes)
Based on the edits and revision by this user on the original summary in the above examples, Please summarize the above article:
Based on the edits and revision by this user the original email in the above examples, Please write an email based the ahove notes for this user:
20

---

Table 9: Prompt templates for the CoT-edit baseline.
Summarization
Email Writing
Prompt with retrieved user examples
Original summary an article: (agent-
Original summary of an article: (agent- generated summary in retrieved example) Revised summary by user: {user revision in retrieved example)
generated summary in retrieved example)
edit
Revised summary
{user revision in retrieved example]
Original summary an article: (agent- generated summary in a retrieved example) Revised summary by user: {user revision in retrieved example]
Original summary of an article: (agent- generated summary in a retrieved example) Revised summary by a user: {user revision in retrieved example)
FIRST, come up with short phrases that explain edits made by the user in the above examples, to show this user's writing preferenec. SECOND, line, summarize the above given arti- cle using the inferred preference. Your response MUST FOLLOW THE FOLLOWING FOR- MAT: PREFERENCE: <your inferred prefer- list of short phrases> RESULT: <your summary the given article>
Notes: {user-proee previ-povided notes)
FIRST, come with short phrases that explain edits made by the user in the above examples, show this user's writing preference. SEC- OND, on new line, write an email based on the above notes using the inferred preference. Your response MUST FOLLOW THE FOL- LOWING FORMAT: PREFERENCE: <your inferred preference as list of short phrases> RESULT:: <your email based on the given notes>
Table Expense of different methods: number of BPE tokens terms of input, output and total. Each number is the average across 3 runs (unit is -10%).
Method
Summarization Output
Email Writing
Input
Total
Input
Output
Total
Oracle Preference
1.14
0.53
1.67
0.91
0.71
1.62
No Learning
1.06
0.44
1.50
0.85
0.80
E-then-e LPI
1.16
0.83
1.99
0.94
0.79
1.73
Continual LPI
8.14
0.75
8.89
7.89
0.73
8.63
CCL-edit-5-MPNE
7.35
0.65
8.00
11.05
1.06
12.12
7.32
0,64
7.96
10.51
1.03
11.55
CoT-edit-5MMPNET
6.23
0.59
6.82
7.67
0.79
8.47
CoT-cdit-5-BER
6.34
0.58
6.92
7.48
0.78
8.26
CIPHER-I-MPNER
0.72
2.74
1,21
0.73
CIPHER-S-MPP
2.27
0.73
3.00
1.44
0.64
2.09
CIPHER-1 --ERT
2.10
0.71
2.81
1.27
0.73
1.99
2.32
0.71
3.03
1.48
0.73
2.22
21

---

Table 11: Summary of failure cases on summarization task with HUPHER---PPNR
Type of Failures
Summary
Examples
Preference inference based an output- revision pair (f+)
1) Not totally wrong but in- sufficient, ie. the inferred preference only captures few aspects of user's s latent preference. This is most common for news articles and Reddit posts, for which the user shows nuanced pref- crence for several aspects. 2) Sometimes fail to in- fer some important aspects, even though the user edits clearly show such prefer- ence.
The dominant missing aspect is brief or short sen- tences across different context, although the agent can infer keywords such as concise. For news article context, the agent tends to infer the prefer- ence keyword whimsical. The agent has difficulty to infer subtle aspects. including invoke personal reflection, immersive, positive, parallel structure, inquisitive, skillful foreshadewing.
(the most common fail-
ure type)
The agent often could not infer second-person nar- rative. For question answering style, the agent occasionally only learns consistent format.
Consolidation of induced preferences from retrieved interac- tions (fa)
Overall, this step can cap- ture the majority preference relatively well, although it tends result more gen- eral preference compared to the retrieved ones.
When both specific phrase second-persond narra- tive and general phrase narrative or narration oc- cur in retrieved examples, the agent often chooses to give final preference not including the second- person perspective aspect.
Retrieval of historical examples relevant to the given context
The retricval part in general works reasonably well, with more than half of the trieved example being truly relevant to the given con- text. Note that one incor- rect retrieved example typ- ically does not affect the performance, as we instruct the agent to only use the most represented preference keywords among all five re- trieved examples.
The agent sometimes retrieves wrong examples for Wikipedia context when its topic very relates to other context, c.g. wrongly retrieving past ex- amples
articles and movie reviews when the topic in given Wikipedia context relates these domains.
Table 12: We report retrieval accuracy as the percentage of total retrieved document representations across all time steps and seeds that are of the same document source type the context document for which they were retrieved. We use seeds. We retrieve 600 examples for k = 1 and 2970 examples for k = 5.
Method
Summarization
Email Writing
CIPHER-I-B
72.00
25.83
CIPHER-I-N 1-M
82.00
26.33
CIPHER-5-B CIPHER-5-M
65.79 76.33
26.57
25.45
22

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Discussed intro and experiment section.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We provide thorough failure case analysis in Appendix.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: NA.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yss]
Justification: We provide extensive details in the main paper and Appendix and will follow up with releasing code and linking it from the main paper.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use publicly available data in our paper which we cite in the paper. We will release code in the future and link it to the paper. We believe we provide enough details in the paper to reproduce all results in the paper including hyperparameters, prompt templates, etc.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide hyperparameter details in the Appendix.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: NA.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Answer: [Yes]
23

---

Justification: NA.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: NA.
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We have added broader impact statement the Appendix.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer:[NA]
Justification:
We do not release models or data with this paper.
Licenses existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all data sources and models that use.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: We will release the interactive environments in the future on Github and provide detailed README.
Justification: We do not provide a new dataset in this paper, but we do use common existing datasets that are available on Huggingface.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We provide survey details in the Appendix.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA.
24


Reviews:
[{"review_id": "hfRfHe2mro", "reviewer": "Reviewer_Kfzn", "summary": "The paper first proposes, PRELUDE, session level personalization for a writing assistance task. Here a model must learn natural language user preferences from edits made by the user to outputs generated a model during a session. Next the paper proposes an algorithm, CIPHER, which leverages an LLM to infer natural language preferences given edits from a (simulated GPT4) user. The approach is evaluated in two writing tasks: summarization and email-writing and compared to a host of reasonable baselines and evaluated for its ability to reduce cumulative edit costs by the user in a session and for CIPHER's ability to recognize the latent preference.", "questions": "- Line 131, \"We use cosine similarity for computing proximity..\": Are the context and preference concated to obtain an embedding for computing cosine similarity?\n- Please consider citing work on instruction induction (https://arxiv.org/abs/2205.10782 and follow ups) and discuss its differences to the proposed task and approach.\n- Do I understand correctly that there are 5 summarization (sub-)tasks and 4 email writing (sub-)tasks and this could be used to compute accuracy of a random predictor for the preference?\n- Table 2 or Fig 2: It would be meaningful to know the value of T at which maximum accuracy was reached for task/preference identification. Please consider reporting this in Table 2 or reporting the accuracy dynamics plotted against T.\n- ICL-edit descriptions: What does the notation y_z_t refer to? Over what corpus is this retrieval performed? Is it reasonable to think that ICL-edit would have higher performance at lower T and also need to make fewer calls to the LLM? If yes, please consider discussing this tradeoff, this would not be a slight on CIPHER.\n- Thinking of the task: Is it reasonable to think that some instances see a much higher edit distance than others - in these cases it would easier for the user to reword the prompt to generate a wholly new output rather than edit the output. Or an edit would amount to a complete re-write. This was noted in prior work on editing model outputs in work on speech transcription: https://www.cs.cmu.edu/~jbigham/pubs/pdfs/2016/asr-threshold.pdf. Did your exploration/analysis of model results spot such instances? I wonder if there is scope in future work for only inferring preferences from edits in some sessions and suggesting a prompt re-write in others. It would be meaningful to add a brief discussion of this to the paper.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed edit based natural language preference learning task is novel and represents meaningful contribution to the field. While the proposed approach, CIPHER is does not make any significant technical advances it represents a reasonable first approach to the task and is worth exploring at the outset.\n- The proposed task also represents a natural setting in writing assistance making it a meaningful problem to explore.\n- The paper is well written for the most part.\n- The evaluation setup in the paper is clever for evaluating the proposed task and could be useful to future work in this space.", "weaknesses": "- Some aspects of the proposed experimental setup seem quite simplistic or unrealistic for the proposed task: 1) It appears that the task of identifying \"user preference\" is mainly a reframing of task identification, the number of tasks in the two domains explored is small (5 summarization tasks, 4 email writing tasks). In realistic personalization scenarios I would expect a greater number of preferences. However, this drawback is likely a necessary part of initial work in this space and not a reason for rejection IMO. 2) The number of rounds in a session appears to be very high, T = 200. I understand this to be the number of times a user makes an edit to an output - its unclear if this is realistic or if one should expect CIPHER predictions for smaller T to be significantly degraded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their useful feedback. Please find our responses below in the order in the review.\n\n> The number of rounds in a session appears to be very high...\n\n**Evaluation at different T:** Since our setup is an online learning setting, we can evaluate the method and baseline at any value of “T” up to the value we tried. This is because there is no separate train and test steps. To do this, please look at Figure 2 and just choose any value of T on the x-axis upto to the maximum we tried (which is 200) and look at the cost values at the chosen x-axis. Note that we don’t see gains with a handful of examples since preference varies across contexts, and you need to see enough variety and then also learn the underlying preference before we can properly learn.\n\n> .... Are the context and preference concated to obtain an embedding for computing cosine similarity?\n\n**Cosine similarity computation:** The cosine similarity is computed only on context embeddings. Given the context in the current round, we compute its embedding and compare the cosine similarity of it against all context embeddings in the previous rounds. We then compute the closest $k$-previous context embeddings and retrieve the inferred preference associated with these context embeddings. The idea here is that we want to use the preference of similar contexts, and not necessarily just find similar past preferences.\n\nThe cosine similarity is computed based on two context embeddings: one is the embedding of the given context at the current round, and one is the embedding of context provided in past rounds (i.e. context in the history). This similarity computation is for retrieving the most relevant context in the history with respect to the given context.\n\n> Please consider citing work on instruction induction... \n\n**Instruction Induction:** We will be happy to cite and discuss the instruction induction paper and the follow ups.\n\n> this could be used to compute accuracy of a random predictor for the preference?....\n\n**Clarification on sub-tasks and evaluating preference prediction**: Both summarization and email writing have document from different sources (5 for summarization and 4 for email writing) which have different hidden user preference. This is an attempt to capture real world scenario where a user’s preference is different across context (e.g., writing email to a friend versus writing a conference review). The document source is hidden from the agent. Further, the documents, i.e., context, from different sources are shuffled and presented in a random order to the agent to avoid any temporal correlation. \n\nThe CIPHER algorithm (or any algorithm implementing the PRELUDE framework) infers a preference description for each context (document). Each document has a hidden user preference that is determined based on its source. We compute a 0-1 accuracy based on whether the BERTScore of the inferred preference and the true preference of the document source is closer than that of the inferred preference and the true preference of any _other_ document source. Note that this score (called accuracy in Table 2) has no learned parameters (see Line 187-192 for more details). \n\n>  value of T at which maximum accuracy was reached...\n\n**Plotting accuracy vs $T$:** We would be happy to include this plot in the revision.\n\n> What does the notation $y_{z_t}$ refer to? Over what corpus is this retrieval performed?....\n\n**ICL-edit descriptions, source of retrieval, notation and tradeoff:** We want to clarify that all retrievals in this paper happen only on the agent’s past history. We do not use external sources for performing retrieval. This avoids mis-alignment errors arising due to the choice of retrieval corpus. Although using a separate retrieval corpus could be a nice extension.\n\nAs shown in Protocol 1 and 2, we have $T$ rounds of online interaction where in round $t$ we have the agent response as $y_t$. Therefore, $y_{z_l}$ refers to agent response in round $z_l$. We explain $z_l$ more below.\n\nFor ICL-edit baseline, in round t we find the $k$ closest past rounds based on cosine similarity between current round’s context and previous round context.. Thus, $z_1$, $z_2$, …, $z_k$ refers to the index of the previous round that we retrieve. Therefore, $y_{z_l}$ refers to agent response in round $z_l$ which is the $l^{th}$ retrieved past round.\n\n>  ...Is it reasonable to think that ICL-edit would have higher performance at lower T...\n\n**ICL-edit performance at lower T:** We can evaluate ICL-edit performance for any value of T in Figure 2 as explained in point 1 above. We can see that the gap between baselines in the beginning is small although CIPHER-5-M does slightly better than both ICL baselines. However, at the end CIPHER gets the best performance. \n\nICL-edit agent always makes a single LLM call across all round as we do not use LLM for retrieving. We instead use cosine similarity for retrieving examples. We create a prompt using the retrieved examples and the context and generate a response using the LLM.\n\n> ...in future work for only inferring preferences from edits in some sessions and suggesting a prompt re-write in others...\n\n**Mixing user edits and prompt rewrite:** This is a very interesting question.  In our experiments, we did not observe cases of complete re-write, and this could be either because GPT-4 LLM is good at generating a reasonable response, or that we need to work on more complex problems where LLMs struggle more before we see these cases. In real-world deployments, it is possible that there are cases where the entire output needs to be edited, and users can then choose other feedback mechanism like prompt re-writing, or language feedback. We would add a discussion on this direction and learning from these heterogenous feedbacks is an important future work direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sZA35G0ijx", "reviewer": "Reviewer_U3Kc", "summary": "The paper discusses interactive learning of LLM-based language agents based on user edits on the agent’s output. It first proposes the framework that infers a description of the user’s latent preferences based on historical edit data, and then uses an LLM to infer these user preferences. The proposed solution is tested on two tasks involving interactive environments: summarization and email writing.", "questions": "What's your future plan to enhance the quality of the learned preference?", "limitations": "The authors have adequately addressed the limitations", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written, and the proposed algorithm/framework is well-illustrated. It also discusses an important topic: the interactive learning of LLM-based agents. Additionally, the experiments are comprehensive, including discussions on the qualitative analysis of learned preferences, human evaluation, failure case analysis.", "weaknesses": "The paper leverages LLM to learn user preferences, which is sometimes not very reliable. Additionally, the retrieval of historical examples might not be relevant to the given context. I believe the authors have recognized these drawbacks and mentioned them in the limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their useful feedback. Please find our responses below in the order of the review.\n\nThere are complementary improvements happening in the general field of LLM prompting, inference, and planning. Some of the future work directions in which we can incorporate these ideas to improve CIPHER are as follows:\n\n- CIPHER can be used with any language agent which might in itself can be a complex system (e.g., using multiple LLMs or a separate memory). In this paper, we used a simple language agent that does greedy decoding of an LLM given a prompt. Going ahead, experimenting with more complex agents especially those focused on planning and reasoning is a useful direction and can help further improve the learning. \n\n- Another direction is to improve the quality of preference as suggested by the review. One way to do this is to learn a retrieval model instead of relying on static BERT and MPNet embeddings. Existing approaches that learn a retrieval model can be relevant for this [e.g., Mysore et al., 2023].\n\n- Another direction is that instead of using cosine similarity, we can learn a dense model that takes both contexts as input and predicts a similarity score. These dense models can better capture similarity although they are more expensive to use. A hybrid approach can also be useful to balance these trade-offs, where we use cosine similarity to narrow down the choices, and then use a dense model over these choices. These retrieval methods can be directly incorporated into Algorithm 1 by simply replacing Line 3 with these methods.\n\n\n### Reference\n- PEARL: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers, Mysore et al. 2023 [https://arxiv.org/pdf/2311.09180](https://arxiv.org/pdf/2311.09180).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XKL1Z5pqf9", "reviewer": "Reviewer_EkvH", "summary": "This paper investigates interactive LLM alignment by analyzing user edits to an agent's responses. The proposed framework, PRELUDE, enhances the agent's alignment with user preferences without extensive fine-tuning, thus avoiding high costs and potential performance degradation on other tasks. PRELUDE infers user preferences from historical edits and uses them to guide response generation. The study introduces CIPHER, an algorithm using a LLM to predict user preferences based on context and past edits. Tested in summarization and email writing scenarios, CIPHER demonstrates good performance over other methods by reducing edit distances and computational costs, while maintaining user preference alignment.", "questions": "1. What is (and how to decide) the granularity of each edit? Take the example in Figure 1 for instance, there are lots of edits on $y_t$ to obtain the $y_t'$. Are you merging all edits together? If yes, then the value $T$ in Protocol 1 is simply 1? But this seems contradicts to the statement in line 165 where the $T=200$. So are you treating each single modification as one edit? More discussions on this part is needed.\n\n2. What is included in the context $x_t$ when $t>1$ in Protocol 2 and Algorithm 1? \n\n3. What is the \"Accuracy\" metric in Table 2? It seems this metric is not included section 4.1?", "limitations": "The author does provide some failure case analysis in Appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Study an interesting and important problem\n2. The proposed framework is overall intuition and reasonable\n3. The paper presentation is clear and very accessible", "weaknesses": "No significant weaknesses but some method design choices can be further elaborated (better motivated). See below questions for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their useful feedback. Please find our responses below in the order in the review.\n\n> What is (and how to decide) the granularity of each edit? .... So are you treating each single modification as one edit? More discussions on this part is needed.\n\n**Clarification on edits:** Each round represents a single query from the user to the LLM agent (e.g., the writing assistant). Figure 1 shows a single round in detail. The LLM agent is given a context and has to generate a response. In Figure 1, this context is ```Article: {user-provided article} Please summarize the above article.``` where ```{user-provided article}``` is replaced by the given article. The user then makes edits to this summary as shown in Figure 1. The edits here represent all token deletions, additions, substitutions, etc. that the user makes to the given response. In the next round, the user will ask the LLM agent about very likely a different article and may have a different request. In our experiments, we evaluate the agent on 200 rounds where each round contains a different context.  Specifically, for the email writing example, there is a request to summarize 200 separate documents, one document per round. Importantly, note that different rounds are not different edits to the same document!\n\nNote that our setup is an online learning setup where the agent is evaluated on each round and there is no separate train/test split. This is because, in real-world personalization tasks, there is no separate train and test phase.\n\n> What is included in the context $x_t$ when $t>1$ in Protocol 2 and Algorithm 1?\n\n**Content of the context:** A context includes everything that is given to the LLM agent as input. Suppose we are in a writing application and there is a field where the user can write a query to an LLM agent. Then the context will include the user query along with content on the screen of the application. The context may include other things such as whatever the application knows about the user (e.g, their calendar, etc.). Figure 1 illustrates this setting where the context is ```Article: {user-provided article} Please summarize the above article.``` where ```{user-provided article}``` is replaced by the given article. \n\n> What is the \"Accuracy\" metric in Table 2? It seems this metric is not included section 4.1?\n\n**Accuracy metric.** The accuracy metric measures the accuracy of the inferred preference. This metric is defined on Line 187-194 but we didnt specify the name “accuracy” in the text that we used in Table 2. We apologize for this confusion and we will state it clearly in the revision. \n\nIn summary, each context contains a document that comes from one of the $\\mathcal{S}$ document sources. Each document source has a unique user preference. This is designed to capture the real-world phenomenon where users have different preferences depending upon what they are doing (e.g., writing an email to a friend or writing reviews for a conference). We evaluate a 0-1 score depending upon whether the BERTScore similarity of an inferred preference for document source $d \\in \\mathcal{S}$ is closer to the true preference of $d$, than to the true preference of any other document source $d' \\ne d$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N3IYTVuLfN", "reviewer": "Reviewer_YQaz", "summary": "In this paper, the author formulates a new task that the user may want to edit the agent's response to make its later responses more personalized. The author proposes a method called PRELUDE to learn preference descriptions of users from users' previous edited contexts in an interactive framework. The author also proposes the CIPHER method to consolidate induced preferences with retrieval from users' historical edits. To verify the effectiveness of the proposed method, the author conducts experiments in two practical environments.", "questions": "See \"Weakness\" above. If the author could address my concerns, I'm willing to improve my rating.", "limitations": "The author provides an analysis of \"Broad Impact Statement\" in the Appendix D.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed a new task of user editing during the interaction process between the user and agent.\n2. The methods are intuitive and easy to follow.\n3. Many experiments are conducted to verify the effectiveness of the proposed methods.", "weaknesses": "1. In practice, will someone take a lot of time to give feedback to the agent? It seems that users may refuse to provide revised responses for agents to get personalized responses. Maybe there can be some ways to obtain users' preferences in an implicit manner.\n2. Can the cost function with Levenshtein edit distance currently reflect the gap in user preference? It seems the user preference belongs to the semantic space, while the Levenshtein edit distance is just comparing their differences in token sequences.\n3. The embedding methods it utilizes in the process of converting historical text into vector representations do not have a fine-tuning process. So it is a pre-trained language model with fewer parameters, but there could be a critical problem in terms of retrieval gaps. I think the most significant factor that determines which piece should be retrieved, is the user preference. However, it seems that most pre-trained language models mainly embed and index texts according to their contents, rather than the styles like user preferences. Therefore, I think there may exist a large gap in the process of retrieval.\n4. The user simulator that is implemented by GPT-4 should also be evaluated to verify it can align with humans in the real world.\n5. The matrix of accuracy requires a classification model, but the author does not mention it in the paper.\n6. The author does not provide open access to the source code and data, but the author chooses \"YES\" in checklist 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their useful feedback. Please find our responses below in the order of the review. We start with an important misunderstanding.\n\n> In practice, will someone take a lot of time to give feedback to the agent? ...\n\n**Major Clarification on the naturalness of user edits:** The main motivation for using user edits is that they are naturally generated as users do edits for their own natural needs and not because we ask them or pay them to do so. This is different from comparison-based preferences that are typically explicitly elicited from annotators through crowd-sourcing. \n\nTo understand this, let’s imagine a writing application that has an LLM agent as a writing assistant. At any given time, a user may query the LLM agent to do something, e.g., _“summarize this essay”_ say for tweeting about this essay. The LLM agent has access to everything that is on the application window (e.g., the essay) and can use it to generate a response. Once the LLM agent generates a response, the user may find that it is close to what they want but not quite fully right. So, they might make edits to it and they do so because they need the perfect response **for their own goal** and not because we ask them or because they care about improving the application. As they are doing these edits in the writing application, the LLM agent **has free access to these user edits** which we can use to improve the LLM agent.\n\nWe mentioned this in the introduction in Lines 30-31 _(“a natural feedback….user edits…for their own final use”)_. In fact, our setting has an important feature that **“every natural use of the agent yields an edit feedback for learning”**, leading to never-ending learning. We state this in Line 51. We will emphasize this more in the revision.\n\nAnother useful thing about our setting is that we frame this as an online learning problem where there is no separate train or test setting, and the agent is evaluated on every query. This makes it directly applicable to the writing assistant example above where performance on each query matters.\n\n> “Can the cost function with Levenshtein edit distance currently reflect the gap in user preference?”\n\n**Purpose of edit distance:** We want to minimize the user’s effort that arises due to edits they perform. They do these edits because the LLM response isn’t ideal. But after some time, we want them to perform fewer edits and ideally no edits. We use Levenshtein edit distance to measure the “cost of user effort”. If the edit distance is high, then the user ends up doing more work and so the agent should be penalized more. Therefore, **edit distance serves as our primary metric of evaluation as it directly evaluates user effort.** \n\nNow with the PRELUDE framework, we hypothesize that edits happen because of latent user preferences that are not expressed in the user prompt, and which if we knew about we could minimize the edits. Thus, learning the right preference is not the end goal but is a way to minimize the user effort as reflected in edit distance. That said, learned preferences have the added advantage of also providing interpretability.\n\nOur experiments show that our hypothesis is indeed true as the Oracle baseline that uses the true preference achieves minimal edit distance in Table 2. However, it is possible that there are cases where the learned preference is close to the true preference but where edits are higher, or vice versa. We are happy to add a correlation plot in the revision between “accuracy” which measures the preference accuracy and “edit distance” across all 200 rounds to visualize this.\n\n> The embedding methods .... do not have a fine-tuning process.... \n\n**Fine-tuning Embedding Method.** Our main algorithm CIPHER is compatible with several complementary advances happening in the field of LLM prompting. For one, we can directly use a better retrieval method instead of the cosine-similarity with BERT/MPNET embeddings that we use. This can be done by simply replacing Line 3 in Algorithm 1 with a different retrieval mechanism. Our main goal in this paper is not to exhaustively evaluate all these approaches but instead, _initiate the line of improving LLMs with user-edits and doing so by learning latent preference descriptions_. We believe improving retrieval using learned embedding methods or using more complex language agents are all interesting future work directions. Specifically, we believe approaches such as Mysore et al., 2023 (cited in the paper) that learn the retrieval function can be relevant in our case. \n\n> \"The user simulator .... it can align with humans in the real world.”\n\n**Alignment with real user:** We provide indirect evidence of this alignment in the pilot study in Section 4.4 which shows that the ordering of models in Table 2 that learn from GPT-4 user is the same when evaluated by human win-rate. A large-scale study with real users can answer this more conclusively and is a natural future work direction.\n\n> “The matrix of accuracy ...does not mention it in the paper.”\n\n**Accuracy metric:** The accuracy metric is defined on Lines 187-194. We use BERTScore function to compute this. We will revise the text to clarify that the metric “accuracy” in Table 2 refers to this particular metric and we apologize for the confusion.\n\n> The author does not provide open access to the source code and data, but the author chooses \"YES\" in checklist 5.\n\n**Code:** We apologize for the error in the checklist. We will fix this along with any other errors. We will release the code with the camera ready and are happy to provide a copy of the code at this dropbox [link](https://www.dropbox.com/scl/fi/ydj9nx4bpbchqmf9c5hly/aligning_llm_agents.zip?rlkey=9gj5t8bf8q4zap65hhc83n3yg&st=zbejafp8&dl=0)\n\nWe have already linked the existing datasets we use in Table 4 in Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper investigates interactive learning of language agents by analyzing user edits to an agent's responses, and proposes a framework, PRELUDE, to learn natural language user preferences from edits made by the user to outputs. An algorithm, CIPHER, is introduced to infer natural language preferences given edits from a (simulated GPT4) user. The approach is evaluated in two writing tasks: summarization and email-writing, and compared to reasonable baselines. 

Reviewers agree on the following:

Strength:
- The experiments are comprehensive. 
- The paper addresses an important problem in the interactive learning of LLM-based agents, and the proposed framework is overall intuitive and reasonable.
- The paper is well-written and easy to follow, and the proposed algorithm/framework is well-illustrated. 

Weakness:
- Oversimplified setup: the latent preferences consists of 5 for summarization tasks and 4 for email writing tasks. In realistic personalization scenarios there would be a greater number of personalized preferences.
- The GPT-4 implemented user simulator should be evaluated to ensure it aligns with real human behavior.

Overall the reviewers have a similar opinion that the framework is novel and experiments are comprehensive, therefore I am leaning towards an accept.
Decision: Accept (poster)