Aligning LLMs by Predicting Preferences from User Writing Samples
Authors: Stéphane Aroca-Ouellette
Natalie Mackraz
Barry-John Theobald
Katherine Metcalf
Venue: ICML 2025 poster (2025)
Keywords: NLP
LLM
preference learning
user demonstrations
benchmark
Primary area: deep_learning->large_language_models
TL;DR: PREDICT is a method that uses LLMs to infer user preferences as natural language descriptions for conditioning and guiding agent behaviors based on observations of user behaviors.

Abstract: Accommodating human preferences is essential for creating aligned LLM agents that deliver personalized and effective interactions. Recent work has shown the potential for LLMs acting as writing agents to infer a description of user preferences. Agent alignment then comes from conditioning on the inferred preference description. However, existing methods often produce generic preference descriptions that fail to capture the unique and individualized nature of human preferences. This paper introduces PROSE, a method designed to enhance the precision of preference descriptions inferred from user writing samples. PROSE incorporates two key elements: (1) iterative refinement of inferred preferences, and (2) verification of inferred preferences across multiple user writing samples. We evaluate PROSE with several LLMs (i.e., Qwen2.5 7B and 72B Instruct, GPT-mini, and GPT-4o) on a summarization and an email writing task. We find that PROSE more accurately infers nuanced human preferences, improving the quality of the writing agent's generations over CIPHER (a state-of-the-art method for inferring preferences) by 33\%. Lastly, we demonstrate that ICL and PROSE are complementary methods, and combining them provides up to a 9\% improvement over ICL alone. Code: https://github.com/apple/ml-predict

Aligning LLMs by Predicting Preferences from User Writing Samples
Stéphane Aroca-Ouellette 12
Natalie Mackraz
Barry-John Theobald 3 Katherine Metcalf
Abstract
great idea" or "summarize this new article for share share with my friends". As the writing style learned by an LLM during pretraining is generic, it may not match the user's preferred writing style and voice (Chakrabarty et al., 2024; Santurkar et al., 2023), leading to outputs that feel impersonal, misaligned, or requiring extensive editing.
Accommodating human preferences is essential for creating aligned LLM agents that deliver personalized and effective interactions. Recent work has shown the potential for LLMs acting writing agents to infer description of user preferences. Agent alignment then comes from conditioning on the inferred preference descrip- tion. However, existing methods often produce generic preference descriptions that fail to cap- ture the unique and individualized nature hu- man preferences. This paper introduces PROSE, method designed to enhance the precision of preference descriptions inferred from user writ- ing samples. PROSE incorporates two key ele- ments: (1) iterative refinement of inferred prefer- ences, and (2) verification of inferred preferences across multiple user writing samples. We evalu- ate PROSE with several LLMMs (i.e., Qwen2.5 7B and 72B Instruct, GPT-mini, and GPT-4o) summarization and an email writing task. We find that PROSE more accurately infers nu- anced human preferences, improving the qual- ity of the writing agent's generations over CI- PHER (a state-of-the-ar method for inferring preferences) by 33%. Lastly, we demonstrate that ICL and PROSE are complementary meth- ods, and combining them provides up to 9% improvement over ICL alone. Code: https: / //github... com wttp://pppp/m/m//eeiiie..
Existing approaches learn preferences rely on preference rankings (RLHF) (Ziegler al., 2019; Rafailov al., 2024), demonstrations (Ouyang et al., 2022; Shaikh al., 2024), prompting (Zhou et al., 2022), and user edits (Gao et al., 2024). However, methods such as RLHF and SFT (on user demonstrations) require a large number of samples, and do not learn the preferences a form users can interpret or in- teract with. In-context learning (ICL) from user demonstra- tions does learn from small number of user demonstrations, but lacks interpretability and offers limited control to the user, and prompting approaches require the challenging task of identifying a high-quality prompt "Zamfirescu-Percci et al., 2023). Furthermore, methods that learn from user edits ignore data about user preferences and style that are available from observing how the user completes writing tasks on their own.
Gao et al. (2024) introduces CIPHER to establish the ben- efits of aligning LLM through prompting by learning description of user preferences compared to ICL conditioned on user demonstrations (i.e., needing fewer tokens, inter- pretable representation, and a modifiable representation). The preference description is learned from user edits on the assistant's generations. However, CIPHER does not enable the LLM to reflect on and refine its inferred preference de- scription, which limits the assistant's 's ability to adapt to user nuanced writing style.
1. Introduction
In this paper, we build on CIPHER and introduce PROSE (Preference Reasoning by Observing and Synthesizing Examples), a novel approach that leverages two key innova- tions to enhance the precision and efficacy of the preference description inferred from user demonstrations: (I) itera- tively refining the inferred description until the assistant's generations closely align with the user, and (2) verifying the inferred preferences across multiple user demonstrations.
People increasingly rely on LLM-powered AI Assistants to complete tasks on their behalf, such as creating written "write a professional email about the following
materials:
Work done during internship. Department of Computer Science, University of Colorado, Boulder, CO USA AAple, Cu- pertino, CA USA. Correspondence Stéphane Aroca-Ouellette <stephane. wstephane.aroca-ooe www.aarooollleeeeee ouelletee @ colorado.edu>, Katherine Metcalf <kmetcalf@ Axmetcaff@apple.coom.
The inferred description is used to condition the LLM to generate writing more aligned with the user.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We evaluate PROSE on PRELUDE (Gao al., 2024), the

---

PROSE: Inferring User Preference Deseriptions
assistive writing benchmark accompanying CIPHER, and identify several limitations. First, the ground truth prefer- ence sets often overlap, lack diversity, and match the default LLM behavior. Second, edits are performed only if the assistant generation is inadequate, meaning it is not possi- ble to distinguish between good and excellent generations. Lastly, PRELUDE relies on user edits as the learning signal, meaning the assistant's initial draft can limit the quality of the final writing sample. To address these limitations, we introduce novel assistive writing benchmark, PLUME (Preference Learning from User Emails and Memos).
circumvents these issues by learning from implicit user sig- nals, breaking down preferences into sub-components to generate tailored user-preferences, all without the need for fine-tuning
Preference-Conditionn Agents Combined preference in- ference and conditioning has recently gained traction, with the following three works most aligned with PROSE.
Peng et al. (2024) explores preference learning in quadrupedal mobile manipulation using an object detec- tion module to map image observations to text. An LLM then infers preferences by comparing pairs of trajectories. These preferences are in turn used to improve task align- ment with user preferences. Shashidhar et al. (2024) train preference inferring model that outputs a set of rules to use during generation, and demonstrate improved personal- ization set of writing tasks. Lastly, Gao et al. (2024) propose the PRELUDE environment, where an L.M learns writing style preferences in a collaborative authoring task. We discuss this work in detail in Section 4.
We systematically evaluate the benefits of PROSE on PLUME using four LLMs ranging in size and ability, and find that PROSE outperforms CIPHER by 33% (Gao et al., 2024). Additionally, we demonstrate PROSE can be com- bined with ICL to further improve over CIPHER by 47% and up to 9% over ICI. In all, our contributions are:
PROSE: A new method infer user preferences.
PLUME: An improved benchmark for preference in-
ference from user writing demonstrations.
These methods all rely on a single inference step, whereas our approach uses iterative refinement to learn more precise preferences, and preference verification across several user examples for robustness.
An in-depth ablation study on PROSE's s iterative refine- ment and consistency verification steps
An analysis comparing learning explicit preference descriptions and conditioning directly on in-context examples
3. PROSE
2. Related Work
PROSE aligns an Al writing assistant with a user's pref- erences Pa by learning a preference description paese that allows the assistant (an L.LM) to mimic the user's demon- strations wa, which are determined by p.. For example, learning that articles should be summarized in the style of an old timey radio broadcast.
Personalizing LLMs In natural language generation, prompting (Radford et al., 2019) and in-context learning (Brown et al., 2020) have proven effective methods for con- trolling the generation of text, especially in a preference- driven context (Sun al., 2023; 2024).
Each time the user gives the assistant a new task or provides a new task-description and demonstration pair (Jinsk; w,,,, following (Gao et al., 2024) PROSE retrieves up to three previously observed demonstrations relevant to the given task along with the preferences inferred from those demon- strations from its interaction memory. The retrieved prefer- ences are then aggregated to form the preference descrip- tion Prese using the prompt in Figure (Appendix F. which is used to condition the assistant during generation: Wa = generatete 1m, Tansk Pdese). If no demonstrations have been seen, the Al assistant not conditioned on any preferences, Wa generate (11m, Ttas).
Some prior approaches for adapting models to user pref- erences involve RLHF (Stiennon et al., 2020) and fine- tuning (Tan et al., 2024; Zhuang et al., 2024), which can be compute-intensive and inaccessible to some practition- ers without the budget or scale of needed data. To reduce data requirements, Shaikh et al. (2024) propose treating user demonstrations as implicitly preferred over all model outputs, allowing for more efficient preference modeling. Another line of work aims to minimize compute demands by identifying and selectively adjusting internal activations to steer model behavior (Li et al., 2023; Turner et al., 2024; Lindsey et al., 2025). While effective for promoting broad, predefined objectives- suchh as improving truthfulness or reducing toxicity-it it remains unclear how such techniques can generalize to individual users without explicit guidance. With the rise of L.MSs with strong instruction-followin ca- pabilities, methods like prompting to adapt to user's profile have become more popular (Shen et al., 2024; Salemi et al., 2024); however, these methods too often rely on explicit user feedback to optimize prompts (Lin et al., 2024). PROSE
If the Al assistant's generation, WA does not match the user's demonstration, w.. the inferred preference description plese is updated via iterative refinement (Section 3.1) steps and a preference consistency verification (Section 3.2) step- PROSE's contributions. Iterative refinement alternates be- tween updating the inferred preference description Pate by comparing the agent's generation, Wa, to the user's 's demon-
2

---

PROSE: Inferring User Preference Descriptions
1a
User Preferences (p.))
User Demonstrations
(w,)
Save the Demonstration
description (faas)
User Demonstration (N.)
4
2
Inferred Preference Descriptions (Pdeso)
Inferred Preference
Assistant
Components (p.)
LLM Generation (v,)
Iterative Refinement
Consistency Verification
LLM
( I S'terations)
Preference Description (Pdesc)
1b
Preference Aggregation
Interaction Memory (Wtakk Wp P,))
Interaction Memory (Atask Wi. P.l
Preference Update Prompt (Sec. 3.1) have prevt ously idensified this preference: Pdese
Breakdown Prompt (Sec. 3.2)
Consistency Verification Prompt (Sec. 3.2)
Validete the following preferences
infercedi following
Seeed Steekk wrotes
preference: Pdese
againat the Dollowing behaviors va
Rowsoer, ", differa Evan M4
set prefereccss. Forsat of preferences as uSaH list an vingle line and prefaced with "Weeerennsss:"
Instraction: Horma this preference
Instruction:
Doea the behavlor confilms or redtredict the preference? Select atrengiy contimn the preferance (+2)
the following:
set
Refline the preference descript fen by addingg sanovungg, updatinn the preferance in to better unitate the usor.
Instruction:
strongly contrachete the preference (-2)
Figure I: Overview of PROSE. (top) The user provides a task description and demonstration to PROSE, which executes iterative refinement and then consistency verification step. Iterative refinement updates the inferred preference description by generating writing sample conditioned on the current preference description, comparing the sample the user's demonstration, and updating the preference description to better describe the user's demonstration until the LLM's generations match the demonstration or a maximum number of iterations S' is reached. The description is then broken into a set of component parts, and each component's consistency with prior demonstrations is verified with LLM-as-a-Judge. (bottom) Example PROSE prompts (for full prompts see Appendix F.I).
stration, Wa, and rerunning generation conditioned on the updated p until either the maximum number of itera- updates to the
The process of generating Al assistant writing samples, com- paring to the user demonstrations, and updating the inferred preferences continues until either the candidate solutions exactly match the user's demonstrations, the preference de- scription is unchanged between subsequent update steps, or a maximum number of iteration steps is reached (S). Qual- itative examples of the consistency verification procedure are in Appendix F.5.
tive refinement steps (S) is reached inferred preference description are made. Consistency ver- ification breaks the final Pdese into preference components and prunes components that are not supported by previously seen demonstrations.
visualization of PROSE (top) and the prompt summaries (bottom) for each of its preference inference steps are pro- vided in Figure 1. The algorithm is provided in Appendix A, and the complete prompts are in Figure 8 (Appendix F.1)'.
3.2. Consistency Verification
After the preference description is improved through itera- tive refinement, each component of each preference descrip- tion in Pdese is verified against relevant, previously observed user demonstrations. The verification step removes prefer- ence components that were incorrectly inferred or are overly specific to single demonstration.
3.1. Iterative Refinement
To improve Paese. the LLM is prompted to compare and contrast Wa and wn and then modify Puese such that the modification reduces the difference between wa and ww: 16+1 generate(1 1m, Tupdate Pdeser Wuy Wa), where Tupdate = "Preffrrence Update Prompt" in Figure 1. The updated preference description is accumulated in Paese Pi -aae..-.,, Plaeeee ) where is the iterative refinement step.
Paese
Consistency verification operates on the component level (e.g. "use emojis", "use alliterations"). Therefore, the natu- ral language preference descriptions (e.g. "write a tweet with emojis and alliterations") produced by iterative refinement are first broken into components by prompting the LLM to convert the preference description into ordered set of preference components. The preference components are ag- gregated over all preference descriptions to help avoid over
PROSE then conditions the LLM on the updated preference description pat2 to generate a new writing sample w
1 codee coming soon!
3

---

PROSE: Inferring User Preference Deseriptions
fitting: P.- = U.fovd (generate 1m, Fbreakdown Pdese)); where 3'reakdown BBeekkooonn Prompt" in Figure 1.
PRELUDE
PLUME
Metric
Acc.
Sim.
Acc.
Sim.
Levenshtein dist
-0.43
-0.39
0.01
-0.11
PROSE verifies each preference component in Po against each of the relevant user demonstrations by prompting an LLM to assign a score Visure E [-2, 2] indicating how strongly the demonstration confirms the preference: Vsccore
PPCM
0.42
0.42
0.39
0.73
Table 1: Subset of Pearson correlation (pp,a) between pref- erence quality metrics and generation quality metrics across both the summarization and email tasks. Best correlation in each framework is bold. Sim. (Preference similarity) and PPCM (Per Preference- Coneronne- Match) are described in Section 4.2. Full results in Appendix C.I.
wall w..] 10 (generate(1 1m, Iverificationn Wi.PPeew )),
where Everiiication is *Consistency Verification Prompt" in Figure 1. If Uscore is below the specified threshold (t), the preference component is removed. The task description, user demonstration, and final preference components (Zusk: We, Pe) are then stored in PROSE's interactive memory. Qualitative examples of the consisteney verification procedure are in Appendix F.4.
curacy² -ndd and generation quality metric- I Levenshtein distance (Levenshtein, 1966) between the LLM generation and user edited generation. For a each summary and email topic, we generate the powerset of PRELUDE's ground truth preferences and create a population of agents. Each agent conditioned on a subset from the powerset and completes its assigned task for five seeds. The quality of the inferred preferences and of the resulting generations is measured according to PRELUDE's performance metrics. We calcu- late the Pearson correlation between each of PRELUDE's preference quality and generation quality metrics:
4. Assistive Writing Benchmark
4.1. PRELUDE
Gao et al. (2024) propose PRELUDE (PREference Learning from User's Direct Edits) to evaluate algorithms that infer preferences for assistive writing tasks. Success is defined as: (1) maximizing the quality of the inferred user's preferences and (2) minimizing the amount of work required by user to edit the generated text into an acceptable form.
Cov(P,G)
PP..G P
PRELUDE consists of two tasks: summarizing articles and writing emails from notes. Each task has a set of users, and each user has distinct preferences per summary and email topic (e.g., summarize an encyclopedia article versus news article). The summarization and email writing tasks have five and four users respectively. See Table 9 ( Appendix D) for the mapping between users, topics, and preferences.
where P denotes the measured preference quality and G denotes the measured generation quality. We report a subset of the results in Table (Full results in Appendix C.1).
The results, reported in Table 1, show a weak correlation (< 0.5) between PRELUDE's preference accuracy and Lev- enshtein distance metrics. The accuracy metric relies on the "highest" BERTScore, and therefore cannot differentiate par- tially correct preferences from perfectly correct preferences. Moreover, the Levenshtein distance varies substantially be- tween generations even when conditioned on the exact same preferences (an illustrative example is in Appendix 1. Gao et al. (2024) allude to this as motivation for their two- stage editing process, and when we compare the results to a version of PRELUDE where the user always generates sum- maries or emails directly from the article or notes instead of editing the agent's ' summary or email (PRELUDENNER))) we see a further drop in correlation. However, we propose addressing this issue using improved metrics.
To solve given task, the agent must write summary or email using the provided article or notes along with any preferences the agent has inferred. The user is then asked if the agent's generation is satisfactory based on the user's true preference. If the agent's 's generation is satisfactory, the agent accrues no penalty. If the agent's 's generation is not satisfactory, the user edits the agent's 's generation, and the agent is penalized based on the extent of the edits. The agent observes the user's edits to improve its inferred preferences.
We analyze PRELUDE and find that the (1) chosen metrics, (2) the editing process, and (3) the ground truth preferences are key limitations of the benchmark, that lead weak correlation between the quality of the inferred preferences and the quality of the generated writing.
The Editing Procedure Relying on binary label to indi- cate whether generation matches the user's preferences is inherently ambiguous. It is not possible to distinguish between generations that align with 65% versus 100% of preferences. Even if this ambiguity is resolved, genera-
Metric Correlation As the goal is to infer user preferences, the measure of the agent's generation quality (i.e., the user- bdst- penalty) must be highly correlated the quality of inferred preference. We measure the correlation between PRELUDE's preference quality metric - preference ac-
aa preference is correct if its BERTScore (Zhang* et al., 2020) with true preference set greater than the BERT with any other preference set.

---

PROSE: Inferring User Preference Descriptions
tions not selected for editing incur no cost and provide no incentive to further improve the quality of the inferred pref- erences. Lastly, the editing process unduly influences the user's writing, as demonstrated in Appendix E.2.
Similarity has a stronger correlation with each writing gener- ation metric than PRELUDE's accuracy metric. Looking at the generation quality metrics, Levenshtein distance consis- tently has the weakest correlation and PPCM the strongest. Notably, the pairing of Preference-Similarty (preference quality) and PPCM (generation quality) provides the highest correlation in every situation and are the primary metrics we report in PLUME.
Preference Sets We observe the following limitations with PRELUDE's preference sets: (1) certain preference com- ponents have minimal impact on the generated text, due to unclear definitions (e.g., "skillful foreshadowing") or similarity to default LLM behavior (e.g.. "clear"); (2) pref- erences are repeated across several task topics (e.g., "short", "brief", "concise" appear in four of five summarization pref- erence sets); and (3) there is a large variance in preference set complexities across users (e.g.. "targeted to young chil- dren, storytelling, short sentences, playful language, interac- tive, positive" vs. "question answering style"). PRELUDE's preferences are in Appendix D (Table 9)
The Editing Procedure In place of the editing, PLUME has the agent and user independently solve each task to (1) enable the agent to learn from every user example, unless the agent's generation exactly matches the user's; (2) remove ambiguity about whether a generation should be edited and incur a cost; (3) provides a smoother curve along which to evaluate different methods; and (4) prevents agents from influencing users.
Knowledge of Topics Instead of treating each task topic as distinct user, PRELUDE introduces the additional challenge of context awareness; each user has different preferences based on the task's topic. Therefore, prior to writing a summary or an email the agent must first identify the correct context, an orthogonal challenge inferring preferences.
Preference Sets PLUME reworks the preferences according to the following criteria: (1) each preference set contains an equal number of components; (2) within each task, prefer- ence sets have a shared structure; (3) as much as possible, preferences components are orthogonal to each other, avoid- ing overlapping preferences (e.g.. "write in the style of old- timey radio" and "use archaic language") or contradictory preferences (e.g.. "use emojis" and "use a formal tone"); and (4) preferences components do not follow the LLMs default behavior - i.e., generating an output conditioned on no preference should lead to a lower score than when generating on the preference component. PLUME's prefer- ences are in Appendix D (Table 9). We encourage future researchers to use PLUME with different preference sets to adjust difficulty or examine specific concepts.
4.2. PLUME
To address PRELUDE's limitations, we develop a new envi- ronment PLUME (Preference Learning from User Memos and Emails) based on same underlying tasks and topics as PRELUDE. As in (Gao et al., 2024), PLUME uses GPT-4o as a proxy human user. In the following sections, we pro- vide detailed description of how PLUME addresses each of PRELUDE's s limitation.
Knowledge of Topics As this work focuses on how to infer preferences, the version of PLUME used in all experiments assumes a distinct known user per topic. We note that PLUME is easily adaptable to use hidden topics if desired.
Metric Correlation We investigate and compare new pref- erence and generation-quality metries. For the preference quality metric, we evaluate an LLM-as-a-Judee (Zheng et al., 2023) metric that prompts an LLM to identify how simi- lar the inferred preference description is to the true prefer- ence description on a 5-point Likert scale, which we call Preference-Simillrrtty. For the generation quality metric, Levenshtein distance (In-L-
5. Experimental Set Up
All experiments consist of three phases. First, the user pro- vides a demonstration using their true preferences. Second, the agent completes the user's task using its currently in- ferred preferences (if any). Finally, the agent compares its generation with the user's example to infer new preferences to use going forward.
we evaluate enggh-nnomaliin dist), BERTScore, and an LLMMaa----Judge (Zheng al., 2023) metric inspired from the editing procedure in PRE- LUDE. The LLM-as-a-Judke evaluation is a per preference- component match (PPCM) that asks an LLM how much a component of a the ground truth preference is exhibited in a piece of writing on five point Likert scale from "clearly contradicts" (score of -2) to "clearly exhibits" (score of +2). This is repeated for each component of the true preference set, and compute the mean score across components. The full prompts used for both of the LLM-as-a-Judh metrics are shown in Appendix B (Figure 4 and Figure 5).
All Al assistants are evaluated on their ability to complete email writing and article summarization tasks on behalf of the user. Each task has different types (e.g., email to your boss versus email to a family member), and each user's 's preferences differ based on the task type. The assistants are evaluated along two dimensions: preference quality to measure the similarity between true and inferred preferences (see Appendix B. I), and generation quality to evaluate how
The results in Table 3 (Appendix C.1) show that Preference-
5

---

PROSE: Inferring User Preference Deseriptions
Summarization
Emails
Tasks Mean
Method
Pref. Sim.
PPCM
Pref. Sim.
PPCM
Pref. Sim.
PPCM
No Learning Baselines
NPC Oracle
0.00
4.09
0.00
0.91
0.00+0.00 3.87
40.00
3..67
1.71
3.89 10.06
1.95
1.002
Learning Baselines
ICL
0.00
1.05
4..00
1.394
0.00
+0.05
CIPHER-1
1.204
-0.05+0.06
10.7
0.33+0.05
1.4440.04
0.14
CIPHER-5
1.24
-0.0840.08
1.69
0.25
1.46
0.09
PROSE Ablations
PROSECt
1.234
0.51
1.467
0.97
1.34
0.74.0.00
PROSE,
1.30
0.47
10
0.84
1.3240.00
0.65
PROSE
1.35+0.10
0.49+0.11
+0.8
1.0440.06
+0.07
0.76
PROSE v
1.37
0.75+0.09
1.50
1.21
1.43
0.98+0.06
PROSENV
1.47+0.06
0.87+0.10
1.38
1.18
1.43
1.02.40.06
PROSE PROSEFan-ICt. 1.34
1.51
0.90
1.47 20.08
1.24
1.49
1.07:20.05
1.34
1.39
1.65.10.05
1.37
1.49:0.04
Table PROSE's performance on the two tasks measured by the quality of inferred preferences (Pref. Sim.) and preference compliance (PPCM) compared against no preference conditioning (NPC), true preference generation (Oracle), in-context learning (ICL), CIPHER (Gao al., 2024), and ablations over PROSE's components. Results are the mean and pooled standard error across the four LLMs and five seeds. Best results are bolded, second best are underlined.
well an agent's writing aligns with the user's true prefer- ences (see Appendix B.2). Both performance measures use LLM-as-a-Judge to assess the similarity between the true and inferred preferences, and between the true preferences and the agent's generations.
Pdese given only the user's demonstration wy: (2) PROSE infers Pdew given the user's demonstration Wy and the initial assistant generation w and (3) PROSE refines Paese over S S inference steps given the user's demonstration w., and the initial assistant generation w==0 PROSE refines Pdese
S inference steps given the user's demonstration W., and the iteratively refined assistant generations wa (0.5]] Comparing PROSE, and PROSE, measures the effect of comparing assistant generations the user demonstration when inferring preferences. The differ- ences between the PROSE,,. and PROSE.,..S quantifies the role of increasing the number of refinement steps. Lastly, comparing PROSE and the the complete PROSE al- gorithm PROSE clarifies the effects of comparing the user demonstration to the assistant's s generation conditioned the latest inferred preference description.
The agent aligns itself with four (email) or five (sum- marization) users with five demonstrations per user. Performance is evaluated per task as the mean across all demonstrations, users, and task type. Each task is run over five seeds (standard error reported over the seeds). The ground truth user preferences by task and task type are in Appendix D (Table 9). The performance of four LLMs is reported and compared: Siven.5-7BBInS--
GPT-4o-mini, and GPT-4o et al., 2024; OpenAI, 2023). For all LLMs, S and v are determined via a hyper- parameter sweep over v E 0.0.2 0.5, 0.75, and
RQ2: Does filtering preferences that are not relevant to multiple user demonstrations improve performance?
2,5. In our experiments 5 = for all LLMs, and
0.25 for Qwen2. 5- Instrrttct 0.5 for both GPT-Gomodels, and Qwen2. 5-728-In--- T1st is used as a synthetic human. The synthetic human prompts can be found in Appendix F.2
To answer this question, we evaluate a variant, PROSE that does not use the preference consistency verification step Section 3.2.
0.75 for For all experiments GPT-40
RQ3: Is conditioning preferences better than condi- tioning on demonstrations?
5.1. Research Questions
To answer this question, we compare PROSE, CIPHER (Gao et al., 2024), and ICL on all tasks, task types, and user profiles. We additionally combine the PROSE and ICL to
RQ1: Does iterative refinement improve performance?
We consider three variants PROSE: (1) PROSE,, infers
6

---

PROSE: Inferring User Preference Descriptions
measure the extent which they are complementary.
RQ2. We investigate the benefit of verifying preferences by comparing PROSE to PROSENV. Here, but consistent of 1.5% and 1.7% for Pref. Sim. and PPCM respectively when using preference consistency verification.
modest
5.2. Baselines
In addition to the PROSE baselines outlined Section 5.1, we implement the following models.
RQ3. While on average across LLMs PROSE outper- forms CIPHER and all PROSE ablations, ICL outperforms PROSE. However, Figure 2 shows that PROSE's perfor- mance scales better with the quality of the underlying LLM (e.g., 5-ree2..-72--nnseee baselines except Oracle. Notably, when using GPT-4o, PROSE outperforms ICL (1.35 vs 1.32 task mean Ap- pendix C.2). We further investigate the benefits and limita- tions of PROSE and the learning baselines by comparing the performance across preference sets (Figure 3), and find that ICL excels on sets with the strongest structural pref- erences (e.g., Chat Forum Posts which includes "write in the style of a tweet"). In contrast, PROSE excels on the preference sets requiring more nuanced understanding of tone (e.g., Paper Review, which includes "be sharply criti- cal"). From examining logs, we notice that the LLMs are less adept at inferring encompassing structural preferences and often try capture these preferences using multiple relevant, but imperfect preferences (e.g.. "use emojis for emphasis", "use 1-2 specific hashtags") As PROSE and ICL seem to have complementary strengths, we combine the two (PROSEFanHII)) )) for a gain of 7.8%, 8.9%, and 51.1 % over PROSE, ICL, and CIPHER when using 4o as the agent's LLM.
We implement CIPHER-1 1 and CIPHER-5 (Gao et al., 2024), and an in-context learning (ICL) agent using previously observed user demonstrations. The CIPHER baselines are adapted to learn from PLUME's user demonstrations instead of user edits.
vs. 4PT than all
We then implement three additional baselines. An agent that solves the task with no preference conditioning (NPC), providing a lower-bound of performance. An oracle agent (Oracle) that receives access to the user's 's true preference, providing an upper bound of performance, and a varia- tion of PROSE that is conceptually equivalent to CIPHER, PROSECE, but uses PROSE's improved prompt templates. PROSECE uses a single LLM generation, a single inference step, and uses no preference consistency verification.
6. Results and Discussion
We present our main PLUME results in Table 2. Results on PRELUDE can be found in Appendix C.3. To compare tasks on generation quality with metrics on different scales, we use percentile score, where 0% corresponds the no preference conditioning baseline (NPC) and 100% to the Oracle baseline. Percent improvements are reported as the difference in scores on this scale. Overall, PROSE outperforms PROSEcr: by 12%, and CIPHER by 33%.
Human Evaluation To further validate the effectiveness of PROSE, we ran human evaluation with 16 participants (3 are ML researchers; 9 women and 7 men; age in [19, 58]). Participants completed within subjects AB test com- paring PLUME+ICL generations to ICL generations and PLUME+ICL generations to CIPHER generations. Partic- ipants evaluated the final LLM generations (i.e. the gen- eration after seeing all previous demonstrations) across all five seeds for two different preference sets for the email task and two different preference sets for the summarization task. This leads to total of 20 survey items per method comparison. We used the responses to compute a win rate for PLUME+ICL compared to each of ICL and CIPHER. For PLUME+ICL versus ICL, we see an average win rate of 69.49 69.4%. For PLUME+ICL versus CIPHER, we see an average win rate of 91.8% The human evaluation results are in line with our synthetic evaluation results and support the effectiveness of the synthetic evaluation.
RQ1. In our first question, we set out to verify whether generating iterative candidate trajectories is beneficial to inferring preferences. Comparing PROSE to its ablated versions on the action/gencratioon quality metric (PPCM), shows that each component of the iterative refinement pro- cess improves performance. Comparing PROSE with no comparison generation - PROSE., - to PROSE with a single LLM-ggnnrated comparison generation - PROSE,, we observe that providing the comparison generations is beneficial when inferring preferences (3.8% mean improve- ment). This result supports the algorithmic decisions in (Gao et al., 2024; Peng et al., 2024). Allowing for multiple refinement steps provides further increase in performance (Table 2: PROSE,, vs. PROSE,...S.1 7.8% mean im- provement). This can be explained by the LLM having more chances to infer correct preferences. Lastly, when comparing PROSE SS:11 1 to PROSEpal we see another 3.2% improvement. This highlights the benefits of updat- ing candidates after each inference step using the newly inferred preferences. In all, iterative refinement provides mean improvement of 14.8%.
Discussion. Our results demonstrate that using iterative refinement and consistency verification improves over CI- PHER terms of preference description quality, genera- tion quality, and performance stability (i.e., performance increases with the number of demonstrations, see Figure 6). Additionally the performance difference between CIPHER
7

---

PROSE: Inferring User Preference Deseriptions
Summarization
Emails
CIPHER-1
2.0
2.0
ICL
1.5
PROSE
PROSE+ICL
1.0
1.0
Oracle
NPC
PPCM
PPCM
0.0
0.0
0.5
1.0
11.0
11.5
-1.5
-2.0
2.0
-wen2.5-7B-Instruct Owen2.5-72B-nnstiee GPT-4o-mini GPT-4o
Qwen2.5-78 (rrenn.5-7--nntttee I--nstrruct Qwen2.5 B-IIstrucct
GPT-4o-mini
GPT-4o
Preference compliance performance (PPCM) for CIPHER-1, in-context learning (ICL), PROSE, Oracle, and no preferences (NPC) for different preference-iffrringg LLMs. The LLMs are sorted by MMLU performance: Qwen2. -
Figure 2:
74.2, GPT-40-mim 82, Qwen2. GPT-40 is the proxy human with mean and standard error reported over seeds.
86.1, and GPT-4c 88.7.
and PROSEcE highlights the impact of our prompt-tuning efforts. In this regard, PLUME's prompts (Appendix F) can serve as a valuable starting point for extensions to other tasks, however, task-specific adaptations should be made.
Lastly, while developing PROSE, we learned the importance of phrasing the preference description in the LLM's 's "own words". We initially sorted the preference components by length before aggregation, however, this led to an average performance drop of I 11% across tasks relative to keeping the LLM's order for the preference components. This finding is inline with other work that shows that LLMs sensi- tive to the order of list items (Pezeshkpour & Hruschka, 2024; Aroca-Ouellette et al., 2021). We believe future work investigating the impact of the ordering may yield useful insights.
Our results suggest that consistency verification provides only a modest improvement PROSE. Therefore, bet- understand its impact, examine the learning logs and find that consistency verification effectively prunes ir- relevant preferences e.g.,"be concise and direct" and preferences that overfit to specific passages- e.g., "include personal details about characters". However, the pruned preferences typically have minimal impact on perfor- mance metrics as they rarely contradict the true preferences. Moreover, the pruned preferences do not drastically alter the generations the orthogonal preferences often match the LLM's s default behavior while the overfit preferences become irrelevant and ignored. As such, the current metrics have difficulty measuring the presence of these irrelevant preferences. Nevertheless, we believe valuable to prune the irrelevant preferences, as they reduce the number of tokens required.
6.1. Limitations and Future Work
While PROSE and PLUME provide a number improve- ments, their limitations and challenges provide interesting avenues for future work. First, in this paper we focus on learning with the fewest user demonstrations possible. How- ever another aspect efficiency is the total number of to- kens, and adding more refinement and preference consis- tency verification steps increases the number of tokens used. In our experiments, PROSE used 5.87x (prompt) and 6.07x (generated) more tokens on average than PROSECE Given the monetary and environmental cost of LLMs, re- ducing the number of tokens while retaining performance is an important area for improvement. Lastly, full-scale human trial would provide greater understanding of the benefits and limitations of the proposed method. We look forward to investigating this more closely in future work.
We find PROSE is competitive with and complementary to ICL while providing several advantages: (1) preferences are easier to interact with than a dataset of in-context examples as a user can view and modify the inferred preferences, (2) at inference time, PROSE requires approximately 1 of the prompt tokens, and (3) the inferred preference description can benefit wider range of tasks (e.g. human-agent co]- laboration (Liu al., 2024), sample efficient imitation/rein- forcement learning, and generating personalized preference pairs for RLAIF (Sun et al., 2024)).
8

---

PROSE: Inferring User Preference Descriptions
PersonalProblem
PaperSummary
PaperReview
PROSEPMOSE-KE
idL
CIPHEN-
ICL
PROSE
CEMER1
NewsArticles
ChatForumPosts
Encyclopedia
PaperAbstract
MovicReview
19:00
Figure 3: Generation quality (PPCM) for PROSE, CIPHER-1, I, in-context learning (ICL), and PROSE+ICL by Email (top) and Summary (bottom). GoT-4o is the agent's LLM with mean and standard error reported over 5 seeds.
7. Conclusion
Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S.. Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: http:::ttt/ooott httpt:s:///rooceednggs..s. neurips. cc / htsp/aper__fieeeee httppeeeiiieeeppppp/////////ee 1 4457c0060...044994414400000000e. pdf.
In this paper, we present novel algorithm, PROSE, and new benchmark, PLUME. PROSE's two novel contribu- tions guide an LLM to better infer preferences from user demonstrations by: (I) iteratively refining preferences by conditioning an LLM on each refinement step's 's updated preference description to see its impact, and (2) decompose preferences into components and verify the components against relevant user demonstrations. We demonstrate that the proposed method improves LLM's ability to align to a user by as much as 33%.
Chakrabarty, T., Laban, P., Agarwal, D., Muresan, S., and Wu, C.-S. Art or artifice? large language models and the false promise of creativity. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-34, 2024.
Impact Statement
The proposed method allows for greater personalization of assistive agents. However inferring user's preferences could be seen invasion of privacy. These methods should be applied only with explicit consent from human users.
Gao, G., Taymanov, A., Salinas, E., Mineiro, P., and Misra, D. Aligning Ilm agents by learning latent preference from user edits. In Thirty-eigth Conference on Neural Information Processing Systems, 2024. URL https: //arxiv. org/ aww////44000 /2004 15269.
References
Aroca-Ouse OueeleOe,,,,tee, S., Paik, C., Roncone, A., and Kann, K. PROST: Physical reasoning about objects through space and time. In Zong, C., Xia, F., Li, W., and Nav- igli, R. (eds.), Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pp. 4597- 4608, Online, August 2021. Association for Computa- tional Linguistics. doi: 404. URL https: ac /ttaaacaaaattcooe org/202.. 2021. Findings-acl.. ac1.104/..
Levenshtein, I. Binary codes capable of correcting dele- tions, insertions and reversals. Soviet Physics Doklady, 10:707, February 1966.
Li, K., Patel, O., Viégas, F., Pfister, H., and Wattenberg, M. Inference-time intervention: Eliciting truthful answers from a language model. In Oh, A., Naumann, T., Glober- son, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, vol- ume 36, pp. 41451-41530. Curran Associates, Inc., 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Lin, X., Dai, Z., Verma, A., Ng. S.-K., Jaillet, P., and Low, K. H. Prompt optimization with human feedback, 2024. URL htwp:: //tp://aavvvv./e/eeeeeee wwwg/bs//////eeoeeoeoooeeeeeeeee 17346.
Askell, A., Agarwal, S., Herbert-Vosse A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J.,

---

PROSE: Inferring User Preference Deseriptions
Lindsey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Turner, N. I.., Citro, C., Abrahams, D., Carter, S., Hosmer, B., Marcus, J., Sklar, M., Templeton, A., Bricken, T., McDougall, C., Cunningham, H., Henighan, T., Jermyn, A., Jones, A., Persic, A., Qi, Z.,, Thomp- son, T. B., Zimmerman, S., Rivoire, K., Conerly, T., Olah, C., and Batson, J. On the biology of large language model. Transformer Circuits Thread, 2025. rccitttsrriiriirs.oee pubb 2025/attribution graphs/biologe
Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Salemi, A., Mysore, S., Bendersky, M., and Zamani, Lamp: When large language models meet personaliza- tion, 2024. URL https: //axxiv////vvvva///aa///a org/ abs 2304. 11406.
URL http://ccc..iito...rrraaa/oooooo .tmll.
Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., and Hashimoto, T. Whose opinions do language mod- els reflect? In International Conference on Machine Learning, pp. 19971430004. PMLR, 2023.
Liu, J., Yu, C., Gao, J., Xie, Y., Liao, Q. Wu, Y., and Wang, Y. Lim-powered hierarchical language agent for real-time human-ai coordination. In Dastani, M., Sichman, J. S., Alechina, N., and Dignum, V. (eds.), Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2024, Auckland, New Zealand, May 6-10, 2024, pp- 1219-1228. International Foundation for Autonomous Agents and Multiagent Systems ACM, 2024. doi: 055555/363356773366699997 URL https: /dll../../ acm.
Shaikh, O., Lam, M., Hejna, J., Shao, Y., Bernstein, M., and Yang, D. Show, don't tell: Aligning language models with demonstrated feedback. arXiv preprint TrXiv:2406.00888 2024.
Shashidhar, S., Chinta, A., Sahai, V., and Tur, H. Un- supervised human preference learning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference Empirical Methods in Nat- ural Language Processing. pp. 3412-3445, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10. 10.18663/v1/2024.cmmll/ main. 200. URL https: /ecctts://aclaae http///aaaatthhhccco org/ 2024. emnlp-male .00//
OpenAI. Gpt-4 technical report. ArXiv, ab//233..08774 2023. URL https://arxiv. 08774.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Shen, X., Zhang, R., Zhao, X., Zhu, J., and Xiao, X. Pmg Personalized multimodal generation with large lan- guage models, 2024. URL attps://arxiv.cr abs/2404.0867.
Peng, A., Bobu, A., Li, B. Z., Sumers, T. R., Sucholutsky, I. Kumar, N., Griffiths, T. L.,, and Shah, J. A. Preference- conditioned language-guisn guided abstraction. In Proceed- ings of the 2024 ACM/EEE International Conference Human--Rbot Interaction, pp. 572-581, 2024.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, Learning to summarize from human feedback. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546
Pezeshkpour, P. and Hruschka, E. Large language mod- cls sensitivity the order of options in multiple-choice questions. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2006-2017, Mexico City, Mexico, June 2024. Association for Computa- tional Linguistics. doi: 10. 11111655/112224.fmmii nacc.. 130. URL https: a antho org./024 findings mraeiingsnnacl.1333 naac1.130/.
Sun, Z., Shen, Y.. Zhou, Q., Zhang, H., Chen, Z., Cox, D., Yang, Y., and Gan, C. Principle-driven self-alignment of language models from scratch with minimal human supervision. In Thirty-sevnnth Conference Neural Information Processing Systems, 2023. URL https: / / //operrevil ew net. / forum?
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask leamners. 2019.
Sun, Z., Shen, Y., Zhang, H., Zhou, Q., Chen, Z., Cox, D. D., Yang, Y., and Gan, C. SALMON: Self-alignment with instructable reward models. In The Twelfth In- ternational Conference on Learning Representations, / /oppnreviei net/ forum?
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization:
2024. URL https:
i id=xJbem38UMx.
10

---

PROSE: Inferring User Preference Descriptions
Tan, Z., Zeng, Q. Tian, Y., Liu, Z., Yin, B., and Jiang, M. Democratizing large language models via personalized parameter--effiiien fine-tuning, 2024. URL https:// arxiv. org/ abs 2402 04401.
Turner, A. M., Thiergart, L., Leech, G., Udell, D., Vazquez, J., Mini, U., and MacDiarmid, M. Steering language models with activation engineering, 2024. URL https: //arxiv. org/ abs /aoos/200 230 10248.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J.. Zhang, J., Yang. J., Yang. J., Zhou, J., Lin, J., Dang,
K., Lu, K., Bao, K., Yang, K. Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y.,
Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 2024.
Pameire J., Wong, R. Y., Hartmann, B., and Yang. Q. Why johnny can't prompt: how non-ai experts try (and fail) to design Ilm prompts. In Proceedings of the 2023 CHI Conference on Human Factors Computing Systems, pp. 1-21, 2023.
Zhang*, T., Kishore*, V., Wu", F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL htttps://opeereeeeeee id=SkeHuCVFDr.
net forum?
Zheng, L.,, Chiang, W.-L., Sheng. Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, 7., Li, D., Xing, F., Zhang, H., Gonzalez, E., and Stoica, I. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-sevntth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: //opereview. net http://frrmmmduccctteeee dduccHHPDlaa..
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human- level prompt engineers. arXiv preprint 2rXiv:2211.0991 2022.
Zhuang, Y., Sun, H., Yu, Y., Qiang, R., Wang, Q., Zhang, C., and Dai, B. Hydra: Model factorization framework for black-box I1m personalization, 2024. URL https: //arxiv. org/ htce:/bbs/24466 2406.400 02888.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint 109109999, 2019.

---

PROSE: Inferring User Preference Descriptions
A. Algorithm
Algorithm Assistant Task Completion
Require: Fask (Task }nstance)
1: Initialize empty preference set: P. f-
2: Retrieve relevant examples:
3: E+ heg..uddwwe.eeuumpessstetrrtcaie
memory)
4: for each E F: do
5: Pa + P. Ue..e
6: end for
7: Aggregate condense preferences:
8: Pdese f generate (11m, Taggrogater Pa)
9:
Sample agent generation:
u f generate (11m, Tuskk Pdeae)
11: Return Completed generation wa. preference description pdese, and relevant examples E
12

---

PROSE: Inferring User Preference Descriptions
Algorithm 2 PROSE: Preference Reasoning by Observing and Synthesizing Examples
Require: Fias TTask }
Require: Wi {User demonstration}
Require: wa {Agent generation]
Require: Pdese (Preference description}
Require: F. {relevant Examples]
1: Initialize Pa +- generate I'weakdown: Pdese)
2: for each E S] do
3: if' : = w/, then
4: Stop refinement
5: else
Refine preferences:
p generate 1m, Tupdane Pleser War wy)
8:
Decompose preference:
9:
Pa Pa U generate 1m, Tbreakdown; P'
10:
Generate new candidate generation:
11:
- + generate (11m, Tiskk Prisse)
12: end if
end for
Initialize empty verification score list:
15: Uscores +- 2
16: for each P. in P. do
17: for each E E do
18:
Verify preference against demonstration:
19:
Uscores + Uscores U generate & Everiticatonn e. Passe)
end for
if mean(Pscores) then
22: Discard Pe from P.
23: end if
24:
end for
25: Add (Tusk: w P.) to interaction memory
13

---

PROSE: Inferring User Preference Deseriptions
B. Metric Definitions
B.1. Preference Inference Quality
Preference Description Length As conditioning on unnecessary tokens when generating responses aligned with user preferences undesirable, we measure the number of tokens in the preference description. The preference length (Pref Len) is the number of characters used describe user's preferences, which is highly correlated with the number of tokens required.
Preference Similarity To assess the similarity between the inferred preferences and the ground truth preferences, the human proxy (GPT-40 in this paper) is prompted to evaluate how similar each inferred preference is to each ground truth preference following:
Preference Similarity Illm (udgeettee,,,, inferred),
(I)
where is the true preferences (see Appendix D Table 9), inferred is the inferred preference description, and judge function that prompts the human proxy LLM to evaluate how well given inferred preference aligns with the true preference scale of 0 +4 (see Appendix Figure 4).
System Prompt
You experienced editor that is evaluating how similar writing preferences are.
User Prompt
You received the following description of a user's writing preferences:
Inferred preference: <inferred preference i>
"How similar are the inferred preferences the true writing preferences below?
True preference: strue-preference.
Analyze how the preferences would impact user's writing. After reasoning, select one of the following options: extremely similar, very similar, moderately similar, slightly similar, not at all similar Your final selection should be on a new line prefaced with "Verdict:"
Figure 4: I.M-as--a-Judge prompts to assess the similarity between the true and inferred preferences. The system prompt is prepended to the user prompt following the LLM's chat template. "<...>" indicates that the text is formatted from variable. inferred.preeerreee preference.i one of the inferred preferences. true pruueppreferene... refers to one of the k true preferences that the user has.
B.2. Generation Quality
To assess the quality of the preference conditioned LLM's generations, the human proxy (GPT-4o in this paper) is prompted to evaluate how well the given generation complies with the each of the ground truth user preferences. The generation quality is then compute as the mean score over ground truth user preference components following:
truel Ilm_judge assistant
PPCM
truel
where true is the set of true preferences (see Appendix D Table 9), assistant attempt is the assistant's summary or email, and llm_judge is function that prompts the human proxy LLM to evaluate how well a given assistant solution aligns with the true preference on a scale -2 to +2 (see Figure 5).
The prompt used by the I.L.Mas-a-ludge for generation quality evaluation are shown in Appendix Figure 5.
14

---

PROSE: Inferring User Preference Descriptions
System Prompt
You are an experienced editor that is evaluating writing samples.
User Prompt
You received the following (summary [email]:
<agentt.competioon:
Does the above (summary email} exhibit the following preference: <true <true. www.pprffeee....... .i>? Identify, analyze, and reason about specific excerpts that show similarities or contradictions of underlying preferences. After reasoning. select one of the following options:
clearly exhibits, somewhat exhibits, neither exhibits nor contradicts, somewhat contradicts, clearly contradicts Your final selection should be on a new line prefaced with "Verdict:"
Figure LLM-as-a-Judge prompts for the per pecference-oompnent match metric (PPCM) used in the PLUME environment. The system prompt is prepended to the user prompt following the LLM's chat template. "<...>' indicates that the text formatted from a variable. agent. aegent.compleee on refers to the agent's 's article summary or email, depending on the sub-task. true.preeerence.i refers of the k true preferences that the user has.
15

---

PROSE: Inferring User Preference Deseriptions
C. Extended Results
Additional results tables and figures discussed in the main body the paper.
C.1. Metric Correlation
The metric correlation results for the assistive writing tasks (Table 3).
PRELUDE
PRELUDE NoEdit
PLUME B B.Score
Metric
Acc.
B.Score
P. Len.
P. Sim.
Acc.
B.Score
P. Len.
Sim.
Acc.
P. Len.
P. Sim.
Summarization
L-dist
-0.47
-0.53
-0.43
-0.46
-0.08
-0.14
0.01
-0.18
0.02
-0.13
-0.21
-0.10
In-L-dist
-0.50
-0.57
-0.48
-0.51
-0.22
-0.36
-0,23
-0.42
-0.08
-0.33
-0.40
-0.36
PPCM
0.49
0.63
0.53
0.54
0.51
0.65
0.54
0.58
0.34
0.65
0.71
0.71
Emails
L-dist
-0.26
-0.32
-0.27
-0.26
-0.10
-0.31
-0.36
-0.27
-0.11
-0.19
-0.25
-0.19
In-L-dist
-0.23
-0.31
-0.27
-0.26
-0.08
-0.34
-0.42
-0.28
-0.12
-0.34
-0.41
-0.38
PPCM
0.18
0.30
0.43
0.18
0.20
0.32
0.45
0.19
0.48
0.79
0.74
0.79
Across Both Tasks
L-dist -0.43
-0.43
-0.31
-0.39
-0.09
-0.17
-0.03
-0.20
0.01
-0.15
-0.21
-0.11
In-L-dist -0.45
-0.45
-0.32
-0.42
-0.18
-0.27
-0.08
-0.32
-0.09
-0.32
-0.39
-0.35
PPCM
0.42
0.48
0.37
0.42
0.45
0.52
0.39
0.46
0.39
0.68
0.71
0.73
Table Pearson correlation between preference similarity metrics and generated writing similarity metrics broken down by task (summarization email). For Levenshtein distance (L-dist) and length-normalized Levenshtein distance (In-L-dist) lower is better, inverse correlation is expected. For PPCM, Acc. (Accuracy), B.SCore (BertScore), Len (Preference Description Length), and P. Sim. (Preference Description Similarity) higher is better. Best correlation in each framework is bold. Best overall correlation is underlined. See Appendix B for full description of each metric.
16

---

PROSE: Inferring User Preference Descriptions
C.2. Baselines and Ablations per LLM
The baseline and PROSE ablation results for Qwen2. 5b Instruct, Qwen2. 5 --b Instruct, GPT do-mini, and GPT-4o. The results in Table 2 are averaged across these four LLMs and results reported in this section.
Summarization
Emails Pref. Sim.
Method
Pref Len
Pref. Sim.
PPCM
Pref Len
PPCM
No Learning Baselines
NPC
0.00
1.00
-1.12.0.04
0.00
0.00
00.97 10.04
Oracle
120.80+0.00
3..8
1.75
118.50.0.00
3.90
1.9540.01
Learning Baselines
ICL
6237.54=1163.61
0.00
1.42
0754.734978
0.00
1.41+0.08
CIPHER-1
50.40+1.08
0.90
-0.33
51.55
1.32
-0.01+0.04
CIPHER-5
49.78+1.86
1.04 +0.05
-0.11
49.04
1.6640.05
-0.01
PROSE Ablations
PROSECE
306.05=15.72
0.59
0.07
390.29
0.99
0.62+0.06
PROSE,
274.05_20.33
0.45
0..0
353.77-33.87
0.84
0.1640.15
PROSE,,
291.45=15.41
0.61+0.11
0.03
301.27=26.19
1.00
0.45+0.01
PROSE.,,,.52
574.73-27.75
0.77
0.40
611.92
1. 1.14
0.83
PROSENV 745.49
0.90
0.48
753. 75.90
0.97 #0.11
0.72+0.04
PROSEFulI 604.69-31.19
0.8940.10
0.56 09
698. 698.34-41.000
0.91
0.78+0.10
PROSEFaIIIL 6829.38
0.76
1.23
7435.39
0.96
1.52+0..0
Table 4: Qwen2. 5-7-222577-Insttuun + PROSE's performance the two tasks measured by the quality of inferred preferences (Pref. Sim.) and preference compliance (PPCM) compared against preference generation (NPC), true. preference generation (Oracle), in-context learning (ICL), CIPHER (Gao et al., 2024), and ablations over PROSE's components. Results are the mean and standard error across five seeds. Best non-Oracle results per task are bolded.
17

---

PROSE: Inferring User Preference Deseriptions
Summarization Pref. Sim.
Emails
Method
Pref Len
PPCM
Pref Len
Pref. Sim.
PPCM
No Learning Baselines
NPC
0.00
0.00
-1.11
0.00
0.00
0.86
Oracle
10.000+0.00
3.88
1.68-0.03
118.50+0.00
3.85+0.06
1.95+0.01
Learning Baselines
ICL
6237.5441163.61
0.00
0.12
6754.734975.88
0.00
1.37 +0.0s
CIPHER-1
88.36-1.91
1.2641.04
0.03_
83.01 12.03
1.82
0.56
CIPHER-5
141.2346.64
1.22+0.05
0.06=0.06
141.4548.35
1.60
0.34
PROSE Ablations
PROSECE
359.01+23.18
1.25
0.62=0.10
375.76+6.64
1.56
1.0640.05
PROSE,,
333.35113.53
1.35
0.05
369.00 22.115
1.29 10.07
0..09
PROSE
304.55+16.14
1.41 +0.07
0.52=0.09
352.71
1.75
1.17:0.00
PROSB.,,,55/
635.88
1.42
0.04
829.24 33700
1.56
1.39 +0.08
PROSENV
937 937.37
+057
0.97
1004.21
1.3640.13
1.32+0.
PROSE
628.28
1.60+0.11
0.99=0.07
880.98 -22.95
1.55
1.38
PROSEFaI-IC..
1117.28
1..1
1.38_..05
7624.144:973.68
1.45
1.70
Table 5:
PROSE's performance on the two tasks measured by the quality of inferred preferences (Pref. Sim.) and preference compliance (PPCM) compared against no preference conditioning (NPC), true preference conditioning (Oracle), in-context learning (ICL), CIPHER (Gao et al., 2024), and ablations over PROSE's components. Results are the mean and standard error across five seeds. Best non-Oracle results per task are bolded.
Summarization Pref. Sim.
Emails Pref. Sim.
Method
Pref Len
PPCM
Pref Len
PPCM
No Learning Baselines
NPC
0.00
0.00
02
0.00 118.5010.00
0.00 3.95
-0.01
Oracle
120.8011.000
3.8410.08
1.7040.04
1.93
Learning Baselines
ICL
1237.54-11688
0.00
1..07
6754.73
0.00
1.38
CIPHER-1
138.91 13.67
1..4
0.24
148.69 1.51
1.79 10.03
0.37
CIPHER-5
74.93
1.2640.10
0.05
78.0942.17
1.74.0.08
0.30
PROSE Ablations
PROSECE
384.12=12.65
1.4440.07
0.67
412.09
1.57+0.03
1.0340.08
PROSE,
255.81
1.55
0.47+40.1
364.86_13.11
1.40+0.10
1.01
PROSE,
321.00 321.0948..3
1.7640.11
0.70+L0.07
375.89 24..73
1.75
1.24
PROSE, 151
551.56-16.29
1.4640.11
0,82
745,58
1.50+0.08
1.30++.13
PROSENV 699.89
1.4840.07
0.94
798. 798.89-37.
1.30
1.1640.07
PROSE 575.88
1.58
0.91
699.25 233.42
1.60
1.25
PROSE 6795.69=1167.6d
1.42+0.08
1.27+0.00
7542.95+997..0
1.52+0.08
1.70+0.04
Table 6: GPT-46 GPT-4o-mini PROSE's performance the two tasks measured by the quality of inferred preferences (Pref.
Sim.) and preference compliance (PPCM) compared against no preference conditioned (NPC), true preference generation (Oracle), in-context learning (ICL.) CIPHER (Gao et al., 2024), and ablations over PROSE's components. Results are the mean and standard error across five seeds. Best non-Oracle results per task are bolded.
18

---

PROSE: Inferring User Preference Descriptions
Summarization Pref. Sim.
Emails Pref. Sim.
Method
Pref Len
PPCM
Pref Len
PPCM
No Learning Baselines
NPC
0.00
0.00
- 11.06.0.04
0.00
0.00
0.88
Oracle
120.80+0.00
3.84
1.69
118.50+0.00
3.85+0.06
1.97+0.01
Learning Baselines
ICL
6237.54=1103.6
0.00+0.00
1.24
6754. 0754.734975.30
0.00+0.00
1.4040.04
CIPHER-1
60.80+1.24
1.23
- -0.1440.07
67.62
1.75
0.41
CIPHER-5
56.03 +1.52
1.43
0.111 10
56.81 +2.00
1.7640.05
0.38+0.06
PROSE Ablations
PROSECE
1
1.6340.05
0.
342.27=12.06
1.71
1.1840.10
PROSE,
228.11 11..4
1.84
10.1
303.7447.32
1.84
1.30
PROSE,
249.26=11.70
1.62+0.12
0.70+0.10
317.5426.48
1.82+0.09
1.2940.10
PROSE, St11
428.30 118.58
1.83
0.92
534.29
1.79
1.33
PROSENV
489.66 13.24
1.94
1.07
513.60
1.90
1.5040.07
PROSE PROSEFuIHIL
446.73=10.71
1.98
1.1540.07
532.3447.78
1.83+0.08
1.5540.03
1114.28
1.77:4..10
1.4640.05
7297 9997.6224900.54
1.64
1.67
Table 7: GPT-4o PROSE's performance the two writing tasks measured by the correctness of inferred preferences (Pref. Sim.) and preference compliance (PPCM) compared against no-preference conditioning (NPC), true preference generation (Oracle), in-context learning (ICL), CIPHER (Gao et al., 2024), and ablations over PROSE's components. Results are the mean and standard error across five seeds. Best non- Oracle results per task are bolded.
19

---

PROSE: Inferring User Preference Deseriptions
C.3. PRELUDE Results
Results on PRELUDE (Gao al., 2024) for PROSE and baselines: a no-preference conditioning (NPC), oracle preference baseline, in-context learning (ICL). CIPHER-I, and CIPHER-5 (Gao et al., 2024) (Table 8). To directly evaluate the ability to infer preferences, provide all models with ground truth knowledge of the source of the documents. On the summarization task, PROSE outperforms all baselines on action/generation quality. On the email writing task, PROSE outperforms all baselines on the PPCM metric, but slightly underperforms CIPHER- on the poorly correlated Levenshtein distance metric (see Section 4-Metric Correlation for issues with Levenshtein distance).
Results this table further support issues with the current preference-quality metrics. the email writing task, the no-learning baseline (which always uses an empty preference), has a higher accuracy than any learning method, which may be due to the significant overlap between preference sets in the task. Further, in both tasks, the highest prefenence-quality scores do not lead to the highest action-quality scores. We encourage future work to look into alternative preference-quality metrics.
We lastly note that PRELUDE has substantially smaller range between the no-preference conditioned (NPC) and oracle preference baselines relative to PLUME. On PPCM, PRELUDE has range 2.45 and 0.62 for summarization and email writing respectively, while PLUME has ranges of 3.17 and 2.91 for the two tasks. This further supports PLUME as the primary evaluation environment.
Summarization
Method
Accuracy
BScore
Levenshtein
PPCM
No Learning Baselines
NPC
0.20+0.00
0..4340.00
107.80
-0.7440.10
Oracle
1.00
1.00+0.00
1.08
1.62+0.09
Learning Baselines
ICL
0.20
-0.43.4.00
104.24
0.71
CIPHER-1
0.61 10.06
0.13
48.01 10.77
0.74
CIPHER-5
0.46+0.02
0.02+0.01
33.86
0.82+0.50
PROSE
0.68 20.12
0.01 +0.03
9.30
1.1840.16
Emails BScore
Method
Accuracy
Levenshtein
PPCM
No Learning Baselines
NPC
0.25
0.37
48.12411.44
0.8640.08
Oracle
1.00
1.00
1.0242.03
1.57
Learning Baselines
ICL
0.25
-0.37
53.38413.4
0.87
CIPHER-1
0.03
-0.1610.04
12.06
1.04
CIPHER-5 0.25+0.00
-0.08+0.04
13.3449.57
1.09+0.08
PROSE 0.01
-0.22+0.02
14.05
1.10
Table 8: PRELUDE Results. PROSE's ability to infer the correct preference set and generation quality across the two PRELUDE tasks compared against no-preference conditioning baseline (NPC), a method with access to the true preferences (Oracle), in-context learning (ICL), and CIPHER (Gao et al., 2024). Results are reported as the mean and standard error across five seeds. Accuracy and Bscore (BERTScore) (Zhang* et al., 2020) are preference-qualle metrics, while Levenshtein distance and PPCM (per preference-component match) are action-quality metrics.
20

---

PROSE: Inferring User Preference Deseriptions
C.4. Preference Inference and Conditioning Performance by Number of User Samples
In Figure 6 we show the impact of the number of samples for given user according the measures for inferred-preferentt and generation quality metrics.
Summarization
Emails
PREDICT
2.0
ICL
Full+IC.
1.5
Pret. Sim.
1.5
Pref. Sim.
1.0
PREDICT
0.5
0.5
CIPHER-1
Ful+-CL
2.0
1.5
1.5
1.0
1.0
0.5
0.5
PACM
PRCM
0.0
0.0
-0.5
PREDICT CIPHER-1
-0.5
PREDICT
-10
-1.0
CIPHER-1
-1.
-15
Ful+ICL
Full+ICL
-2.0
-2.0
0.0
0.5
1.0
2.5
3.0
3.5
4.0
1.5
2.0
3.5
4.0
Samples User
Samples per User
Figure 6: Performance for PROSE, CIPHER-1, in-context learning (ICL., and PROSE+ICL given different numbers of user samples learn from. Mean and standard error over five seeds for preference quality (Pref. Sim.) and preference- conditioned generation quality (PPCM). GPT-4o is the LLM.
21

---

PROSE: Inferring User Preference Descriptions
D. PRELUDE vs. PLUME Preference Sets
The preference sets used for each document source and environment (PRELUDE vs. PLUME) are given in Table 9.
Document Source
Task Version
User Preferences
Summarization
interactive, playful language, positive, short sentences, storytelling, style targeted to young children
PRELUDE
News Articles
adopt a step-by-step structure, include a simile, use ampersands (&) instead "and"s, s. write in the style of children's 's book
PLUME
brief, immersive, invoke personal reflection, second person narrative, show emotions
PRELUDE
Chat Forum Posts
adopt a header and sub-header structure, include rhetorical questions, use ALLCAPS to emphasize words, write in the style of a tweet brief, bullet points, parallel structure
PLUME
PRELUDE
Encylopedia Pages
adopt rhyming structure, include modern slang, use semicolons (:) when possible, write in the style of screenplay
PLUME
PRELUDE
inquisitive, simple English, skillful foreshadowing, tweet style, with emojis
Paper Abstract
adopt question-answering style structure, include personifications, use archaic language, write in the style of podcast question answering style
PLUME
PRELUDE
Movic Review
adopt a consiono-s structure, include onomatopoeias, use imagery, write the style of old timey radio
PLUME
Email Writing
PRELUDE
conversational, informal, no closing
Personal Problem
be intensely emotional, include alliterations, use formal tone, write in second person narrative
PLUME
PRELUDE
call to action, casual tone, clear, positive
Paper Review
sharply critical, include several short and punchy sentences, use paren-
PLUME
thetical asides, write using assertive expressions
PRELUDE
engaging, personalized, professional tone, thankful closing
Paper Tweet
be blatantly sarcastic, include hyperboles, use an informal tone, write in third person perspective professional greeting and closing, respectful, straight to the points, struc- tured
PLUME
PRELUDE
Paper Summary
be highly inquisitive, include several long and flowing sentences, use emojis, write using conditional expressions
PLUME
Table 9: The user preferences for each assistive writing task (summarization vs. email writing), task topic, and benchmark version.
22

---

PROSE: Inferring User Preference Deseriptions
E. Illustrative Examples of Issues with PRELUDE
E.1. Levenshtein Distance and Multiple Generations
We present here an inherent issue with Levenshtein distance. For all three generations, we prompt gpt-4o to summarize the encyclopedia article on hotdogs. The first two generations (A. and B.) use identical prompts, in which we ask use the preference: "use haiku". In the third generation, ask calculate the Levenshtein distance between each pair, reported in Table 10. Notably, the Levenshtein distance and length normalized Levenshtein distance, in which lower score represents higher similarity, would both indicate that generation A and C are more similar to generation A and B, even though generation A and B share the same exact same prompt while generation uses a entirely different preference. BERTScore does a better job of identifying the most similar texts, however the PPCM scores, which compares writing directly to the true preferences, clearly labels both Generation A. and Generation B. as "strongly exhibiting" the preference "use a haiku", and Generation C. "strongly contradicting" the preference.
the preference "use single short sentence". We then
Generation A.
A sausage in bun, Toppings vary near and far, Hot dog, rich in roots.
Generation B.
Grilled steamed sausage, In a bun with cheese spice, Street food's warm embrace.
Generation C.
A hot dog a sausage bun with various toppings such as mustard, ketchup, and relish
Pairing
Levenshtein Dist.
Norm. Levenshtein Dist. BERTScore
PPCM per generation
A-B
0.85
0.27
(2
A-C
0.76
0.20
(2, -2)
B-C
0.81
0.10
(2,
Table 10: Levenshtein distance between each pair of generations.
23

---

PROSE: Inferring User Preference Deseriptions
E.2. Editing Influences the User
In this next example, we demonstrate how the editing procedure can influence the output of the user. We again ask an LLM to summarize the encyclopedia article on hot dog. Let's assume, inferring agent has incorrectly inferred the preference "question answering style". Using this preference, the preference-conditioned agent writes:
What a hot dog?
A hot dog is dish made of sausage (typically a wiener or frankfurter) served a partially sliced bun. It can refer to the sausage itself, which is grilled, steamed, or boiled.
How is hot dog typically prepared?
Hot dog preparation varies, but common condiments include mustard, ketchup, relish, and onions. Some toppings can also include cheese sauce, sauerkraut, chili, bacon, or jalapeños.
Where did hot dogs originate?
The sausages used in hot dogs were culturally imported from Germany, and the dish hecame popular in the U.S. as a working-class street food. Over time, hot dogs became closely associated with American culture, particularly baseball.
Where are hot dogs most popular?
While they are especially associated with New York City, hot dogs became widespread across the United States throughout the 20th century, becoming an integral part of American cuisine.
We ask the user if the above is satisfactory for it's true preference of: "style targeted to young children". It does not, so we then ask the user to edit the above for true preference "style targeted to young children", and the user writes:
What is hot dog?
A hot dog is a yummy snack made of sausage (like wiener frankfurter) inside soft bun. Sometimes people call the sausage itself a hot dog, and it can be cooked by grilling, steaming, or boiling.
How is hot dog usually made?
Hot dogs can be made in different ways, but they often have yummy toppings like mustard, ketchup, relish, and onions. Some people like to add even more fun stuff like cheese, sauerkraut, chili, bacon, spicy jalapeños.
Where did hot dogs come from?
The sausages in hot dogs came from Germany, and a long time ago, they became super popular in the U.S. as quick and tasty street food for busy people. Now, hot dogs are big part of American culture, especially at baseball games!
Where are hot dogs the most popular?
Even though hot dogs are popular all over, they are especially loved in New York. Over the years, hot dogs have become favorite food all across the United States!
While the above summary does correctly follow the true preference, it is still formatted in a question answering style, which not part of the user's preferences. Le.. the preference-inferrig agent's error has directly impacted the user's writing. This another reason why PLUME removes the editing procedure.
For reference, if the user is directly asked to summmarize the encyclopedia article on hot dogs using it's true preference of "style targeted to young children", it outputs:
A hot dog is yummy food that's made with a sausage, like wiener or frankfurter, inside soft bun. You can cook the sausage in different ways, like grilling boiling. People love to put tasty toppings on their hot dogs, such as ketchup, mustard, cheese, and even onions!
Hot dogs came from Germany, but they became super popular in the US, especially baseball games. Now, you can find hot dogs all over the country, and in some places, like Chicago, people have special ways of making them!
24

---

PROSE: Inferring User Preference Deseriptions
F. Prompts
F.1. Preference Inference and Preferenee-Condittoned Agent Prompts
The prompts used by PROSE for writing sample generation and task completion in the PLUME environment are in Appendix Figure 7. The prompts used by PROSE infer the user preferences from user examples are provided in Appendix Figure for the PLUME environment.
System Prompt
You are an experienced writer. Adapt your writing to heavily emphasize the provided preferences.
User Prompt
You have the following preferences: <prefereence.desozippiin
Using these preferences, write short (summar) ry [mmal]] 1} about (this these) (article notes):
[START OF (ARTICLE [OTES]]
<task.content>
[END OF (ARTICLE [OTES}]
Encapsulate the (summary email 1 triple quotes
susmarry email}>
Figure 7: LLM prompts for the preference-conditioned agent and for task completion on the PLUME's summarization and e-mail writing tasks. The system prompt prepended to the user prompt following the LLM's chat template. "(...l..))" means that of the options is selected based the task and "<. >" indicates that the text is formatted from variable. inferred.preeeeeeee
refers to one of the inferred user preferences.
25

---

PROSE: Inferring User Preference Deseriptions
System Prompt
user is completing writing tasks. The user has an underlying set of preferences that explains why they write way they do.
User Prompt
Aggregation Task
are tasked to curate prompt to guide a specific style of writing. currently have the following list of preferences related to writing styles:
[<preference cercccedderriie ptiin ... wwnffrrredppreeeeeenee..
Unfortunately, these preferences may overlap contain redundancies. Please review the list and condense it by combining similar or overlapping preferences, ensuring that the distinct intent behind each one remains clear so that writer can easily follow them. Ensure the condensed list is concise, non-redundant, and preserves the original level of specificity. When applicable, preserve the exact wording. Return the revised preferences same format as the original list.
Inference Task
received a new task. The task is to (summarize write an email about) the following: <article notes>
We have previously identified the following preferences: pereference.descrippee Based on these preferences, we wrote this (summary email]: <assistant.outppu
However, this differs from the user's {sunmary emmai]. The user wrote this (summary ema i 1}: <user output>
Refine the list of preferences by adding, removing, updating preferences in order better imitate the user.
While refining the preference set, you should:
Identify and reason about differences between our writing and the user's writing.
Consider writing traits from distinct quirks to broader stylistic tendencies.
Provide concise set of preferences in the imperative form.
Be precise; make the fewest possible changes the preference set.
Do not qualify, dilute, soften existing preferences.
Refine only the preferences if clear difference exists. Otherwise, preserve the current preferences.
Provide a concise set of specific preferences in the imperative form. After reasoning, output the refined set of preferences on single new line and prefaced with "Preferences.".
Figure 8: LLM prompts for preference inference on PLUME's summarization and e-mail writing tasks. The system prompt is prepended to each user prompt following the LLM's s chat template. "(...I...}" means that of the two options is selected based on the task and "<.... indicates that the text is formatted from a variable. user refers to how the user completes the task, assis itsistand ant -oupput how the assistant completes the task, and inferredpreferencee to one of the inferred user preferences. Continued on next page.
26

---

PROSE: Inferring User Preference Descriptions
User Prompt
Preference Breakdown Task
You inferred the following preference string:
cenferred..reeeeeeeee wtw.ceeeeddeedeeeeeeeeeee
Format this preference into concise set of preferences. Format the final of preferences JSON list single line and prefaced with "Preferences:". Each element in the JSON list should be a string. The final output should look like:
Preferences:
[<preference 1>,.... <preference i>>....
Consistency Verification Task
Validate the following preference:
"<inffereed.prfeerecce.. against the following writing:
<user_output>
Does the writing confirm or contradict the preference? Select one of the following: strongly confirms the preference, somewhat confirms the preference, is neutral toward the preference, somewhat contradicts the preference, strongly contradicts the preference. Your final decision should be output on a separate line prefaced with "Verdict:".
Figure 8: LLM prompts for preference inference on the PLUME's summarization and e-mail writing tasks. The system prompt is prepended each user prompt following the LLM's chat template. "(...I-.)" means that of the two options selected based on the task and "<... >" indicates that the is formatted from variable. user.output refers to how the completes the task, assi sttnttttouttatoooo how the assistant completes the task, and inferred to one of the inferred user preferences.
27

---

PROSE: Inferring User Preference Deseriptions
F.2. Synthetic Human Prompts
The prompts used to have GPT-4o play the role of our synthetic human for PROSE are given in Appendix Figure 9. The "human" is instructed complete the task in the same way the preference-conditione agent when completing the writing tasks (see Appendix Figure 7).
System Prompt
You experienced writer. Adapt your writing to heavily emphasize the provided preferences.
User Prompt
You have the following preferences: <ground truth preference descriptions
Using these preferences, write a short fsummary [email] about {this these] farticle notes]:
[START OF (ARTICLE) [OTES]] <task.content>
[END OF (ARTICLE [OTEE]]]
Encapsulate the (sumnary email] in triple quotes
....
<
summaryy <maill>>
Figure 9: LLM prompts for the synthetic human on the PLUME's summarization and e-mail writing tasks. The system prompt is prepended to the user prompt following the LLM's chat template. "(...I..}} means that of the two options is selected based on the task and "<...> >" indicates that the text is formatted from a variable. inferreedpreference.. refers to one of the inferred user preferences.
...
28

---

PROSE: Inferring User Preference Deseriptions
F.3. Preference-Con Conditioon Agent Baseline Prompts
The prompts used in the no-preference baseline are in Appendix Figure 10 and for the in-context learning baseline are in Appendix Figure 11. For the in-context learning baseline, the number of examples matches the number of examples used when coalescing prevoiusly inferred prompts (see Appendix Figure 8).
System Prompt
You experienced writer.
User Prompt
Write short (summarry ry lemail) 1)
about (this (these) (article notes):
[START OF (ARTICLE [OTEE}]]
<task_content>
[END OF (ARTICLE [OTEE}]
Figure 10: LLM prompts for the no preference baseline in the PLUME environment. The system prompt is prepended to the user prompt following the LLM's chat template. "<...>" indicates that the text is formatted from a variable. task content refers to the content of either the article to be summarized or the notes to include in the email, depending on the sub-task.
29

---

PROSE: Inferring User Preference Deseriptions
System Prompt
You experienced writer. Adapt your writing to heavily emphasize the provided preferences.
User Prompt
You have previously observed the following examples:
Example 0:
{Article Notes}:
[START OF (ARTICLE ]OTES]]
<task_content>
[END OF (ARTICLE) [NOEE]]]
[Article Notes]:
1894
Ecompletion.r
Example
(Article Notes]:
(START OF (ARTICLE NOTES]]
ctask.conted
[END OF (ARTICLE ]OTES}]
[Article Notes]:
<completion
Using the same style as these examples, write short (summary lemail) about {this these) (article) notes}:
[START OF (ARTICLE) [NTEE]]] <task.content>
[END OF (ARTICLE [OTES]]
Encapsulate the (summary email] in triple quotes
< <usumary email]>
LLM prompts for the in-context learning baseline in the PLUME environment. The system prompt is prepended to the user prompt following the LLM's chat template. "<... >" indicates that the text is formatted from variable, and completio on_ refers to an example completion provided for in-context learning. task.content refers to the content of either the article to be summarized or the notes to include in the email, depending on the sub-task.
Figure 11:
30

---

PROSE: Inferring User Preference Deseriptions
F.4. Qualitative Verification Consistency Examples
Preference components after refinement
Most relevant true preference OR Notes Write in the style of children's s book
Write in a whimsical, playful, and narrative style using vivid and childlike imagery
Maintain hopeful, conversational, and informal tone
Write in the style of chi Idren's book
coneee Mention geographical context early in a simple manner
Overfit to specific example
Focus on the sequence of events with explicitly numbered steps labeled as 'first,' 'next,' 'then," "after that,' and "finally"
Adopt step-by-step structure
Use simple and metaphorical language for emotional aspects Include personll details about characters
Include simile
Overfit a specific example
Use ampersands for conjunctions
Use ampersands (&) instead of Mand''s Write in the style of children' book
Conclude with a practical reminder or lesson emphasizing support and teamwork
Uss--hetorical questions sparingly
Discarded because they were used more than sparingly
Capitalize key traits/actions for emphasis
Use ALLCAPS to emphasize certain words Adopt personn narrative
Use third-person perspective
Use emojis strategically
Write in the style of tweet Write in the style of tweet Write in the style of tweet Inrelevant
Use informal and playful language
Use hashtags to encapsulate themes
Minimize emotionally charged phrases
Use direct questions
Include rhetorical questions
Be-coneise and direct
Irrelevant
Focus emotional impact and highlight key themes
Incorrect, but related to the content of many chat forum posts Incorrect, but related to the content of many chat forum posts Include modern slang Adopt rhyming structure
Highlight internal conflict
direct address with casual and contemporary tone
include character interactions with informal dialogue and con- sistent rhyming couplets
employ a simplified screenplay format focusing on dialogue and voiceover
write in the style of screenplay
incorporate slang and playful language
Include modern slang
highlight excitement and stakes dynamically
Loosely related to screenplay
use thematic transitions between sections include metric prefixes explicitly
Write in the style of screemplay
Overfit to a specific example
Table 11: Qualitative examples Verification. Strikethrough indicates the preference component was pruned. Verification successfully removes overfit of irrelevant preferences. On occasion, it discards relevant, but misqualified components.
31

---

PROSE: Inferring User Preference Descriptions
E.5. Qualitative Iterative Refmement Examples
Refinement step
Inferred preference descriptions
True preferences
adopt a third person narrative, include rhetorical questions, use ALLCAPS to emphasize certain words, write in the style of tweet
Use rhetorical questions, capitalize for emphasis, be concise, include symbols or emojis, focus emotional impact.
2
Use rhetorical questions, capitalize for emphasis. be concise, include symbols or emojis, focus on emotional impact, use hashtags, use symbols for brevity.
3
Use rhetorical questions strategically. capitalize for emphasis, be concise, limit symbols emojis, focus emotional impact, use 1-2 hashtags, use symbols for brevity.
4
Use rhetorical questions strategically, capitalize for emphasis, be concise, limit symbols, focus on emotional impact, use 1-2 hashtags, use symbols for brevity, incorporate emojis for emphasis.
5
Use rhetorical questions strategically, capitalize for emphasis, be concise, use "&" for brevity, focus on emotional impact, use 1-2 hashtags, fewer emojis for emphasis, highlight key themes.
True preferences
adopt a question guestion-answering answering style structure, include personifications, use archaic language, write in the style of a podcast
Use poetic and narrative style with vivid imagery and metaphor; employ archaic language and conversational tone; structure writing like narrative script; directly address the audience enhance engagement.
2
Use a podcast or broadcast format with vivid storytelling imagery and metaphor: employ slightly modern archaic language and conversational tone: structure writing as a continuous narrative; directly address the audienee enhance engagement.
3
Use podcast broadcast format with named episodes; employ consistently archaic language with varied vocabulary; use thematic and specific metaphors; enhance audience engagement with direct questions and conversational elements; structure writing with clear narrative are and defined sections.
4
Use podcast or broadcast format with creatively thematic episode titles; employ consistently archaic and poctic language with varied vocabulary; with rhetorical style and subtle questions: structure writing with clear narrative are and poctic conclusions. Use a podcast or broadcast format with creatively thematic episode titles evoking transtormation and mystery: employ consistently archaic language throughout; use vivid and personified metaphors; enhance audience engagement with frequent rhetorical questions; structure writing with clear narrative are and conclude with harmony and enlightenment.
vivid and personified metaphors; enhance audience engagement
5
True preferences
adopt a second person narrative, include onomatopoeias, use imagery, write in the style of old timey radio Use vivid imagery and metaphor, adopt narrative style, address the reader directly, and create an immersive experience.
2
Use vivid imagery and metaphor, adopt narrative style, address the reader directly with a conversational and auditory tone, create an immersive and nostalgic experience.
3
Use vivid imagery and metaphor, adopt narrative style, address the reader directly with conversational and auditory tone, create an immersive experience with a focus on auditory imagery and live storytelling. Use vivid auditory imagery and metaphor, adopt narrative style reminiscent of radio broadcast, address the reader directly with conversational and auditory tone, create an immersive experience with a focus on auditory storytelling.
4
5
Use vivid auditory imagery and metaphor with nostalgic and whimsical tone. adopt a narrative style reminiscent of classic radio broadcast, address the reader directly with a conversational and auditory tone, immersive experience with focus on auditory storytelling.
Table 12: Qualitative examples of Iterative Refinement. Bold indicates modifications performed by the refinement step. Note: the true preferences are for reference only, the initial refinement step is conditioned on empty preference deseription.
32


Reviews:
[{"review_id": "6RIneGZmzi", "reviewer": "Reviewer_wsxk", "summary": "This paper is an extension work following Gao, 2024. Agent alignment is achieved by conditioning on an inferred description of user preferences. Yet, existing methods often lead to generic descriptions that fail to capture the unique, individualized aspects of human preferences. To address this limitation, this paper introduces PROSE, a novel technique that enhances the precision of preference descriptions derived from user writing samples. To overcome the challenges in prelude, this paper proposes iterative refinement of preferences and verification framework.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and questions. We appreciate you finding PLUME to be effective and PROSE to be efficient and effective. \n\n**Response to “weaknesses”**\n\n(1) Our contributions are spread across our algorithmic developments in PROSE and our benchmark improvements in PLUME.  \n\nIn addition to improving the quality of PLUME’s evaluation protocols and prompting, we validated and improved the ground truth preferences to ensure the proxy humans are both sensitive to them when “writing” and when “evaluating”. If the proxy humans are not sensitive to the ground truth preferences then there is little signal in the training data and the results are meaningless. Additionally, we adapted PLUME to provide demonstrations as the data source from which to learn from, not edits. \n\nOur algorithmic contributions demonstrate the importance of validating inferred preferences across all relevant samples and iteratively refining the inferred preference description per sample. CIPHER (Gao et al., 2024) does neither. \n\n(2) Thank you for raising this concern and we will make sure the distinction is made clear in the paper. CIPHER includes a refinement step when observing a new sample, at which point they retrieve relevant samples, combine their preference, and refine this aggregated preference using the new sample. PROSE similarly does this aggregation step across retrieved examples. However, PROSE’s contribution is adding iterative refinement, where the preferences are refined several times using the same sample during preference description inference. This novel algorithmic contribution is distinct from CIPHER’s single pass method, and provides an 11% improvement on average. In addition to the iterative refinement step, PROSE’s contribution is the preference verification step. Reviewer CfWS describes these contributions as adapting the preferences learning process “from a 1-shot process to a progressive refinement problem.”  \n\n**Response to questions**\n\n(1) Yes, this is handled in the example retrieval step, which is not a contribution of PROSE, therefore it is not discussed in detail. Given sufficient context about both the demonstrations and the current task, only those preferences relevant to the current context will be retrieved. Therefore, if the task states the email is to be written to my boss then preferences that were learned from emails to my boss will be retrieved. \n\n(2) We find that PROSE performs better by 33% due to our algorithmic contributions (iterative refinement and preference verification) and improved prompting. We find the biggest improvements relative to CIPHER are after the first step suggesting PROSE is more data efficient (cf. Appendix C.4. Figure 6). Additionally, we adapted PRELUDE to be able to provide demonstrations for an algorithm to learn from. \n\nWe also improved PLUME’s ground truth preferences compared to PRELUDE such that the proxy humans are sensitive to them both when “writing” and “evaluating”. Additionally, when selecting the preference sets, we validated that they are not default behaviors in the LLMs, such as preferences like “brief” and “use bullet points” in PRELUDE. Having preferences that are default behaviors limits the utility of generation quality metrics for evaluating an algorithm’s ability to personalization as no personalization is needed.  When combined with our metric improvements, PLUME provides more meaningful results and evaluations.  \n\n**Response to missing related work**\n\nThank you for the feedback that more of a discussion of model alignment methods in the space of inference time interventions. We will make sure to expand our related work section to include these.", "claims_and_evidence": "Yes. The claims made on the experiment results and comparison are supported by evidence.", "methods_and_evaluation_criteria": "Yes. This work directly compares to the relevant prelude benchmark, and no learning, ICL.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "Yes. I have checked the experiment setting, benchmark and settings. The experiment design is similar to prelude benchmark and it makes sense.", "relation_to_broader_scientific_literature": "This work is based on the prelude framework [1]. This work improves the evaluation metric, editing process and adds a verification procedure. The improvement makes the user edit framework more efficient.\n\n[1] Gao, Ge, et al. \"Aligning llm agents by learning latent preference from user edits.\" arXiv preprint arXiv:2404.15269 (2024).", "essential_references_not_discussed": "There is no literature missing directly related to the proposed framework. However, I would encourage to include more model alignment works in the paper. There are a few recent works focus on alignment without introducing additional trainable models. (Please see following)\n\n[1] Li, Kenneth, et al. \"Inference-time intervention: Eliciting truthful answers from a language model.\" Advances in Neural Information Processing Systems 36 (2023): 41451-41530.\n\n[2] Turner, Alexander Matt, et al. \"Activation addition: Steering language models without optimization.\" arXiv e-prints (2023): arXiv-2308.", "other_strengths_and_weaknesses": "Strength:\n\n1. This paper directly points out the weakness of the prelude framework. I have tested the code base of prelude before, and discovered the similar problems (metric, editing process) in the prelude as the authors pointed out in the paper. Thus, I believe this work's improvement over prelude is important and makes sense.\n\n2. The proposed methods to improve the prelude is effective and can be directly measured by the common evaluation methods.\n\n3. The PROSE framework is efficient and effective compared to other alignment methods in literature, e.g, reward model training, fine-tuning.\n\nWeakness:\n\n1. The proposed PROSE framework heavily relies on prelude, which lacks novelty. The improvement over prelude basically focuses on introducing better evaluation metric, better process and prompting for editing, with additional verification process. However, there is **no fundamental change** compared to prelude. This makes the work seems to have limited contribution.\n\n2. This work seems to claim the refinement procedure is novel (From line 106 to the end of the paragraph). In fact, the prelude framework also includes the refinement of inferred preferences. If you check the code of prelude, it includes a step to aggregate the currently inferred preference with the previous ones.", "other_comments_or_suggestions": "I believe one challenge existing in prelude are not solved in the proposed PROSE framework: PROSE and prelude assumes the user's preference towards a topic is static. This makes the alignment easier than realistic case. However, in practice, the preference can depends on both topic and occasion, e.g, email to co-worker or manager? I would suggest use more complicated dataset, where the retrieval is not only based on the similarity between questions (or topic), but also based on occasions.", "questions_for_authors": "1. As discussed in the previous part, is it able to extend the PROSE from static preference to varying preference depending on different occasions?\n\n2. Besides the improvement in metric, prompting, and verification process, is there any **structural or fundamental** advantage over prelude? For example, can adapt to user preference in fewer rounds, or saving user editing effort.\n\nI could consider changing scores if my concerns are addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XhhqGpzES9", "reviewer": "Reviewer_Wbtm", "summary": "The paper introduces PROSE (Preference Reasoning by Observing and Synthesizing Examples), a method for aligning large language models (LLMs) with user preferences inferred from writing samples. It improves upon previous sota/baselines on various aspects, proposes new metrics and preference frameworks, and provides a new dataset for writing preference alignment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and questions. We appreciate that you find the problem important, our evaluation thorough, and both our algorithm and benchmark improvements to be meaningful contributions. \n\n**Response to questions/”weaknesses”**\n\n**(1a)** Thank you for letting us know the table is confusing. We will expand the caption to include the information you recommend to specify each acronym and include the main message. We will also add the formula for how we calculated correlation: COV(X, Y) / (std(X) * std(Y)) where COV is covariance, std is standard deviation, X is the preference similarity scores and Y is the generation quality scores. \n\n\n**(1b)** Thank you for raising this question. We realize the issue is with how we present our evaluation metrics to imply that using PPCM is a given. Instead, we used the correlation test to select both PPCM and P.Sim. We tested a variety of different metrics, including those used by PRELUDE (which have a low correlation), prior to selecting PPCM (see Appendix C.1 Table 3). We selected the generation and preference inference metrics with the highest overall correlation. \n\nWe want the measures of generation and preference inference quality to be correlated as correctly inferred preferences should lead to high-quality generations. For example, generating text conditioned on the ground truth preferences should have a higher generation score than when conditioning on the preferences for another user, no preference, or an incomplete preference set.  \n\nWe don’t want the two metrics to be orthogonal because they are measuring the quality of two different, but high correlated, things -- the inferred preferences versus the generation quality. We want to make sure both closely align with the user’s true preferences. Therefore, we are measuring both according to the same attribute. \n\n\n**(2)** Assigning each user the same number of preferences was a design decision we made so all users are similarly difficult. We did not want a scenario where a model could perform very well in aggregate by focusing only on those users with the fewest number of preferences to learn. \n\nBy shared structure we mean each preference set has preferences over shared attributes for the writing. For example, format (e.g. screen play versus tweet), tone of voice (e.g. inquisitive versus sarcastic), or use of literary tools (e.g. rhyming versus alliteration). This design decision was also made to create user preference sets that were similarly challenging.  \n\nAdding additional preferences and structures to the preference sets is straightforward to add and adapt the benchmark, which we encourage future researchers to do. \n\n\n**(3)**  We have run a human evaluation with 16 participants (3 are ML researchers; 9 women and 7 men; age in [19, 58]). Participants completed a within subjects AB test comparing PLUME+ICL generations to ICL generations and PLUME+ICL generations to CIPHER generations.  \n\nParticipants evaluated LLM generations for two different preferences for the email task and for the summarization task for a total of 20 survey items per method comparison. We used the responses to compute a win rate for PLUME+ICL compared to each of ICL and CIPHER. The only difference between the two comparisons is the LLM generations compared. \n\nFor PLUME+ICL versus ICL, we see an average win rate of 69.4%. For PLUME+ICL versus CIPHER, we see an average win rate of 91.8%. The human evaluation results support our synthetic evaluation results.", "claims_and_evidence": "The main claims of the paper and their supporting evidence are:\n\n- PROSE produces more precise preference descriptions than CIPHER – This is supported by experiments on PLUME, showing a 33% improvement in preference alignment.\n\n- Iterative refinement improves alignment between LLM generations and user preferences – Ablation studies demonstrate that increasing the number of refinement steps improves performance by 14.8%.\n\n- Consistency verification enhances robustness – Though its effect is smaller (1.5–1.7% improvement), the verification step helps filter irrelevant or overfit preferences.\n\n- PROSE and ICL are complementary – Combining PROSE with ICL leads improvement.\n\n- improved dataset PLUME - The authors analyze PRELUDE’s limitations (e.g., weak correlation between inferred preferences and generation quality) and show that PLUME provides more consistent evaluation metrics.\n\nThese claims are well-supported by empirical results. The arguments are well-written and limitations are discussed.", "methods_and_evaluation_criteria": "See question 1b & 2.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "No notes, well done. I have a question about the setup (see question 3.)", "relation_to_broader_scientific_literature": "This work builds on previous efforts in preference learning for LLM alignment, particularly CIPHER (Gao et al., 2024) and in-context learning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Overall the paper is well-motivated and well-written. I believe the problem is an important one, the evaluation thorough, and this paper not only improves upon previous methods but also provides a better benchmark than PRELUDE. Graphs are nice and clear. The discussion on limitation addresses questions about computational costs. \n\nFor weakness, please see questions for authors section.", "other_comments_or_suggestions": "Typo: line 45 left \"share share\"", "questions_for_authors": "(1a). I was confused about table 1 until I read through page 5. Please write out all the acronyms out in the caption, and explain the main message of table 1 (Preference similarity is a better metric?)  Also, please provide the formula of how you calculated the correlation for checking statistical validity. \n\n(1b). The point of the R correlation test is also unclear to me - since you are using the PPCM metric already, why do you want another metic that highly correlates with it? Since they are testing different thing (preference vs. generation quality), don't you want them to be as orthogonal as possible? (i know in reality a good response usually scores high on both, so empirically it's hard to show. But I wonder the reasoning behind the test.)\n\n2. About preference set specification on page 5 bottom of left column. Is equal # of preference & the same structure necessarily a good thing? Wouldn't that limit the generality of the benchmark?\n\n3. Would PROSE still be effective if applied to real users instead of synthetic GPT-4o proxies? Have you considered testing with human participants to validate that inferred preferences align with subjective user expectations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5tiKS5kzIk", "reviewer": "Reviewer_CfWS", "summary": "This paper introduces PROSE, a method for inferring precise and personalized user preferences to enhance LLM-based writing agents. The approach employs iterative refinement and cross-sample verification to generate more accurate preference descriptions compared to existing techniques. Alongside PROSE, the authors introduce PLUME, a new benchmark specifically designed for evaluating preference learning in assistive writing tasks. Through experiments on summarization and email writing tasks across multiple LLMs, PROSE demonstrates a significant improvement over the state-of-the-art CIPHER method for preference inference. Additionally, the author demonstrate that combining PROSE with in-context learning (ICL) lead to further performance gains.\n\n### update after rebuttal\nI appreciate the authors' detailed responses and the addition of the human evaluation. I will maintain my current score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and questions. We appreciate that you find PLUME and our comprehensive evaluation to be valuable contributions along with our learning about the importance of iterative refinement. \n\n**Response to questions**\n\n*Re your question about sorting preference components:* We also tried sorting alphabetically and observed similar performance differences. The impact of ordering has been previously identified in work on multiple choice QA tasks [1], and we think this is related. Additionally, we hypothesize the LLM is doing some level of prompt engineering on itself when it infers the user’s preferences, and that the preference ordering is part of this. As we know from the prompt engineering literature, small changes to the prompts can have a large impact on LLM generations. \n\n*Re your question about conflicting preferences, context sensitivity, and changing preferences:* The issues of contextual preference variations and preference evolution over time can be solved with the example retrieval step. By retrieving the most contextually relevant and temporally local examples from memory, PROSE should be able to handle preference drift and context variations. This then largely becomes a problem of constructing a sufficiently detailed context. However, for this paper, we identify the retrieval step as an orthogonal issue to inferring preference descriptions, so do not tackle it. \n\n**Response to \"weaknesses”**\n\n*Re limited task selection:* Our task selection is based on those used in previous work and specifically targeting “write on my behalf” tasks. The other types of tasks that exist in the LLM literature, such as dialogue or question answering, do not fall under “write on my behalf”. The “write on my behalf” task is a common one people ask LLMs to do, which motivated our selection of it. The approach of learning from demonstrations we take with PROSE is not relevant to other tasks such as question-answering or dialogue because in those other tasks the user is asking the LLM to complete tasks they have not done. \n\n*Re human evaluation:* Thank you for this feedback. We have run a human evaluation with 16participants (3 are ML researchers; 9 women and 7 men; age in [19, 58]). Participants completed a within subjects AB test comparing PLUME+ICL generations to ICL generations and PLUME+ICL generations to CIPHER generations.  \n\nParticipants evaluated LLM generations for two different preferences for the email task and for the summarization task for a total of 20 survey items per method comparison. We used the responses to compute a win rate for PLUME+ICL compared to each of ICL and CIPHER. The only difference between the two comparisons is the LLM generations compared. \n\nFor PLUME+ICL versus ICL, we see an average win rate of 69.4%. For PLUME+ICL versus CIPHER, we see an average win rate of 91.8%. The human evaluation results support our synthetic evaluation results. \n\n**Response to missing related work**\n\nThank you for the feedback that our related work would benefit from discussing DITTO and \"Unsupervised Human Preference Learning\". We agree there are important parallels to discuss with both papers. We have referenced DITTO in the introduction as another method for learning from demonstrations, and agree it should also be mentioned in the Related Work section. We will additionally reference and discuss \"Unsupervised Human Preference Learning\" in future revisions of the paper, as we believe their method and PROSE are complementary; PROSE would likely benefit from leveraging a model that is directly trained to infer preferences, and, as their method also revolves around reducing the delta between generic and user responses, their M_L model would likely benefit from the iterative refinement and consistency verification proposed in PROSE. \n\n \n\n[1] Pezeshkpour, P., & Hruschka, E. (2023). Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308.11483.", "claims_and_evidence": "The paper provides convincing evidence through quantitative experiments, with tables 1-2 and figures 2-3 showing consistent performance improvements across different LLMs and writing tasks. Additionally, the ablation studies and analysis provide convincing evidence of the iterative refinement component's effectiveness. Finally, fig. 2 and tab.2 show that PROSE combined with ICL provide additional benefit and this is consistent across different models and taks.\n\nThe paper makes broad claims about generalizability but this would require evaluation on more diverse writing tasks and not only summarization and email writing tasks. Also, the paper states that consistency verification \"prunes irrelevant preferences\" but provides limited quantitative evidence on how effectively it identifies truly irrelevant vs. simply uncommon preferences. Finally,the evaluation relies heavily on automated metrics. While these are the standard, an additional human evaluation would strengthen claims about real-world preference alignment and writing quality improvements.", "methods_and_evaluation_criteria": "I find that the methods and evaluation criteria/metrics are well aligned with the problem of inferring anf applying user wiriting rpeferences. The method addresses limitations of previous approaches, and evaluates them in a systematic manner. Additionally, the creation of PLUME as a specialized benchmark for this task is particularly valuable, though expanding task diversity and incorporating more human evaluation would further strengthen the assessment of the method's practical utility.", "theoretical_claims": "The paper does not present theoretical claims due to the empirical nature. The paper describes the iterative refinement process that continues until either the candidate solutions exactly match the user demonstrations or a maximum number of iterations is reached, but does not make theoretical claims about convergence guarantees.", "experimental_designs_or_analyses": "The experimental design is generally sound and appropriate for evaluating the proposed method. The PLUME benchmark provides a needed framework for evaluating preference learning. The most significant validity concern is the reliance on synthetic preferences and LLM-as-judge evaluation, which are practical compromises but may not fully represent real human preference dynamics. The paper acknowledges these limitations appropriately, which strengthens the overall scientific integrity of the work.", "relation_to_broader_scientific_literature": "Besides the evident ones, e.g. PROSE extends CIPHER in inferring preferences directly from demonstrations rather than explicit feedback connecting to the literature for learning from implicit signals, this work is connected to the body of work on personalization through preference modeling, self-refinement in LLMs, reflection in LLMs and somehow to writing style transfer.", "essential_references_not_discussed": "The paper does not discuss parallels with these two works that appear to be somewhat relevant. \"Show, Don't Tell: Aligning Language Models with Demonstrated Feedback\" Shaikh et al published in ICLR 2025. This is very recent but the arxiv version is from June 2024.\nThis work introduces DITTO, a method that directly aligns language models to user-demonstrated behaviors using a very small number of demonstrations as feedback. Like PROSE, DITTO focuses on learning from user demonstrations rather than explicit feedback, but employs online imitation learning principles, providing an alternative theoretical framing. DITTO specifically addresses personalization in emails and other writing tasks that directly overlap with PROSE's evaluation domains. DITTO reports substantial improvements (19% points on average) over few-shot prompting and other methods. PROSE should have compared against or at least discussed this performance benchmark.\n\nAnother work is \"Unsupervised Human Preference Learning\" by Shashidhar et al, 2024 from ACL 2024. This paper proposes using small parameter models as preference agents to generate natural language rules that guide larger pre-trained models. Like PROSE, this work leverage natural language descriptions of preferences to guide language model outputs, rather than direct parameter updates and aims to achieve personalization without requiring full model fine-tuning, highlighting efficient approaches to customization.", "other_strengths_and_weaknesses": "__Strengths__:\n- the idea of combining iterative refinement and coss-sample verification lead to perform preference learning from a 1-shot process to a progressive refinement problem\n- the paper is well written and easy to follow\n- comprehensive evaluation and PLUME is a valuable contribution\n- the 33%improvement over CIPHER demonstrate the validity of this approach\n\n__Weaknesses__:\n- there is a limited task selection, only summariation and email writing. this leave the open question of how well PROSE generalizes to other types of texts.\n- The evaluation relies entirely on automated metrics without human assessments of quality or preference alignment, leaving open questions. Although the need for a full-scale human evaluation is listed in the limitation sections, a small sample submitted to human evaluation could help better evaluate the claims of the paper.", "other_comments_or_suggestions": "see questions.", "questions_for_authors": "- You mention that sorting preference components by length before aggregation led to an 11% performance drop, which is an intriguing finding. Have you explored other ordering strategies for preference components, and do you have insights into why component ordering has such a significant impact on performance?\n\n- How does PROSE handle cases where user demonstrations contain genuinely conflicting preference signals?\n-if a user writing preferences evolve over time or vary depending on the context or the topic, how might PROSE be adapted to handle preference drift or contextual preference variations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper provides an improvement over a paper by Gao et al. 1] Gao, Ge, et al. "Aligning llm agents by learning latent preference from user edits." arXiv preprint arXiv:2404.15269 (2024).  To quote the reviewrs: Overall the paper is well-motivated and well-written. I believe the problem is an important one, the evaluation thorough, and this paper not only improves upon previous methods but also provides a better benchmark than PRELUDE. Graphs are nice and clear. The discussion on limitation addresses questions about computational costs.

There is no reason not to accept this paper - there are no major flaws and it is a solid contribution advancing the state of the art.  All reviewers agree with accept, the only question is on impact and importance.  The contribution is seen as incremental.  

Here are the strengths of the paper as contributed by the reviewers:

.	This paper directly points out the weakness of the prelude framework. I have tested the code base of prelude before, and discovered the similar problems (metric, editing process) in the prelude as the authors pointed out in the paper. Thus, I believe this work's improvement over prelude is important and makes sense.
.	The proposed methods to improve the prelude is effective and can be directly measured by the common evaluation methods.
.	The PROSE framework is efficient and effective compared to other alignment methods in literature, e.g, reward model training, fine-tuning.
•	PROSE produces more precise preference descriptions than CIPHER – This is supported by experiments on PLUME, showing a 33% improvement in preference alignment.
•	Iterative refinement improves alignment between LLM generations and user preferences – Ablation studies demonstrate that increasing the number of refinement steps improves performance by 14.8%.
•	Consistency verification enhances robustness – Though its effect is smaller (1.5–1.7% improvement), the verification step helps filter irrelevant or overfit preferences.
•	PROSE and ICL are complementary – Combining PROSE with ICL leads improvement.
•	improved dataset PLUME - The authors analyze PRELUDE’s limitations (e.g., weak correlation between inferred preferences and generation quality) and show that PLUME provides more consistent evaluation metrics.


The weaknesses of the paper focus on the impact of the contribution.  All reviewers agree that it is a contribution, the debate is around how much of a contribution it is:

"The improvement over prelude basically focuses on introducing better evaluation metric, better process and prompting for editing, with additional verification process. However, there is no fundamental change compared to prelude. This makes the work seems to have limited contribution."

 "I would suggest use more complicated dataset, where the retrieval is not only based on the similarity between questions (or topic), but also based on occasions." 

"This work seems to claim the refinement procedure is novel (From line 106 to the end of the paragraph). In fact, the prelude framework also includes the refinement of inferred preferences. If you check the code of prelude, it includes a step to aggregate the currently inferred preference with the previous ones."

This paper received three reviews all of which indicate "weak accept."  Despite a lack of discussion, all three reviewers acknowledged the rebuttal and two responded.  Two reviewers responded with one sentence comments.  One responded that they would raise their score in response to the authors rebuttal from weak reject to weak accept and one responded that they would maintain their score after reading the authors rebuttal.
Decision: Accept (poster)