Aligning Language Models with Demonstrated Feedback
Authors: Omar Shaikh
Michelle S. Lam
Joey Hejna
Yijia Shao
Hyundong Justin Cho
Michael S. Bernstein
Diyi Yang
Venue: ICLR 2025 Poster (2025)
Keywords: personalization
few-shot learning
human computer interaction
alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We highlight the effectiveness of using a very small number of demonstrations (<10) for task or user-specific alignment; and contribute a method that iteratively aligns an LLM to a user’s demonstrations by treating default outputs as dispreferred.

Abstract: Language models are aligned to emulate the collective voice of many, resulting in outputs that align with no one in particular. Steering LLMs away from generic output is possible through supervised finetuning or RLHF, but requires prohibitively large datasets for new ad-hoc tasks. We argue that it is instead possible to align an LLM to a specific setting by leveraging a very small number ($<10$) of demonstrations as feedback. Our method, Demonstration ITerated Task Optimization (DITTO), directly aligns language model outputs to a user's demonstrated behaviors. Derived using ideas from online imitation learning, DITTO cheaply generates online comparison data by treating users' demonstrations as preferred over output from the LLM and its intermediate checkpoints. We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts. Additionally, we conduct a user study soliciting a range of demonstrations from participants ($N=16$). Across our benchmarks and user study, we find that win-rates for DITTO outperform few-shot prompting, supervised fine-tuning, and other self-play methods by an average of 19\% points. By using demonstrations as feedback directly, DITTO offers a novel method for effective customization of LLMs.

Published as a conference paper at ICLR 2025
ALIGNING LANGUAGE MODELS WITH DEMONSTRATED FEEDBACK
Omar Shaikh"
Michelle S. Lam" Stanford University m1 am4 @cs. stanford.edd
Joey Hejna"
Stanford University osha ah@stanford.eddu
Stanford University
jhejna @cs stanford.
Yijia Shao
Hyundong Cho USC
Michael S. Bernstein Stanford University
Diyi Yang
Stanford University
Stanford University
ABSTRACT
Language models are aligned to emulate the collective voice of many, resulting in outputs that align with no one in particular. Steering LLMs away from generic output is possible through supervised finetuning or RLHF, but requires prohibitively large datasets for new ad-hoc tasks. We argue that it is instead possible to align an LLM to specific setting by leveraging a very small number (< 10) of demon- strations as feedback. Our method, Demonstration I'Terated Task Optimization (DITTO), directly aligns language model outputs to user's 's demonstrated behav- iors. Derived using ideas from online imitation learning. DITTO cheaply generates online comparison data by treating users' demonstrations preferred over output from the LLM and its intermediate checkpoints. Concretely, DITTO operates by having an LLM generate examples that are presumed be inferior expert demonstrations. The method iteratively constructs pairwise preference relation- ships between these LLM-generated samples and expert demonstrations, potentially including comparisons between different training checkpoints. These constructed preference pairs are then used to train the model using a preference optimization algorithm (e.g- DPO). We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts. Additionally, we conduct user study soliciting range of demonstrations from participants (N 16). Across our benchmarks and user study, we find that win- rates for DITTO outperform few-shot prompting, supervised fine-tuning, and other self-play methods by an avg. of 19% points. By using demonstrations as feedback directly, DITTO offers novel method for effective customization of LLMs]
INTRODUCTION
Large language models (LLMs) are trained for general-purpose use. In practice, however, they are often applied to very specific tasks for very specific users. Consider task as simple writing an email: our preferred email depends on personal writing style, the specific email task, or the target audience (a friend, stranger, etc.). As a result, there can be mismatch between the universal style (Santurkar et al.] 2023 Chakrabarty et al. 2023) trained into an LLM via instruction and preference tuning, and the specific style needed for applications. LLM outputs feel unopinionated and generic because of this mismatch.
While existing approaches such as supervised or preference finetuning are effective, they can require a large corpus of (un)acceptablee behavior (on the order of 2 1K samples (Zhou et al.) 2024 Ouyang ) which in turn requires unrcasonably high effort from an individual. RLAIF methods like Constitutional AI (Bai et al. 2022) automate pairwise preference collection with an LLM, but align models to general principles that may not capture fine-grained preferences. Although prompting is data efficient, finding an effective prompt can be tedious end-users often rely on brittle prompting
al. 2022)),
heuristics (Zhou et al. 2022 Zamfirescu- Samfirescu-Pereira et al. 2022). How might we efficiently communicate preferences and align language model to new individual or task?
"Equal Contribution
Code: https//githuh.coo//AA..NI hwtp:///w..........eeoooeoenccn.

---

Published as a conference paper at ICLR 2025
Demonstration ITerated Task Optimization (DITTO)
LM Generations
User Demos (n<10)
t=k
tk-1
t=0
LM
A
LM
A
>
LM
A
LM
A
B
B
B
C
C
Figure 1: DITTO iteratively aligns LLMs to demonstrated behavior. When user supplies demonstrations (through edits to a model's output, past preferred interaction history, or writing examples from scratch), DITTO treats these demonstrations as preferred all model behavior, including earlier iterations of the trained model. Using demonstrations as feedback allows for cheap generation of online comparison data and enables few-shot alignment with just handful of samples.
This paper introduces a framework for aligning LLMs to specific settings by providing small number of demonsirations (Fig. [ Rather than using prompts, principles, or pairwise preferences, we show that we can achieve strong alignment with individuals by leveraging a small number of user-provided examples of desired behavior. These examples can be drawn from a user's existing interaction logs, or from direct edits made to LLM outputs. Our approach, DITTO, scaffolds a handful of these demonstrations (< 10) into substantial dataset of preference comparisons, by treating users' demonstrations as preferred over model output from both the original LLM and models' earlier training iterations. This augmented dataset of iinoonstrationgroonddee to update the language model using an alignment algorithm like DPO (Rafailov et al. 2023). We additionally show that DITTO can be interpreted as an online imitation learning algorithm, where data sampled from the LLM is used to distinguish expert behavior. This perspective allows us to prove that DITTO can extrapolate beyond the performance of the demonstrator ($3).
comparisons can then be used
Since DITTO focuses on user/task- specific alignment, we benchmark DITTO through (1) an eval- uation on datasets of -pehor-ppecific writing ($4.1) and (2) user evaluation ($4.)) on real-world tasks defined by human participants. Our author-specific datasets include writing from blog posts to emails to articles. We find that win rates for DITTO outperform methods like SFT (avg. 11% pt. increase), self-play methods like SPIN (20.2% pt.) and few-shot prompting (33.4% pt.) on Mistral 7B. DITTO's advantage holds even when few-shot prompting is done on a more powerful LLM (GPT-4, 18% pt.). Next, conduct user study (N 16), asking individuals to edit generations from GPT-4 in an email-writing task. We use finalized demonstrations as inputs for DITTO. In these realistic user evaluations, DITTO's advantage becomes clearer: DITTO continues outperform baselines, including few-shot prompting (23.9% pt.), user-constructed prompts (27.9% pt.), and SFT (12% pt.). Finally, in a direct comparison between demonstratinns and pairwise feedback, we show that using demonstrations with DITTO is an order of magnitude more sample-efficient for individuals than soliciting pairwise preferences.
RELATED WORK
LLMs and Preference Finetuning. Large language models trained vast amounts of data have been known to perform well with careful prompting (Brown et al.
2020b Wei et al. 2022). Prompting, 2023) to design and often sensitive to
however, can be incredibly tedious (Zamfirescu-Peret Pereira al.
variations. Thus, it has become necessary to either finetune these models on large curated instruction following datasets (Mishra al.
2022 Thoppilan et al. 2022: Chung et al., 2022) and/or employ RLHF, where the LLM is trained to maximize a reward function learned from human preferences as a contextual bandit (Ziegler et al. (2019). Typically, this is done using policy-gradient style methods
(Williams 1992 Schulman et al.
2017) though more recent works Iearn directly from preference data
(Rafailov et al. 2023 Hejna et al.
|
[2024]]] Azar et al. 2023). While these methods are effective at tasks
like summarization (Stiennon et al. 2020 Wu & Hu 2018 Wu et al. (024)) and instruction following
(Ouyang et al. 2022: Nakano et al. 2021) they require thousands to hundreds of thousands of paired comparisons to obtain quality estimate of reward. This makes them prohibitively expensive for a wide range of applications, such as training a customized writing assistant or building a domain- specific chatbot. Group Preference Optimization (GPO) (Zhao et al.) 2023) takes promising step towards few-shot alignment of LLMs; however, preference groups must be pre-defined for meta- learning, which requires large dataset. On the other hand, Gao al. (024)) uses direct edits to distill
2

---

Published as a conference paper at ICLR 2025
latent preferences into prompt-based principles. In place of principles or pairwise feedback, DITTO directly learns preferences from a set of demonstrations, similar to model editing from canonical examples (Hewitt et al. 2024). Drawing from prior studies on programming by demonstration and end-user programming in HCT (Cypher) 1991; Cypher & Halbert 1993), our work aims at soliciting feedback at finer-grained level than binary preferences, principles, es, or prompts.
Self-Improvement. Recent works use iterative sampling to improve LLMs. Aproaches like STaR (Zelikman et al. 2022 2024 Anduuuri et al. 2024) are supervised by verifying the correctness of outputs, while Yuan et al. (2024) and Burns et al. (2023) use (potentially stronger) language models as crities. Unlike these approaches, DITTO does not require external signals besides demonstrations, similar to self-play methods like SPIN (Chen et al. 2024). Unlike
uses thousands of demonstrations and is targeted more towards SFT scale Sataaets-DrTye is designed for fast adaptation in the data-limited setting and thus has few key distinctions. Namely, DITTO does not update the reference policy and uses intermodel comparisons to combat overfitting. We found these changes to be important to obtain good performance with only a handful of demonstrations. In data-abundant settings, other works have shown that an oracle reward function (Gulcehre et al. 2023) model (Lee et al. 2023; Song al. 2024) is sufficient to provide feedback. We consider tasks like personalization, for which there is no abundant data or oracle.
Online Imitation Learning. DITTO builds online imitation learning. which appeals to the long-standing success of learning reward functions from comparisons (Fürnkranz et al. 2012 Akrour et al. 2012). Brown et al. (2099) first showed that with ranked demonstrations,
could improve policy beyond the demonstrator's performance. Follow-ups used automatic noise injection to remove
human rankings (Brown al. 2020a).. Other contemporary approaches online imitation learning
are based on adversarial games between reward and policy players (Ziebart et al. 22008 Ho & Ermon
2016). In our case, we use KL-constrained formulation, like Watsoon et al. (2023). p. Sikchi al.)
(2022) generalizes the adversarial game to a ranking game and thus uses generated comparisons like
DITTO. Unlike DITTO, however, these approaches explicitly require learning a reward function and
are designed for continuous control- not for L.Mss.
DITTO
While prior work uses thousands of comparisons to align LLMs, DITTO instead uses only a handful of expert demonstrations to alter model's behavior. This type of cheap, rapid adaptation enabled by our core insight: that online comparison data can be easily obtained from demonstrations.
3.1 NOTATION AND BACKGROUND
A language model can be viewed as policy π [ty (z) that produces a distribution over completions to a prompt x. In RLHF, our objective is to train an LLM maximize reward function r(x: (x.y)) v) that measures the quality of prompt-completion pair (z, v). Typically, a KL-divergence constraint
added to prevent the updated model from straying too from a base LM (Ziegler al. (019).
which we denote as Tee. Altogether, RLHF methods optimize the following objective,
T (yyl²)
[TT = (xyr(())) (Ix.)) - a log
(1)
Teet Eier(yla) x))
which maximizes the expected reward over the prompt distribution p subject to a KL-constraint mod- ulated by a. Usually, this objective is optimized using a comparison dataset of the form {(x, yd v')], where the "win" completion y" is preferred to the "loss" completion y', which we write as y" Σ V²
While this objective is ubiquitous in prior work (Ouyang et al.) 2022 Rafailov et al. 2023), it is typically applied in the context of population based reward functions learned from large comparison datasets collected via multitude of annotators. In contrast, consider r(ær, y) to be the objective of a single individual. In this regime, collecting thousands of comparisons from one user is infeasible. Instead, we assume access to a small dataset of expert demonstrations, denoted Dg. We assume these demonstrations to be generated from the expert policy тÆ = arg max, (=)= ((, v)], which maximizes reward in expectation. While demonstrations are typically used for SFT, such approaches typically struggle in data- limited settings. On the other hand, it can be difficult to prompt a model to "overcome" the priors induced by its RLHF training. DITTO, as described in the next section, addresses these problems by directly generating comparison data using LM outputs and expert demonstrations. This means that unlike synthetic data generation paradigms (Lee et al. 2023). DITTO does not require model that performs well at the given task priori.
3

---

Published as a conference paper at ICLR 2025
3.2 DITTO
The key insight of DITTO is that the LM itself, along with the expert demonstrations, can generate comparison datasets for alignment, removing the need to collect large number of pairwise preferences. This results in contrastive-lit -ike objective, where the expert demonstrations are positives. Here we provide an intuitive explanation of DITTO; later we provide a more theoretical derivation in $3.3
Algorithm 1: DITTO
Input: :L Tief demos DE = { uf HENN, sample size M, sample frequency K Init :TD < SFT( Dz), =
while not converged do
Dj U ( y/ TT( (=<(11=)))
for k = 1,2, .... K do
Sample batch B - {zz, y², y/ of comparisons from induced ranking:
Generating Comparisons. Consider com- pletion sampled from the expert policy, y ~ (e(+)>)) By virtue of being "expert", y is likely to have high reward, as TE is definition- ally the reward maximizer in expectation. Con-
Dg F Dj ž Di-1 ... 2 Do
W1 < (PO B) # Update policy
t t +
sequently, we would expect samples from any other policy π to have rewards less than or equal to those of TE. i.e., VTr, IE [ w)] 2 En]r(x, Using this observation, we can construct comparisons (x, y yy" where yº = y" by simply sampling completions y" ~ <[|[2> for every demonstration prompt pair in Dg. Though such comparisons are derived from policies instead of indi- vidual examples, they have proven effective in prior work (Brown et al. 2020a). A naïve approach for DITTO would then optimize Eq. (T) using this dataset and off-the-sheil RLHF algorithm. Doing so would increase the probability of the expert responses while decreasing the probability of the current model samples, unlike standard finetuning which only does the former. Crucially, using samples from allows us to construct an unbounded preference dataset given only a few demonstrations. However, we can do better by considering the temporal aspect of the learning process.
From Comparisons to Rankings. Using comparisons only between the expert and single policy t may be insufficient for obtaining good performance. Doing so decreases likelihoods only that specific leading to the overfitting problems that plague SFT in low-data regimes. Analogous to replay in RL (Mnih et al. 2015), we can consider data generated from all policies Iearned over time.
At the first iteration, let the initial policy be no. We can sample from this policy to assemble dataset D2- = {(x, ) Then, we can generate comparison data for RLHF as y ž y"", which we denote Dg ž= Do for brevity. Using these induced comparisons,
update Tu obtain new
policy TT. By definition, E. x [r(x, y) E,, [r(x. y)] as well. It follows that we can also generate comparisons using म Dr > Dj- Continuing this procedure, we generate progressively more diverse comparison dataset using all prior policies. We refer to these as "replay" comparisons.
While this approach is theoretically consistent, it decreases the likelihood of the LM everywhere except at expert demonstrations. Though permissible in data rich scenarios, this may also lead to overfitting with a small DE. However, if assume that the policy improves each iteration, i.e. E [r y)1 2 Ez, Ir(x, [r(,,)], then we can also consider comparisons between policies during the course of learning. Unlike comparisons with the expert, we do not guarantee that this holds; in practice, however, we found that models tended to improve with each iteration, perhaps owing to the convexity of both reward modeling and Eq. ①1. This lets us sample comparisons between the complete ranking of policies:
DE 2 D. D.-1
Dj Do-
The effect of adding these "intermodel" and "replay" comparisons is that the likelihoods of earlier samples (e.g., those in Di) are pushed down more than those of later samples (e.g., those in Dz), smoothing the implicit reward landscape. Our practical implementation aggregates handful of these intermodel comparisons in addition to comparisons with the expert.
A Practical Algorithm. In practice, the DITTO algorithm is an iterative procedure comprised of three simple components as outlined in Algorithm| [ First, we begin by running supervised fine-tuning on the set of expert demonstrations for a limited number of gradient steps. We set this to be the initial policy no. Second, we sample comparisons: at most K times during the training process, we construct a new dataset D, by sampling M completions from ग. for each of the N demonstrations in Dr and add it to the ranking over policies Eq. 2 When sampling comparisons from Eq. 2 each
4

---

Published as a conference paper at ICLR 2025
batch B is comprised of "o%'on" comparisons DE ž D,, 20% "replay" comparisons of the form DE -n2 Dict. and 10% "intermodel comparisons" of the form Dise 2 Djes. Finally, we update the policy using RLHF. Specifically, using batches sampled via the aforementioned procedure, update the policy ne to obtain 4 using the DPO (Rafailov et al. 2023) loss function
CT D) (B w/) [Log (crlog <(y"kz)
<(v'<>) log Tnd
where a the logistic function from the Madley-Teeryy preference model. During each update, do not update the reference model Tet from the SFT policy to avoid straying too far from initialization. DITTO can support any direct preference optimization method as part of the final step (e.g. KTO Ethayarajj et al. (2024) SimPO Meng et al. (2024)) ) In practice, we found that the exact choice of the preference optimization algorithm had limited downstream effect, so we defaulted DPO for all experiments.
3.3
DERIVING DITTO AS ONLINE IMITATION LEARNING
DITTO can be derived through an online imitation learning perspective, where expert demonstrations are used in conjunction with online data to simultaneously learn a reward function and policy. Specifically, the policy player maximizes expected reward max, [Tx, as the reward player minimizes its loss min, C(D",r) over an online dataset D" Concretely, we instantiate this optimization problem using the policy objective in Eq. (I) and the standard reward modeling loss
min ( -EE, [log (r( )" r(x, y/))] s.t. ा = arg max JkL (T,, rT)
(3)
*
As done in prior work (iikehi et al. 2022). take D be dataset of comparisons such that y" ž y" if (x,rr(,,))) > E.r |r( v)]- The superscript indicates that D contains online comparisons between и and the expert TE. By using different choices of regularizers and comparison data, one can arrive different inverse RL (IRL) objectives (Ho & Ermon 2016).
Deriving DITTO. The first step in simplifying Eq. (3) is addressing the inner policy maximization. Fortunately, from Ziebart (2010) we know that the policy objective JKL. has closed form solution of the form x² (y/[z) Tee( )e" (z.u)/a /Z(xr) where ((x) is the partition function normalizing the distribution. Notably, this establishes a bijection between policies and reward functions which we can use to eliminate the inner optimization. By rearranging this solution, we can write the reward
function r as
r(x: y) = a log (v(x))) log ((x).
Furthermore, prior work (Rafailov al. 2024) shows that this reparameterizztion express any reward function. Thus, we can perform a change of variables from r to a by substitution into Eq. (3). giving the DITTO objective
min - Ep- [log a (a m(y"la) a log mar(y"|s)
*(y/ (x) mar(s
)
Note that like DPO, implicitly estimate the reward function. Unlike DPO, DITTO depends an online dataset preferences D= At minimum, the online preference dataset ought to contain comparisons TE 2 ग. Vv. However, any preferences consistent with the ground-truie reward function can additionally be used. We leave this exploration to future work.
Why does DITTO work better than SFT alone? One reason for DITTO's relatively high performance is that it uses far more data than SFT by generating comparisons. Another is that online imitation learning methods can, in some circumstances, perform better than the demonstrator while SFT only mimics the demonstrations. While this is known in the IRL community, we show the following result in Appendix B to relate DITTO's ability to extrapolate beyond the demonstrator to two divergence measures.
Lemma 3.1. (Adapted from Brown et al. (2020a)) Let πt be the optimal policy for Eq. T and be the policy estimated by DITTO using expert demonstrations DE. Extrupolation beyond the demonstrator, i.e. Ex(r(x,V)) > Epg (ra, v)) is guaranteed if Jna( - RE (r(,,(())))) o Dx (*I[**) DxD (f(l
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
We first outline benchmarks, focusing on tasks with subjective preferences (e.g., email writing, essays, articles). We then discuss automatic evaluation, compare DITTO to several baselines, and outline results. Finally, we conduct user study with DITTO, soliciting demonstrations from participants.
4.1 STATIC BENCHMARKS
Data Measuring few-shot alignment with DITTO requires demonstrations from individuals instead of aggregated datasets. We therefore build on prior Author Attribution (AA) datasets. The AA task requires one to determine which author a from set of authors A wrote specific document. We can reframe prior AA classification tasks as effective alignment: aligning an LLM to a specific author should result in generations that are more likely to be attributed to the same author. We collect data from 20 distinct authors from two sources: (1) emails and blog posts from the CMCC dataset (Goldstein al., 2008) that contain only one author and (2) news articles from the CCAT dataset (Lewis et al. 2004). Our AA benchmarks consist of a diverse range of tasks at the author- level; tasks span from writing financial editorials to opinion pieces on controversial topics. High performance on CMCC / CCAT50 requires non-trivial generalization across prompts. For more dataset details and example tasks, we refer the reader to Appendix
Splits and Preprocessing Some of our benchmarks have more writing samples per author than others. While the original CCAT can have more than 50 samples per author, CMCC can have as few as 12. To control for sample count, we randomly select the smallest set of demonstrations available from each author across our training splits (12) for our experiments. We randomly select 10 authors from each dataset, use samples to train, and split the remainder into test and validation. Table 4 in the Appendix describes the finalized train/val/test counts across each benchmark.
Models and Baselines Alongside DITTO, we evaluate supervised fine-tuning (SFT), testing if simply Annettunig on the expert demonstrations DE for longer is effective. We also evaluate SPIN (Chen et al. 2024). an iterative self-play method designed replace SFT. Finally, we test zero-shot and few-shot prompting, including demonstrations directly in the model's context. For few-shot prompting, we add the entire train set of an author's demonstrations in-context. We additionally tried to prompt engineer with zero-shot constraints (c.g., prompting the model to not sound like an LM), with limited success (see Appendix D Our experiments require an instruction following LLM. We use Mistral Instruct v0.2 7B as starting point (Jiang et al. (023) and train using LoRA (Hu et al. (02T). We did try full finetuning for a handful of authors but observed no significant
difference. We therefore used LoRA for all experiments. Finally, we compare against zero/few-shot prompting with a more powerful LLM (GPT-4). Hyperparameter details are in Appendix D
Automatic Evaluation Given that our datasets contain a total of 20 authors, we must train and evaluate a large set of models (20 authors x 7 training paradigms 140 models). To facilitate the evaluation process, we use 4PT-4 to compare the outputs of models across various conditions. Prior work has used GPT to both annotate and evaluate text (Zheng al., 2024). In general, performance lags behind human evaluation; however, for detecting authorship and style similarity, prior work has shown that model-based classification is actually more reliable than non-expert humans (Krishnal eet al. 2020 Hallinan et al.) 2023 Liu et al., 2024, Liu & May 2024) and GPT-4 eval generally outperforms other automatic metries (Kim et al. (023). allowing us to scale hyperparameter search and run evaluation in a more cost-effective manner.
In our setting, we use GPT-4 determine if text sounds more less like specific author. Given an author-written text t and two pairs of generated text from different conditions and b, we prompt GPT-4 to select the text that most closely matches the validation or test text f, and compute averaged head-to-hear win rates. To account for ordering bias, we swap orders and average the judgments. Our evaluation prompt and performance benchmarking details are outlined in Appendix F
Results Our main results, evaluated with GPT-4 eval, are summarized in Table ! Averaged across all authors, DITTO outperforms all baselines, with an average 77.09% win-rate across both CMCC (71.67%) and CCAT50 (82.50%). On CCAT50, DITTO outperforms all baselines across authors but one. On CMCC, DITTO outperforms all other baselines for 5/10 authors, followed by sew-shot
We use the gpt-4-00 towards their own outputs. Queries were run between December 20th, 2023 to May 10th, 2024.
4- 0--4-6633 version of GPT-4. We observed that Turbo versions of GPT-4 were more hiased

---

Published as a conference paper at ICLR 2025
Data
Method
Gargt
Q1
62
Og
06
0g
zero-shot
31. 31.89s.os
43.06
29.17
04
18.52
42.50
19.44
40.28
40.28
31. 31.48
few-shot
63.89,
73.61
65,06
62.50
62.01
55.56
61.81
75.93
63.89
40.28
68.52
zero-shot
27, 23.
31.72
30.56
16.67
29. 63
27.78
30.56
19.44
38. 89
19.11
26.85
CMCC
Mi Mistzal
few-shot
46.89,
61.11
26.39
30.56
12.59
78
37.04
54.17
54.63
SPIN
51 51.56g..8
56.94
48.61
56.91
40.74
73.15
48.15 15
59.26
59.72
31.94
38.89
SFT
56,78
18.06
27.78
86.11
74.07
58.33
43.52
61.81
47.22
81.94
58.33
DITTO
71.672.00
62.50
69.44
79.17
75.93
74.07
67.59
74.07
58.33
81.94
71.30
Mistral GPT
zero-shot
19.351.40
19.44
24.07
25.00
18.52
12.96
20.37
12.04
23.15
16.67
21 1.3
fow-shot
33. 33.70p.19
64.81
53.70
61.11
53.70
17.22
44.41
45.37
61.11
52.78
52.78
zero-shot
18. 18.061.8u
13.89
23.15
15.74
12.96
13.89
22.22
17.59
14.81
28.70
17.59
CCAT
few-shot
10.37
56.18
15.37
35.19
32.41
11.67
39.81
46,30
35.19
31.26
37.01
SPIN
62. 13
56.48
69.44
55.
82 41
70.37
63
58.33
54.63
51, 51.85
67.59
SFT
73. 73.89g.4
61.11
62.01
76.85
72.22
90.56
81.48
80.56
68.52
82.41
73.15
DITTO
82.50, sx
77.78
72.22 22
80.56
77.78
83.33
87.04
89.81
92.59
83.33
80.56
Table GPT-4 Eval: Head-to-head win rates between methods across benchmark test splits. DITTO outperforms all baseline methods average and across plurality of individual authors. a1....10
represents a single model trained on one ten sampled authors from each dataset (see $4). Results averaged across 3 runs, with 3 samples generated from each model with temperature 1.0. We also report win rates averaged across authors, along with standard error of the mean (avgsem).
prompting for 3/10. While SFT serves as a strong baseline (56.78% on CMCC, 73.89% on CCAT), DITTO provides an average 111.7% pt. win rate improvement compared to SFT alone.
Prompted baselines also lag far behind DITTO, especially zero-shot (including closed-source) models (avg. 154.4% pt. decrease on Mistral, ..51.5% pt. on GPT-4). While zero-shot GPT-4 is already finetuned using RLHE, we suspect that this training feedback differs significantly from that of authors in both CMCC and CCAT50. Adding all train instances as a few-shot prompt does help: win rates for few-shot prompting increase compared to zero-shot for both Mistral (120.9499 pt.) and GPT-4 (122.95% pt.) based LLMs. However, including few-shot examples still falls behind applying DITTO (avg. 337.35% pt. decrease for Mistral; 126.99% pt. for GPT-4). Varying the number of demonstrations in the few-shot prompt also yields no improvement (Fig. F.2 in Appendix). We suspect the underlying RLHF priors for out-of-the-box LLMs are fairly strong. Qualitatively, few-shot generations still sound GPT-generated relative to DITTO (Table Z Appendix).
While we do test another self-improvement training method (SPIN), we find that performance is lower than DITTO (avg.) 9.3% pt.)-we we suspect that design decisions for SPIN (e.g.. updating the reference policy, excluding interpolicy / replay comparisons) are targeted towards SFT-scale datasets. We ablate these decisions in and propose reasons for performance degradation.
Finally, we ran an ANOVA test determine whether there were significant differences between conditions, and then ran Tukey test to identify which specific conditions were significant. DITTO's improvements are significant (p < 0.05) compared to all other conditions in Table 1, excluding few-shot GPT on CMCC.
4.2
USER STUDY: TESTING GENERALIZATION TO NATURALISTIC TASKS
Our static benchmarks have focused on pre-existing author attribution datasets, using GPT-4 to mea- sure aligmment. However, GPT-4 eval exhibits a self-enhancement bias, likely inflating performance for LLM-like generations (Zheng et al. 2024 Panickssery et al., 2024). We therefore evaluate DITTO in more naturalistic setting: we conduct a user study to evaluate DITTO and ask users to provide demonstrations for range of tasks. As baselines, we use zero-shot and few-shot prompted GPT-4 along with SFT. Additionally, we ask participants to self-prompt models by iteratively authoring their own prompts to steer the model outputs. Zero-shot, few-shot, and self-prompl emulate what most users would do today to steer LLMs, and SFT provides a strong finetuning baseline.
We recruit 16 participants from social media (Twitter). Many of our participants were Ph.D. students familiar with prompting LLMs; therefore, our self self-prompt baseline offers a strong baseline for additional prompt engineering. Participants were paid /30/ hr; our study was approved by an IRB.
The user study consists of two parts. In the first part, we ask participants to specify four writing tasks (e.g., Write an email to your advisor asking for feedback). Participants asked provide demonstrations for two of the tasks (4 training demonstrations in total). To
User Study Outline
7

---

Published as a conference paper at ICLR 2025
help brainstorm tasks, we generate concrete task suggestions with GPT-4; participants could select from among these or provide their own custom tasks. We randomly split two task prompts into train, and saved two for testing; participants gave two demonstrations each for both the training prompts, to mimic a user willing to only put in minimal effort. Users were provided with default generations from GPT-4 to aid authoring demonstrations, which they could edit ignore. In the second part, we use the two tasks from the test set and show participants generations across all methods. We sampled one output from each method (self-prompl) zero-shot, few-shot, SFT, and DITTO), and solicited 10 pairwise preferences for each test prompt (resulting in 20 preferences total for each user). In all, we collect total of 320 pairwise preferences across 16 users. All comparisons are done blinded the condition. Additional user study details (e.g.. interface, examples of demonstrated feedback, prompts for generating tasks, etc.) are in Appendix
Results Our user study results corroborate findings from static benchmarks. DITTO outperforms baseline methods in aligning to demonstrated preferences (Table with DITTO (68.8% win-rate) SFT (55.5%) few-shot (51.6%) > self- prompt (46.9%) > zero-shot (2..3%). DITTO is significantly better than all other methods (ANOVA + Tukey test, p < 0.05). Additionally, users generally struggle with verbalizing pref- erences into prompts: self- fif-prompting slightly underperforms providing demonstrations in few-shot prompt, and substan- tially underperforms DITTO. We also qualitatively observe that users often edit nearly half of the default output from GPT-4 when authoring demonstrations (examples in Appendix G). with average normalized Levenshtein edit distance = 0.43. Large edits to the output alone highlight the effectiveness of demonstrated feedback an interaction.
Method
Win Rate
GPT-4
zero-shot few-shot
27.3
51.6
self-prompt
46.9
SFT DITTO
55.5
68.8
Table 2: User Study Results. In head-to-head human annotated win rates, DITTO outperforms self- prompted, few-shot, and zero-shot GPT-4 baselines, along with SFT.
To better understand why users in our study selected DITTO outputs, we fit Fightin' Wordd model to identify lexical differences between generations from GPT-4 and DITTO (Monroe al., 2008). Fightin' -Words is used to identify words that are statistically significantly different in Irequency between corpora. It generates log-odds ratios and z-scores, which measure how likely a word is to appear in one corpus compared to another. Many of the words that appear in GPT generated outputs compared to DITTO (Table come from cliche phrases: "greatly appreciate your time and understanding" or "hope this message finds you well." Compared to DITTO, GPT also regularly generates sentences mentioning "trust" and "initiative" in emails drafted to close collaborators. We suspect GPT's writing style is tightly coupled to its RLHF priors.
WHEN DOES DITTO WORK?
Ablation
Win Rate
user must decide on several prerequisites before using DITTO, from how many demos they have to how many neg- atives they must sample from the LMM We explore the impact of these decisions and focus on CMCC, as it covers a broader range of tasks than CCAT. We additionally analyze sample efficiency of demos vs. pairwise feedback in our user study.
Sample only at start
57.3
DITTO
70.1
+ remove interpolicy
68.1 1
remove replay
63.6
+
update Tel
45.8
5.1 ALGORITHM PERTURBATIONS
Table 3: Head-to-head win rates across DITTO algorithm abla- tions on CMCC. We experiment with sampling all negatives at the start, ablating replay and interpol- icy comparisons, and updating the reference policy.
DITTO consists of several hyperparameters: namely, the num- ber of DITTO iterations N {1..4} and negative samples M = {2)..10} generated from our sequence of policies. Sep- arately, we ablate components of DITTO, like the use of inter- policy (Dige Z Djcs) and replay (Dz = Dise) comparisons. We also test an ablation where we do not re-sample data during training and instead sample all negatives only at the start, and where we update मe_ = πe at each iteration like SPIN (Chen et al. 2024 Note that DITTO performance varies from user user. To account for variance between author-level win rates, convert each author's 's averaged win rate to the % improvement from their initial ablation's 's win rate (e.g., % improvement from DITTO iteration, 2 generated negatives, or 1 training demonstration).
8

---

Published as a conference paper at ICLR 2025
Pct. Improvement
60
20
20
20
0
2
12 Generated Nogatives
8
2
3
4
5
DITTO Iterations
Training Demonstrations
Figure 2: Head-to-head win rates across DITTO hyperparameter perturbations on CMCC. First, increasing the number of DITTO iterations improves GPT-4 eval performance (left). Increasing the number of generated negatives also reduces DITTO variance across users while improving DITTO performance (middle). Finally, increasing demos also improves performance, but we observe diminishing returns (right). Error bars correspond standard error of the mean across authors.
Increasing the number of DITTO iterations generally improves performance (Fig. 2p Comparing Iteration 1 to Iteration 4, we observe relative 31.5% increase in GPT-4 eval win rates. Improvement non-monotonic Iteration 2, performance drops slightly (-3.4%). Early iterations might yield noisier samples, potentially reducing performance. On the other hand, increasing negative samples monotonically improves DITTO performance. Generating training for example, yields 21.09% win-rate improvement compared to just 2. Furthermore, as we sample more negatives increases, variance in DITTO performance decreases. However, there is tradeoff associated with increasing the number of negative samples: runtime of DITTO will also increase. In addition, we find that added iterations (> 6) eventually result performance degradation, likely due overfitting. Sampling more negatives 10) also yields plateauing performance.
negatives for each demonstration the
We also find that ablating components of DITTO results in reduced performance (Table 3p. If we sample all negatives at the start-instead of iteratively resampling in online fashion we observe that win rates compared to using DITTO drop from 70.1% to 57.3%. While iteratively ra-sanpiing improves performance, continuously updating Ter during this online process can significantly degrade performance: win rates drop from 70.1% % to 45.8%. We suspect updating overfitting Finally, both replay and intrrpooicy comparisons help DITTO. Removing replay and interpolicy comparisons reduces win rates from DITTO by 6.5 and 2 points respectively.
results in potential
One potential confound for DITTO's performance is that the instruction-followee Few-shot prompting in particular cannot undo the style trained into instruction-folloiing LLM. To isolate this effect, we additionally compared DITTO to a fine-tuned and few-shot prompted base variant of Mistral. Compared to DITTO, still significant degredations moreso than the instruction following model. Win rates against DITTO are 9.4 and 10.4 for few-shot and SFT respectively. We suspect that general instruction-follont following capabilities required "starting point." Jointly learning instruction-followee and demonstrated feedback is too difficult a task to learn from handful of demonstrations.
prior too strong.
5.2 SAMPLE EFFICIENCY
key affordance of DITTO its sample efficiency. In $4 we examined DITTO's performance on the full set of demonstrations from each author. In practice, user may only provide one or two demonstrations, Therefore, we evaluate sample efficiency across DITTO trained smaller subsets of the full training set N {1...7. Like with our algorithm perturbations, we report per-user normalized win rates (Figure 2). First, we observe that DITTO win rates increase rapidly at the start. From 1 < N < normalized performance roughly doubles for each additional demonstration (0% - 5% + 11.9%). However, we observe diminishing returns when supplying extra demonstrations (4 N < 11.9% 15.39%): performance saturates demonstrations increase. A key design decision in using DITTO lies in the selection of demonstrations; we additionally suspect that the quality of provided demonstrations likely also affects DITTO performance. We conduct preliminary analysis of demonstration cohesiveness on downstream performance, testing how demonstration similarity affects DITTO performance (Appendix H.2). We additionally revisit this in future work.
5.3
How DO PAIRWISE PREFERENCES COMPARE AGAINST DEMONSTRATIONS??
core assumption of DITTO lies in sample efficiency coming from demonstrations. In the- ory, user could achieve similar performance by labeling many pairwise preferences with

---

Published as a conference paper at ICLR 2025
an ideal set of demonstrations in mind. As preliminary approximation, author pro- vided demonstrations for the user study and also annotated 500 preference pairs using out- puts sampled from the instruction following Altogether, we constructed a pairwise prefer- ences dataset Daref {(r,, y², "')), where 9: > yj. We then computed win rates be- tween 20 pairs sampled from Mistral trained on (a) 4 demonstrations with DITTO, and (b) on {0...500) preference pairs with just DPO. When we sample pairwise preferences from Thef alone, we observe that generated pairs are out- of-distribution relative to the demonstrations- pairwise preferences do not reach a user's demonstrated behavior (results in Fig. 3 "Base policy," in blue). Even when we finetune Tel on the user's demonstrations, preferences to match DITTO performance (Fig. "Demo-finetuned policy," in orange). This especially damning for methods that align LLMs using samples generated from Net alone (e.g. Constitutional prefereeess generated over OOD samples (relative to the user's true
Mistral 7B (demonstrations in Appendix G.4
Win-rate DITTO
Base policy
Demo-finetune policy
40-
20
0
100 200
300
400
500
Number of Pairwise Preferences
Figure 3: Demonstrations are more sample ef- ficient than pairwise preferences for an individ- ual user. We compared DITTO with 4 demos to pairwise prefs sampled from (1) base instruction- following LM The and (2) Thef fine-tuned on demos. Applying DPO on 500 pairwise preferences- samples from Tre--yields no improvement com- pared to DITTO. Even if demos are used to fine- tune Nee before sampling, one must collect many pairwise preferences to approach DITTO
still need > 500
reward) are essentially irrelevant.
6 CONCLUSION
Current modes for soliciting feedback-like principles or pairwise annotations-Caat to population- level preferences. In this work, we instead highlight the effectiveness of using demonstrations as feedback, and show that a limited number of demonstrated behaviors can provide a strong signal for preferences specific to an individual. We also introduce a new technique, DITTO, that cheaply generates online comparison data from demonstrations, and test DITTO's effectiveness across static benchmarks and user study. Focusing feedback collection at the demonstration level may offer a more diverse overview of individual preferences, and encourage re-evaluation of the interfaces and interactions used to collect human feedback.
Limitations One limitation involves DITTO speed: DITTO is slower than training- frree approaches (prompting) and SFT (15 minutes with DITTO vs. minutes with SFT on 7 demonstrations) A bottleneck lies in sampling, though we suspect mix of prior (e.g., vLLM Kwon et al. (2023)) and future work in LLM inference optimization can improve DITTO's s speed. DiTTO-ed models also tend to "forget" more general capabilities. For example, we observed that models often refused to write programs (generating "I have no idea how to write code."). In Appendix [..] we evaluated forgetting on coding tasks with HumanEval Chen et al. (2021), observing some degradation. However, we entirely mitigate all degradations by selectively dropping DITTO's s LoRA adapter, and routing instructions between the general instruction- foolowwin model and the specialized DITTO LoRA adapter (ala MoE). Finally, because of evaluation and computational constraints, we do not test across model families or sizes. Exploring how DITTO scales is avenue for future work.
Chenn
Future Work One avenue involves analyzing tradeoffs between types of preference data (e.g.. demonstrations vs. preferences vs. principles). While we propose demonstrations as a feedback modality, each type of feedback requires different levels of effort, and the effectiveness depends on the user providing feedback. In addition, preferences provided through demonstrations are often local in quality- users provide demonstrated preference the context of specific domains. Understanding how scaling the amount of local demonstrated feedback affects general- general-purpose model behavior is an avenue for future work. Another key design decision in using DITTO lies in the selection of demonstrations; we additionally suspect that the quality of provided demonstrations likely also affects DITTO performance. While we explore the effect demonstration cohesiveness in Appendix HH.2 (how does demonstration similarity affect DITTO performance), understanding how to select an optimal set of demonstrations for DITTO from a user is an avenue for future work. Given the ability to align models with handful of demonstrations, DITTO could support new interactions for individual end users to orchestrate many task sask-specific models curated to their needs; or may motivate inference-only alignment methods for black box L.MSs that do not require fine-tuning.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Demonstrated feedback is a double-edged sword. While DITTO can enable effective personalization of language models, also suspect that DITTO will be especially useful for model un-alignment, amongst range of other risks (Kirk et al. 2023), However, the current status quo of language model alignment lies with large corporations that practice limited transparency. Models like GPT-4 already espouse dangerous positive stereotypes or unfairly benefit privileged groups due to representation issues in the feedback collection process (Cheng et al. 2023 Ryan al. 2024).
ACKNOWLEDGEMENTS
We thank Eric Zelikman, Matt Jörke, Jan-Philipp Fränken, Michael Y. Li, Michael Ryan, Will Held, Shan Rizvi, Suvir Mirchandani, and Jensen Gao for helpful discussions and feedback. We also thank members of the SALT Lab and the Stanford HCI NLP groups.
REFERENCES
Riad Akrour, Marc Schoenauer, and Michèle Sebag. April: Active preference learning-based bbsed reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK. September 24-28. 2012. Proceedings. Part 11 23, pp. 116-131. Springer, 2012.
Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D Goodman. Star-gate gate: Teaching language models to ask clarifying questions. arXiv preprint arXiv:2403. 19154, 2024.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences. arXiv preprint arXiv:2310. 12036, 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arxiv preprint arXiv:2212.08073, 2022.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pp. 783-792. PMLR, 2019.
Daniel S Brown, Wonjoon Goo, and Scott Niekum. via automatiialll--rakee demonstrations. Conference on robot learning, pp. 330-359. PMLR, 2020a.
imitation learning
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020b.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar. Jan Leike, al. Weak-to-stront Eliciting strong capabilities with weak supervision. arXiv preprint to-sot generalization:
2023.
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. Art or artifice? large language models and the false promise of creativity. arXiv preprint arXiv:2309. 14556, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 2rXiv:2107.03374 2021.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint 24Xiv:2401.0133 2024.
11

---

Published as a conference paper at ICLR 2025
Myra Cheng, Esin Durmus, and Dan Jurafsky. Marked personas: Using natural language prompts to measure stereotypes in language models. arXiv preprint arXiv:2305.18166 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, et al. Scaling instruction finetiond language models. arXiv preprint arXiv:2210.1 1416, 2022.
Allen Cypher. Eager: Programming repetitive tasks by example. In Proceedings of the SIGCHI Humanfactors factors in computing systems, pp. 33-39, 1991.
conference on Human
Allen Cypher and Daniel Conrad Halbert. Watch what I do: programming by demonstration. MIT press, 1993.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01336, 2024.
Johannes Fümkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference- based reinforcement learning: a formal framework and policy iteration algorithm. Machine learning, 89:123-156, 2012.
Ge Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro, and Dipendra Misra. Aligning Ilm agents by learning latent preference from user edits. arXiv preprint arXiv:240..1566, 2024.
Jade Goldstein, Kerri Goodwin, Roberta Sabin, and Ransom Winder. Creating and using corre- lated corpora to glean communicative commonalities. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08). 2008.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu. et al. Reinforced self-training (rest) for language modeling. arXiv preprint ar0iv:2308.. 08998, 2023.
Skyler Hallinan, Facze Brahman, Ximing Lu, Jachun Jung, Sean Welleck, and Yejin Choi. STEER: Unified style transfer with expert reinforcement. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7546-7562, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ 1/2023 findings-emnlp.500. URL https:://cclanthoo..g F Findings emn 1...m.m0m0ppppp.. 506
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. The Twelfth International Conference on Learning Representations, 2024. URL
https: wttp///wwweee..cc//e/e/ee net/ forummid-iXiRiiie
John Hewitt, Sarah Chen, Lanruo Lora Xie, Edward Adams, Percy Liang, and Christopher D Manning. Model editing with canonical examples. arXiv preprint arXiv:2402.06/55, 2024.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Scongjin Shin, Sungdong Kim, James Thome, et al. Prometheus: Inducing fine- fine-graine -rringraindd eval- uation capability in language models. In The Twelfth International Conference on Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2025
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. arXiv preprint arXiv: 2023.
Kalpesh Krishna, John Wieting, and Mohit lyyer. Reformulating unsupervised style transfer as para- phrase generation. In Bonnie Webber. Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 737-762, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.emmlp-main.55. URL http:htsss/cccaaahhootooo http:://aalaanchoogyooooo 0r002000 emnlp-ma mw..maiin5555.. 55
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-626, 2023.
Michelle S. Lam, Janice Teoh, James Landay, Jeffrey Heer, and Michael S. Bernstein. Concept induction: Analyzing unstructured text with bigh-level concepts using Iloom. 2024. doi: 10.1145/ 3613304.3642830 URL.
htt.///i..ogg//// 145/361390- 3642830
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. 2023.
David Lewis, Yiming Yang. Tony Russell-Rose, and Fan Li. Rev1: for text categorization research. Journal of machine learning research, 5(Apr):361-397, 2004.
new benchmark collection
Shuai Liu and Jonathan May. Style transfer with multi-iteration preference optimization, 2024. URL
https::/arxiv.orrg/ 2406. 11581
Shuai Liu, Shantanu Agarwal, and Jonathan May. Authorship style transfer with policy optimization,
2024. URL.attts://arxiv.
abs /b0..////4000333
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 3470-3487, 2022.
Volodymyr Maih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 5 518(7540):529 533--33,,933330-- 2015.
Burt I.. Monroe, Michael P. Colaresi, and Kevin M. Quinn. Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict. In Political Analysis, volume 16, pp. 372-403. Cambridge University Press, 2008.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint urXiv:2112.09332, 2021.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pp. 278-287, 1999.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Arjun Panickssery, Samuel R Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. arXiv preprint arXiv:2404 13076, 2024.
13

---

Published as a conference paper at ICLR 2025
Rafael Rafailov, Archit Sharma, Erie Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.182290 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From q": Your language model is secretly a q-function. arXiv preprint arXiv:2404. 12358, 2024.
Michael J Ryan, William Held, and Diyi Yang. Unintended impacts of Ilm alignment on global representation. arXiv preprint arXiv:2402. 15018, 2024.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Perey Liang, and Tatsunori Hashimoto.
Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
Harshit Sikehi, Akanksha Saran, Wonjoon Goo, and Scott Niekum. A ranking game for imitation learning, 2022. URL
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization of LLM agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Pupers), pp. 7584-7600, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1/265//1/2224.4cllooo org. 201- 409
URL
Moritz Stephan, Alexander Khazatsky, Eric Mitchell, Annie S Chen, Sheryl Hsu, Archit Sharma, and Chelsea Finn. Rlyf: Learning from verbal feedback without overgeneralization. arXiv preprint
arXiv:2402. 2024.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01.22 2020.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze -zze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201. 08239, 2022.
Joe Watson, Sandy H. Huang, and Nicolas Heess. Coherent soft imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Ronald J Williams. Simple statistical gradient- algorithms for connectionist reinforcement learning. Machine learning, 2:229-256. 1992.
Yuxiang Wu and Baotian Hu. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint 20t::2240..00000 2024.
JD Zamfirescu-Pereee Pereira, Richmond Wong, Bjoern Hartmann, and Qian Yang. Why johnny can't. prompt: how non-ai experts try (and fail) design Ilm prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-21, 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15:155476155488 2022.
14

---

Published as a conference paper at ICLR 2025
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-star: Language models can teach themselves to think before speaking. 2024.
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. arXiv preprint arXiv:2310.11523, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judg with mt-bench and chatbot arena. Advances Neural Information Processing Systems, 36, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint prXiv:2211.01911 2022.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 133-- Chicago, IL, USA, 2008.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint Ar0: 08003,,,8859..22 2019.
A
APPENDIX
15

---

Published as a conference paper at ICLR 2025
B DERIVING DITTO AS ONLINE IMITATION LEARNING
For understanding the provided derivations, it is helpful to be familiar with the fixed point solution for Eq. M which was first derived for maximum entropy RL (Ziebart 2010
(" v) = r(z,)) v) (because contextual bandit)
V* (3r) = a log E E y (((x))/aa
1 ((x) Tea (T
(y/z) क (e
where Z(z) = e² (z)/a B (y [er(x Using this information, in conjunction with
Equation we can number of useful inequalities between n², Tefs and an arbitrary æ.
B.1 DERIVING DITTO
Here we provide more detailed derivation of DITTO from an online imtiation learning perspective. In particular, we consider the common two-player min-max interpretation of imitation learning (Ziebart al. 2008; Sikchi al. 2022). but do so with general objective functions.
min L(D", max (T)
In this formulation, DD dataset of preferences such that y" 2 y Jar if if (r y)] > [r y)), iee. one completion is preferred to another if the corresponding policy has higher expected reward. This framework generalizes prior work. For example, we limit ourselves only comparing the expert policy Tg to the current policy π, and add a regularizer, we can obtain the maximum entropy IRL objective from Ho Ermon (016). Choosing JkL. as the policy objective function and maximum likelihood on the Bradley- model as the reward objective we get the following optimization:
min (x.""" WD, [log logor(((z,))) v² r(x, v'))],
max Jal
where Jki is the KL-constrained RL objectve from before, but now dependent the learned reward function. We then select an ordering for the optimization, by making policy learning the "inner" objective as done Ho & Ermon (2016). Sikchi et al. (2022) makes connections between this choice and game theory. This results in the same equation in the main paper, repeated here for clarity.
min {-B (xx DD [log (r((x. r(x,v/)] s.t. = arg max Jka [n, rT)
We can then re-arrange the fixed point equations from maximum entropy RL, obtaining the "DPO- trick":
r(z. v) log x (vi=)) log Z(z).
This alone, however, is insufficient to obtain representation for the optimal policy naively substituting the above does not garuntee that the domain of reward functions can fully expressed by such a reparameterization in terms of the policy. Fortunately, prior work have established both
that such a reparatermization is equally expressive (Watson et al., 2023 Ng et al., 1999) and that
it does not affect the preference model Hejna et al. (2024); Rafailov et al. (2024). Completing this substitution yields the main DITTO objective.
However, DITTO is compatible with other algorithms,, such as traditional RL methods, so long as they can be used to solve the Kl-constraineed RI. objective in Eq. (T) Instead of using the DPO trick, one could use few steps of policy gradient algorithm to update the policy.
Distributional versus Point-wise Preferences. One thing to note is that we construct preferences for DITTO from distributional preferences, ie IE,, [r(,, v)] > [r,[r(z, v)]. However, this only guarantees that completions from one policy are preferred to another in expectation, not necessarily that every realized preference pair follows this relationship. We found that his choice works well in practice, and is actually common in prior work. For example, Brown et al. (2020a) uses a sequence of policies ranked by expected return in combination with Bradley-Terry model. Appendix C of
16

---

Published as a conference paper at ICLR 2025
Stephan al. 2024 shows that artificially sampling comparisons between two policies is consistent with Bradley- Terry reward model. Another possible view of this is that DITTO ends up optimizing an upper bound on the standard reward modeling loss:
Ex |- log (By-r= [r(z, w)] - By [r(x, w))] S (- v²) r(x, v/)
which arises from applying Jensen's is inequality on the negative log-sigmoid function.
B.2 ONLINE IMITATION CAN PERFORM BETTER THAN SFT
Here we show that, under some circumstances, online imitation learning is theoretically able to perform better than SFT on the expert dataset. To do this, we require few building blocks.
Proposition B.1. The objective value Jn of any policy can be expressed in terms of the optimal policy nt as [T) = (TT ) - aEEup [De (TT(|2²) |=>===>>0 [+|=))]
Proof. Note that at convergence, the optimal policy obeys the equality x (v(ax = Tee(yll)) V Thus, we can rewrite the reward function in terms of the optimal policy as
TT (ylax)
r(a, v) a log ( and substitute it into the objective function for the reward.
V (x)
(ylaa)
(T)) By
((r, a log
Nee((ylar)
T" (v/lx)
log (tylz)
E 1x)
a log Teer(y (z)
+ V (x)
n (v/ar)
-Byra((e))
alog Tyylar)
(x)))
a aog (tylx) n" (
Ex [x)] Byy
- alxzp [KL
This also implies that x= is unique (though this is known be true of MaxEnt RL objectives). This means that provided the reference policy is not already optimal, DITTO is able to improve it. Corollary B.2. Given Ree + n², then J (TT) I (TTr)).
This follows by considering proposition in conjunction with the fact that J(T*) 2 I (TTer) and the Kl--divergence is only zero both distributions are equal.
Lemma B.3. (Adapted from Theorem 1 Brown et al. (2020al) Let nt" be the optimal policy for
Eq. ① and f be the policy estimated by DITTO using expert demonstrations Dr. Extrapolation beyond the demonstrator, i.e. (1))))) [rx,, y)] Bn,y~D (r(x, v)] is guaranteed if
JRu(TT" ) - EDg [r(x, y)] > alEaop [Dx (f (+(+=)]] (+|x)) - aBz-p [Dx.. (+(14))((=
Proof. This can be shown via simple sequence of inequalities and application of proposition 1. For brevity, will omit the expectations over the prompt distribution. We proceed directly.
E+(r(z, y)] >
Jut(f) [r,[r(2,1) y)] DxL
Jna(T') Ixt(TT) + (at > [r, Dxx | Jka - a Da. (ft [+]]=²) > [r
Dx. ( [Tet)
Inr(T*) E (r y)] > aD (+||T*) axi. (+ (Ter)
If one wants to directly compare expected rewards, the -aD ([ term in JeL (x*) can simply be moved to the right hand side of the inequality. practice, we choose a fairly small value of a. This means that if the objective value of our optimal policy (reward minus KL) is higher than the average reward of the dataset, then we expect to do better than the demonstrator when our learned policy is closer to the optimal one than the reference.
17

---

Published as a conference paper at ICLR 2025
Source
Author
Train Author
Val / Author
Test Author
CMCC
10
2-3
2-3
CCAT
10
7
3
3
Table 4: Final Aggregate Benchmark Statistics
C DATASET DETAILS
In all, we collect data from a total of 20 distinct authors from two sources: (1) CMCC consists of texts written by 21 students in six different genres (email, essay, interview transcript, blog article, chat, or discussion transcript) covering six different controversial topics (Goldstein et al. 2008). We filter this corpus include only emails and blog posts, excluding sources where multplle individuals were involved (e.g., chat). (2) CCAT (Lewis et al. (004) consists of articles from Canadian Broadcasting Corporation's French Service, sourced from RCV1-v2 Reuters Corpus dataset. Due the large number of training paradigms evaluated in this work, sample articles from 10 authors from each dataset (260 documents total). Table highlights raw counts for each author.
C.I TRAIN / TEST GENERALIZATION
Within-author demonstrations across both user study and benchmarks span diverse range of tasks, like opinion pieces, blog posts, recipe writing, requests to meet, etc. Performing well on these benchmarks requires non-trivial generalization. Here, select a handful of train-test prompts that are representative of the train-test generalization expected from DITTO-ed models.
1. Train: Discuss recent movie or TV show you watched. Test: Share a new recipe you tried and loved.
2. Train: The city of Denver has decided legalize small amounts of marijuana for persons over 21. How do you feel about this?
Test: Do you feel the Catholic Church needs to change ways adapt to life in the s1st Century?
3. Train: Write an email to your professor seeking advice on research topics for an upcoming project.
Test: Outline an agenda for project meeting with new collaborator.
4. Train: Share personal writing rituals and habits for inspiration.
Test: Highlight fellow writer's work and encourage support within the community.
D HYPERPARAMETERS AND TRAINING DETAILS
We random hyperparameter sweep using Ir = {le - 4, 3e - 4, le - 5, 3e 5, le - 6, 3e - 6). epoch = {10, 15, 20, 25, 30), and B = (0.01, 0.05, 0.1). We additionally tune how frequently DITTO samples negatives (K (1,5, 10)); and how many negatives DITTO samples (M = {1, 5, 10)). Finally, we tuned the replay / expert intermodel fractions. selecting between 0.2 / 0.7 / 0.1, 0.25 0.5 0.25 and 0.1 0.7 0.2. We fixed optimal hyperparameters for each benchmark across all our remaining evaluations. We select hyperparameters from searches conducted on the validation set. All training was conducted on A A100 80GB GPU. We use the cosine scheduler for the SFT step, with warmup ratio of 0.1; and the constant with warmup scheduler for DPO with a warmup ratio of 0.25. For dataset, we train with SFT until BCE train loss a given batch approaches 1.00 (early stopping); ideally, want LLM to not overfit entirely to demos before DPO. Finally, we use Adam W across all experiments.
single, randomly selected author from each corpus,
18

---

Published as a conference paper at ICLR 2025
Dataset
CMCC
CCAT
LoRA Rank
Alpha
32
32
SFT Batch Size
4
4
Learning Rate
3e-5
3e-5
DPO Batch Size
R 24
R 24
DPO Learning Rate
le-6
le-6
DPO Grad Steps
40
40
DPO 3
0.05
0.05
DITTO
Negative Samples
10
10
Resample Step-Rate
10
10
Resample Temperature
1.0
1.0
Frac Replay
0.2
0.2
Frac Expert
0.7
0.7
Frac Inter-model
0.1
0.1
Table 5: Hyperparameters across benchmark datasets.
E FEW-SHOT PROMPT
Below writing samples.
EXAMPLE
[prompt_1]
[output_1]
##t EXAMELE :prompt_N]
[output_N]
Respond following prompt same way writing samples. generate output that GPT-1ike: :prompt
Figure 4: Few-shot prompt used to generate outputs for few-shot examples. We additionally test ablations in red text, but find that this reduces win rates for few-shot methods by 4% pts.
F GPT-EVAL
F.I
GPT-EVAL PROMPT
We outline final evaluation prompt below. We re-prompted for every pair of conditions, swapped generation orders to account for positional bias, and computed an averaged win rate. We sample with temperature 0.0 for eval, and use GPT-4 0613.
F.2 GPT EVAL BENCHMARKING RESULTS
We benchmark the performance of our evaluation setup with human data using CMCC by pairing an author's original text with another author's text. CMCC a more suitable dataset than CCAT for this evaluation: can pair texts from different authors that discuss the same topic. In CMCC, there are authors that wrote essays or emails for the same prompt while there are none of such cases in CCAT. For each author in CMCC, ereate 50 samples with the aforementioned setup. leading to total of 950 comparisons. With this setup and using bootstrap sampling, GPT-4 Eval achieves 81.79 2.42% accuracy.
19

---

Published as a conference paper at ICLR 2025
System: You are an impartial evaluator.
You aze an impartial evaluater. Below is a sample of human author's writing and two options.
#+t EUMAN AJCHOR'S WRITING: (damo)
##4 OUTPCT A: [text_a]
++4 OUTPUT B:
(text_b]
Task
based similarity NRITING above? Respond only with a JSON of the following Sormat:
option was written by
"answer*: "<the option similar the HUMAN AJCHOR'S WALTING) either B>"
ALWAYS REMAIN IMPARTIAL WIEN EVASCATING CUTPUTS.
Win Rate v.s. DITTO
60
40
20
0
0-shot
1
2
4
Few-Shot Demos in Prompt
Figure 5: Ablations for the number of demonstrations in few-shot prompted GPT-4. We report win-rate DITTO for varying number of demonstrations in the few-shot prompt. While increasing the number of demonstrations the prompt is positively correlated with improved performance, win rates are well under 50% and improvements non-monotonic, with notable variance continue adding demonstrations.
In addition, when pairing GPT-4's zero-shot outputs to the target author's texts with the same setup as above, we get an accuracy of 98.8 + 0.84%. This indicates that the human text is correctly considered more stylistically consistent than GPT-4's output in most cases and provides evidence that our GPT-4-based evaluation setup is not overly biased towards its own outputs.
G USER STUDY DETAILS AND EXAMPLE DEMONSTRATIONS
G.1
USER STUDY INTERFACE
Our interface consists of two parts: a data collection phase where we solicit tasks (Fig. 6 and demonstrations (Fig. ? from users; and preference elicitation phase (Fig. 8 where we ask
individuals to select between pairwise generations across baselines.
G.2 USER STUDY TASK GENERATION PROMPTS
While users provide their own tasks part of the data collection phase, we also generate prompts to make the process easier. Below, we outline the prompt for task generation. We replace (doma i and (audience) based the input from the user.
20

---

Published as a conference paper at ICLR 2025
Study Part
Task Selection
Task Selection
Fellow are suggestions of patenrial wriling teeks for:
Domai i: writing email
Audience: friend
with
wriling
You will writing example outputs for the selected tasks, Please pick tasks such that all the examples you write will have similar style.
Adid your own look here.
ADD
Write an email to a frfend recommonding a now book you just finished reading.
Write email friend sharing furny childhood
both experienzed.
frland planning a surprise day party
Write email to a friend suggesting move for your next move right together.
friend contessing
keeppng
Write an email to a frfend seeking advice on personal ciiemme you're tacing.
Write email friend eapressing gratitude their
difkkul, lime.
Write
ram niscing about memorable trip you together.
Figure 6: Task Elicitation Screenshot from the User Study, Individuals can either select GPT-4 generated prompts, or write their own.
21

---

Published as a conference paper at ICLR 2025
Demonstratios
Task 1 demonstrations
Pleese prov de examples of your ideal output for the following tack. We have provided some initial suggestions below. Modily the demonstration untill you'd be happy "actually sending yoursell" someone.
You
edits "each" demanstration section.
rephrase existing demonstrations
an colleague requesting feedback project.
Demonsmrtioo
Subject: Request for Feedback on Project
Hi [Colleague's 's Name).
hhope this message finds you wanted reach out you regardling project we nave been working together. value expertise and insights, would greatly appraciato your feodback the progress have made far.
Coule you please same like know you any suggestions for improvement, any concerns you may noticed, and any where you think we are excelling. Your input will be instrumental in shaping the final outcome of the project.
project
your thoughts
Speniiicllly
open scheduling meeting you to provide feedback meet.
Thank you advance for your assistance. forward hearing your thoughts.
Bes: regards,
[Your Name]
Figure 7: Demonstration Collection Screenshot. In this step, users provide demonstrations for 2 of the 4 selected prompts. We give GPT-4 zero-shot completions so that users can edit or rewrite to their liking. In the screenshot above, show the GPT-4 completion before edits.
Text Evaluation Tasks
Task Compose an email proposing a sponsorship package tailored to the interests of potential sponsor from Berkeley High.
For the lwing questons, please celect your preferre trut exomple based the task fitedd above
Please the ecomple that you profer:
Example
Example
Bubject Gustamized Sponsorah Opportantyy Berkeley High
Sucject Unique ef Power Partnerhhppo Gollaboration Proporat from Barisley light
Dver [Nume]]
Hey thene
hope this omail finds well am oxternay excttd proposo spononsht that
with Baenorey
emai Gleen our ahared proeenn eohcaeken, youth, and local communiy orgage nort. am incresilly excted propose an opportuity that we belone it's perfectl with our matual intereats.
Greetings
High lowke
engages
ofterin Sponsoret soremppppppp your parne ship. spenaorengg comas tallowing no benwiibe
Branding Reooght on Gain promenence a Ellver Sponsor, lnd uniin recigr ihon the ewnnt webs aus during our lus event
know, Barkedey High home to vibrarn community of Within creathvs active
Comp Sooth Space achoolee otterigss marchandiso
pos
engage
Figure 8: Pairwise Preference Screenshot. In the final step, users are asked to rank 20 pairwise preferences comparing all training paradigms.
22

---

Published as a conference paper at ICLR 2025
Please generate (n_tasks] ChatCPT writing prompts for the domain of [dommin))" and on audience of "nced"
EXAMPLE9; llere
examples of writing promots based domain of "Wiiting "personal frienda";
email*
Write an email
friend inviting them
Write an email to friand asking for advice.
srite an email to friend sharing recent sccomplishment.
Like our examples, make sure the new taaks you generate will result in wrizing with the style values (e.g. all easual, all formal, all honest, etc.)
Respond with
with following
ttakks" "Task 15",
cTaak 25",
*Taak 35",
23

---

Published as a conference paper at ICLR 2025
G.3
EXAMPLES OF DEMONSTRATED FEEDBACK
We include some examples of demonstrated feedback to highlight the types of changes users make.
G.3.1 USER
Train Prompt
Write an email to friend discussing recent movie TV show you watched.
Original GPT-4 Generation
Subject: Let's dish about the latest movie we watched!
Hey [Friend's Name],
I hope you're doing well! I wanted to chat with you about the movie I just watched last night "Inception". Have you seen it yet? It's a mind-bending thriller that kept me on the edge of my seat the whole time.
The storyline so unique and complex, with layers of dreams within dreams that kept me guessing until the very end. The visuals and special effects were absolutely stunning, and the acting was top-notch. Leonardo DiCaprio was fantastic in his role, as always.
I don't want to give too much away in case you haven't it seen it yet, but highly recommend it if you't re into
films that make you question reality. Let me know you've ve seen and what your thoughts I'd love to hear your take on it!
Looking forward to catching up with you soon.
Take care, [Your Name]
User's Demonstration
Subject: WTF even happens in Tenet?
I know you like Inception, but your boy Nolan's newest movie Tenet and OMG have no idea what happens. Like they are traveling through time, but not like normal time travel, they just move backwards through time. Like everything happens as a movie were played in reverse. So, like, when people who moving forward in time (the way do in real life) see the people who are moving backward in time they actually look like they're 're moving in reverse. And then kenneth branagh like somehow communicating with the future and building weapon to kill everyone for unknown reasons. have watched it three times and literally still have no idea what happens. But the special effects are dope just keep rewatching it, lol. N-E-WAYZ watch it and tell me what you think mmmm-kay?
Test Prompt
Write an email to a friend sharing a new recipe you tried and loved.
DITTO Output
Hey friend! Hope you're doing well. This weekend tried out this new vegan recipe for lentil bolognese (lentil bolognesa?) and it was amazing!! If you try it, the sauce is actually made with lentils instead of ground meat. You have to cook it down for like 45 minutes (you can even make the night before and let it sit) it's the most hands off thing, and it thickens the longer simmers. Then just toss it with pasta, garlic bread and you't re good to go. Lentils are the best for filling in the place of ground meat, and this felt so satisfying vegan cooking instagrams but there's a million ways to make it [...]
meal. So highly recommend, it's all the
24

---

Published as a conference paper at ICLR 2025
G.3.2 USER B
Task Prompt
Email a faculty member requesting guidance on preparing an effective research mentorship applica- tion.
Original GPT-4 Generation
Subject:
Request for Guidance Research Mentorship Application Preparation
Dear [Faculty Member's Name],
I hope this email finds you well. I am reaching out to seek your guidance and advice on preparing an effective research mentorship application.
I am interested in applying for research mentorship opportunity [specific program opportunity if applicable] and believe that your expertise and experience could greatly enhance my application. I have a strong interest in [briefly mention your research interests specific area of study] and I am excited about the prospect of working with you to further develop my research skills.
I would greatly appreciate it if you could provide me with insights what makes successful research mentorship application, including key elements to focus on, potential strategies to highlight my strengths, and any other advice you may have based your experience in this field.
I am committed to dedicating the necessary time and effort to submit a competitive application and value your guidance in this process. Your support and mentorship would be invaluable me as I strive to further my research goals and academic pursuits.
Thank you in advance for considering my request. look forward your advice and suggestions help me craft a strong research mentorship application.
Warm regards.
User's Demonstration
Subject: Request for Guidance Research Mentorship Application Preparation
Dear [Faculty Member's Name],
My name is [my name]] PhD student at [university name]. I am applying [name of this faculty member's 's mentorship program] for Summer 2024. am excited about the possibility of participating in your institute's 's program, and I am reaching out with question about faculty participating in your program.
My research interests and prior experience span several areas: [briefly mention your research interests in a concise list or sentence]. However, these areas are relevant to several faculty within your institute [list 2-3 names] some of whom may not be advising students this summer through your program.
possible, could you please let me know if any of these faculty are participating in your program in summer? would love apply any of these faculty are accepting students.
Thank you!
Test Prompt
Write an email requesting mentorship opportunities for an undergraduate research position.
DITTO Output
Dear [Name],
I am student at [University's Name], and I am searching for research mentorship opportunities this summer. recently came across [Research Club/Organization's 's Name] and was excited to see the variety of programs you offer for undergraduates. I was hoping you could provide some guidance on which programs are still accepting applications, and their respective deadlines. am particularly interested in any program that offers research opportunities, well stipends.
Thank you in advance for your help! someone could let me know, I'd be very grateful.
25

---

Published as a conference paper at ICLR 2025
G.4 DEMONSTRATIONS FOR SAMPLE EFFICIENCY TASK
Task Prompt
Write an email informing lab mates that we will be having ice cream this weekend as lab social.
Demonstration #1
We are gonna get some EYE SCREAM this weekend at [place] for our social. It's getting really friggin hot. Plus, you know, me and ice cream...
Whenever you get time: can you reply to me ASAP so can have a good idea of what the count looks like? I'll send some more details bit time.
See ya'll there!
[Name]
Demonstration #2
ATTENTION!!! VERY URGENT!!
Ice cream this weekend at [place]. We haven't 't had a social bit; plus [person] is gonna join us too. Lemme know if [time] works for you all! If not, can figure something else out. Be there or be a melted ice cream cone, [Name]
Task Prompt
Write an email informing students that there will be no seminar next week. Demonstration #1
Hey folks!
We won't be having seminar this week. Let me know if you have any questions for next week, though!
[Name]
Demonstration #2
Hi everyone!
Just a reminder that there won't be seminar this week. See you next week! As always, feel free reach out if you have any questions about the seminar in general.
[Name]
H SUPPLEMENTARY EXPERIMENTS
H.1 MITIGATING DITTO FORGETTING
We additionally evaluated DITTO on HumanEval Chen et al. (202) with randomly sampled author (a10) on CMCC. DITTO-ed models are specialized specific individual, so we expect some degradation. We can proactively mitigate degradations by selectively dropping DITTO's LoRA adapter, routing instructions between the general instruction- instruction-followint model (Mistral 7B) and the specialized LoRA adapter (ala MoE). To route queries, experimented with the following zero-shot prompt, prompting the general model.
When routing queries, observe degradation pass @ remains the same (0.31). In other words, our prompted router perfectly identifies which tasks are suitable for the adapter. Without this mitigation, we do observe significant degradation (0.31 - 0.13).
26

---

Published as a conference paper at ICLR 2025
I have specia ised model trained on data of the form: :demonetzations;
Should + use the specialized model or more general-gurpese model for the following
task?
thuman_evai__tte
Respond with juat SPBCTALTOED GENERAL.
Answer:
Word
Log Odds
Z Score
Value
appreciate
1.40
2.82
0.000
greatly
1.07
2.44
0.015
time
0.72
2.44
0.015
understanding
1.35
2.42
0.016
message
2.45
2.35
0.019
effectively
1.67
2.17
0.030
trust
2.27
2.16
0.031
initiatives
1.27
1.94
0.052
thought
2.05
1.93
0.054
words
2.05
1.93
0.054
Table 6: Words more associated with few-shot GPT compared DITTO in our user study. GPT relies on cliches and flowery language ("hope this finds you well"), even after few-shot prompting. Users found cliches hard to eliminate completely with prompting alone.
H.2 DEMONSTRATION COHESIVENESS AND DITTO PERFORMANCE
To evaluate demonstration cohesiveness affects DITTO performance, we prompted an LLM to score the cohesiveness a system for LLM-based document clustering. We then computed Pearson's 's R correlation coefficient between cohesiveness scores (1 5 likert scale) and performance increases compared the few-shot baseline. We found moderate positive correlation (R = 0.42) between the cohesiveness of author demonstrations and the downstream performance.
demonstrations, We used scoring prompts from Lam al. (2024).
One could automatically cluster large set of documents into specific sub-styles; and then train DITTO models individually on each cluster. Since L.M.-judged cohesiveness correlates with downstream performance, automatically assembling a set of DITTO adapters from training corpus is potential avenue for future work.
27

---

Published as a conference paper at ICLR 2025
Datuset
CMCC
CCATSO
User Study
Write approsimately 500 word estay to the following prompt: The city of Dewer has decided to legalize small amounts of mari- juana for persons over 21. How do you feel about this?
Write article that starts with the follow- bug: Two of Japan's most troubled banks took action Tuesday end their lony- running fiancial problems, with one finding a merger partner and the other fnallsing radical re- structuring
Write email friend sharing new recipe you tried and loved.
Author-written
Marijuana is one of those things that's still stigmatized for few legitimale reasons, and most of the reasons people feel a certain way about
[-.]] The moves are seen as just the begin- ning of drive by the indebted bank- ing industry to get ready for Japan's is emerg- ing "Big Bang" era financial deregula- tion. Hokkaido Takushoku Bank, Japan's is 10th largest commenciaa bank, said will merge with the smaller Hokkaido Bank Ltd and cease its overseas operations to become "super- "super-regional regional" hank. Such super-regionals their local power base to build large presence that rivals their big.city competi- tors. Meanwhile, Nippon Credit Bank Ltd. one of three long-term credit banks in Japann. expected to hold
Note: samples from the user study have different truin prompis. We have no ground truth for the test prompts.
true of alcohol. One huge, coun-nrlyy important, monumeutal dif- ference between the two is this: govemn ment can and does effectively tax the man- ufacture and distribution alcohol (and to- hacen, for that matter). Denver has taken step the right direction, and mayhe the only step that needs to he taken. Since people are given permission carry only small quanti- ties, theoretically this still legislates against large scale wheeling and dealing while pre- venting cops from having to divert time and attcation minor drug busts, which are often a waste of resources [..]]
Write an email to friend planning virtual movic night.
Guess what are doing Saturday!??? Having a virtual movie night!!!
Here are the details: Date: [Date] [...]
conference later in the day on restructuring that would in- cludé injection of governmeut money and hankruptey filings [..]
I will send you the link later. Tell Grahaa to come too. I've invited Bob and Sarah so I think we' ll have good crow, Do not bail!! [..]
DITTO
The Denver City Council recently voted to approve legalization small amounts of marijuana for residents 21 and over (The Den- ver Post). absolutely support this decision. This is move towards common sense law, It saves police signilicant amounts time and resources. I'd argue that the enforcement all laws concerning drags should take this angle. It's also heneficial for other reasons (revenue to the city, taxpayer dollars spent more effectively, etc). Opponents frequently argue for the "children being present the streets" theory. Fine, add a proviso I...]
[-..] Under that plan, the government vowed to fully privatise Japan's 's trouhled financial institutions by the 1998 business year. "The govermment will help with the mergerss and will provide a stable euvironment in which institutions can lake steps toward becom- ing self- sustainnta,,,,g."" Finance Minister Ryu- taro Hashimoto told reporters after meeting an Tuesday with executives of the twn trou- bled institutions. The had already taken the first necessary steps loward their "dying business" scenarios, in which they have al- most no chance of standing on their own I...I
Hey friend! Hope you'le 're doing well. This weekend I tried out this new vegan recipe for lentil bolognese (lentil bollonessa)) and it was amazing!! If you try it, sauce actu- ally made with leatils instead of groundd meat. You to cook down like 45 minutes (you can even make it the night before and let sit) it's the most hands off thing, thickens the longer it simmers. Then just toss with pasta, garlic bread and you' re good to go. Lenill are the best for filling in the place ground meat, and this fell so satisfying as a meal. So highly recommend. it's on all the vegun cooking instagrums but there's million ways to make it I...]
Few-shot GPT-4
I am certainly intrigued by the city of Deu- ver's recent decision decriminalize small quamitties of marijuana for individuuls who have reached legal drinking age. Given the myriad of implications this policy shift has government, society. and individuals alike, is certainly a development worthy of tboughtiul examination. I...
[..]] In what is soen as a significant develop- ment in the banking industry. stricken Sum- itemo Trust & Banking firm announced a merger deal with estahlished Sakura Bank Ltd. Meunwhile, Tokai Bank Ltd, grappling with similar issues, unveiled comprehensive restructuring plan aiming to nurse the bank back health over the next few years I...I
Subject: YOU NEED TO TRY THIS RECIPE!!! OMG! have GOT hear about this homb diggily recipe I just discos- ered! It's a pasta dish from Italy's 's homemade spughetti aglio olio!! YES, it sounds fancy but trust it's super simple, idiot-proof, and the finished product. it's lip-smackingly DEL- ISH! The main stuff you need spaghetti, gatlic, olive oil, chilii flakes [for the kick), and EVERYONE's fay cheese Parmesan [..]]
SFT
The Mile High City has made headlines once again the resull of un ordinance passed earlier this year. Denver, Colorado's 's muin metro has made history by legalizing small amounts of marijuana for persons over the ago of 21. The Denver Police Department
[...] Yarmatane Seimei First Real Eistate Co Lid will be the first major instittton wound up in Japan since World War Twa. when Japan's 's wartime debx liquidated. the firm said in a statement, The firm said it will request Wednesday that the govern- ment order into special public liquidation scheme which is supposeed to he a last resort for rescued financial institutions. [-.]]
Hey friend, So last night I had some people for dinner, and I cooked this amazing chicken recipe. I'm not gonna lie, it took little bit of work, bui was a skillet chicken marsala, and it tasted al- mazing. I cooked the chicken in butter until it was golden brown on the outside, and then I took out and cooked some shallots and mushrooms
totally worth it. II
now cannot levy charges against those found in possession of the substance in sizes con- sidered "rensonable", :: law, however, bans the sale and consumption of marijuana publicly accessible vicinity. [..]]
pan. Then added some marsala wine, chicken stock, and cream, and simmer about minutes. At very lust, I added the chicken buck to the pan cook through while the sauce reduced.
Table 7: Selected prompts and responses across datasets (CMCC, CCAT50, User Study) and methods (DITTO, SFT, Few-shot GPT-4) for all evaluated models. All generations are produced on unseen prompts drawn from the test set. Compared to the author-written ground truth, we observe that SFT occasionally feels longwinded, or fails to capture quirks of the author's 's writing. Meanwhile, few-shot GPT prompting yields outputs that sound "GPT-like" (myriad of implications. significant development, etc.), or over-does the author's 's original style (i.e. is cringe). Qualitatively, DITTO generations best preserve the author's 's voice and style.
28


Reviews:
[{"review_id": "90TTdZLDNb", "reviewer": "Reviewer_7eqT", "summary": "The paper introduces a novel method, Demonstration Iterated Task Optimization (DITTO), for training large language models (LLMs) with expert demonstration datasets in a more data-efficient manner. Through a mathematical derivation, the authors illustrate how DITTO functions as a form of online imitation learning. They validate the method's effectiveness by utilizing a GPT-4 evaluation scheme and compare it against several other approaches, including Supervised Fine-Tuning (SFT), SPIN, and few-shot prompting. The authors conclude that DITTO is particularly advantageous for training LLMs to adopt specific writing styles or user preference tuning, outperforming other methods in these areas.", "questions": "- Do you think DITTO would be effective for the coding skills or mathematical problem solving skills of an LLM?\n- Have you attempted training the LLM without LoRA, using full fine-tuning instead?\n- What kind of source code is used to generate online responses? If you were to train a much larger LLM (such as LLAMA 72B), would it be feasible to apply the online imitation learning method in the same way?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a data-efficient training method that enables LLMs to follow expert demonstrations. The Reinforcement Learning from Human Feedback (RLHF) data can be continuously generated by simply comparing expert demonstrations with the intermodel's responses. This approach can also be seen as a blend of Reinforcement Learning from AI Feedback (RLAIF) and RLHF, making it a reasonable and effective method.\n- The authors demonstrate the performance improvements of DITTO-trained models using GPT-4 evaluation and validate the method's effectiveness through a large-scale user study.\n- They provide a theoretical perspective on the connection between online imitation learning and demonstrate that online imitation learning can outperform Supervised Fine-Tuning (SFT). The mathematical derivation and explanations are clear, and the results are further supported by meticulously designed ablation studies.", "weaknesses": "- The authors did not investigate potential side effects, such as performance degradation on other benchmark datasets, after training with DITTO. Since the LLM is fine-tuned exclusively on targeted demonstrations, there’s a risk of significant performance drops in broader tasks. It is essential to preserve the LLM's original knowledge and abilities while adjusting its output to align with specific style and preference.\n- Also they overlooks the computational inefficiency of iterative training in an online imitation learning framework. This process requires substantial time and GPU resources, as it involves initializing the policy 𝜋0 (equivalent to SFT), generating responses from 𝜋0, training with DPO, and then iterating to produce 𝜋1, and so forth. These steps are difficult to reproduce and demand more computational power than SFT baseline. Furthermore, achieving faster response generation in the trained LLM would require additional engineering efforts. Although DITTO improves data efficiency, it is also crucial to consider computational efficiency, given the high costs of training and generating responses with LLMs.\n- The authors did not explore the limitations of the DPO algorithm or other potential approaches for training LLMs in a Reinforcement Learning from Human Feedback (RLHF) framework. It is known that the DPO algorithm can pose risks when training on preference datasets, as it may forget data from the \"winning\" side due to inherent mathematical issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5czRTUMuwc", "reviewer": "Reviewer_yVim", "summary": "This paper identifies a key issue: current LLMs, aligned to represent the collective voice of many, often fail to align specifically with any individual preference due to contradictions among them. While guiding LLMs toward a general preference is feasible, it requires substantial preference data. The authors propose a method, DITTO, to align LLMs to specific settings using fewer than 10 demonstrations drawn from existing interaction logs or direct edits to LLM outputs. These demonstrations are treated as \"golden\" examples, while outputs from current and previous LLM checkpoints are rejected. Through author attribution tasks and user studies, they demonstrate the effectiveness and sample efficiency of DITTO.", "questions": "1. In Section 3, you introduce the core method of DITTO and compare it with online imitation learning. What is the purpose of Section 3.3?\n2. How did you determine the percentage distribution of the paired data, specifically the 70% online data, 20% replay data, and 10% intermodel pair data?\n3. In Table 1, for the CMCC dataset, why do the zero-shot and few-shot results from GPT-4 appear the same in column a9, both at 40.28%? Additionally, why do both SFT and DITTO show results of 81.94% without any improvement? How would you comment on this?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces DITTO, a novel method designed to guide LLMs toward specific settings for effective customization, achieving sample efficiency with fewer than 10 demonstrations. DITTO outperforms strong baselines, including SFT and GPT-4 with few-shot prompting. Additionally, a detailed user study further reinforces the reliability of DITTO.", "weaknesses": "1. The static experiments in Section 4.1 are not particularly convincing. Have you considered testing additional baselines or employing other automatic evaluation methods, such as calculating sentence embedding similarity to compare styles?\n2. Have you evaluated DITTO on more benchmarks or tested its generalization ability? I noticed that only three authors were used for validation or testing. Can the DITTO method generalize to tasks beyond writing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l1o6ycyGO2", "reviewer": "Reviewer_41aT", "summary": "The paper proposes a method called Demonstration Iterated Task Optimization (DITTO), designed to align large language models (LLMs) with user-specific preferences using a minimal number of user-provided demonstrations. This method eliminates the need for large-scale datasets typically required for supervised fine-tuning or RLHF. The paper claims that DITTO can significantly improve the alignment of LLMs for user-driven tasks and offers a practical solution for customizing language models. The paper explains their theoretical insights from online imitation learning with practical implementations, demonstrating effective customization for real-world applications like email writing and author-specific content generation.\n\nI recommend accepting this paper, as it tackles a significant challenge and presents an interesting solution that is well-supported in theory and through empirical evidence. This method can have a strong impact on making LLMs more customizable and accessible. However, I strongly recommend that the author provide further empirical evidence that demonstrate the effectiveness of this method on more tasks/datasets - this would significantly improve the quality of this work.\n\nComments:\n- The theoretical grounding in online learning is well-detailed and provides a clear explanation as to why the method works. The empirical validation further strengthens these theoretical claims.\n- The proposed method is designed for practical applications. This is an important factor when applying LLMs in real-world situations.\n\nSuggestions for improvement\n- Consider expanding the evaluation to include a wider range of domains. Specifically, investigate tasks tasks that require general alignment rather than user-specific tasks. This would provide a clearer picture of DITTO’s versatility and scalability. I think even negative results would be very informative.\n- It would be helpful to include a more detailed analysis of how the quality of demonstrations impacts performance. This could include testing DITTO with intentionally ambiguous or low-quality demonstrations to assess robustness.\n- The limitations section could be expanded with a deeper discussion on the trade-offs of using few-shot demonstrations. Exploring scenarios where the approach might fail or require adjustments would strengthen the paper’s transparency.\n- A more granular analysis of failure cases would add depth to the evaluation. This could involve detailed case studies highlighting scenarios where the method struggles.", "questions": "- How does the method scale with larger LLMs, and are there specific challenges in aligning models that have stronger RLHF priors?\n- How does DITTO perform in broader tasks that require more generalized alignment rather than user-specific customization? Could you provide insights into its scalability beyond niche tasks?\n- How sensitive is DITTO to the quality of demonstrations? Could you elaborate on strategies to mitigate the impact of poorly constructed or ambiguous demonstrations?\n- In terms of computational efficiency, how does DITTO compare with existing approaches when scaling to larger datasets or more complex tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- DITTO introduces a new approach to user-specific alignment by using a small set of demonstrations to generate online comparison data. This is innovative and practical for settings where data collection is costly.\n- The paper provides a strong theoretical justification for DITTO, grounding it in online imitation learning. The derivation explains why DITTO can outperform traditional methods like SFT in low-data scenarios.\n- The paper completes various experiments, demonstrating DITTO’s effectiveness across static benchmarks (e.g. email writing, news articles) and in a user study. The method consistently outperforms traditional techniques like few-shot prompting and SFT, providing convincing empirical support.\n- The authors have made the code accessible, allowing for others to reproduce and validate their results", "weaknesses": "- Limited exploration is done into how DITTO scales to broader and more diverse tasks that may require a more generalized alignment. This is seen in how the experiments primarily focus on a small number of demonstrations.\n- DITTO’s approach heavily relies on the quality of user-provided demonstrations. If demonstrations are unclear or poorly constructed, the alignment could suffer. This could limit DITTO’s real-world applicability when high-quality demonstrations are not readily available.\n- The paper primarily focuses on text-based tasks. However, it would be interesting to understand the effectiveness of DITTO’s method in aligning LLMs in other modalities or more complex reasoning situations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZpkBeYoSOj", "reviewer": "Reviewer_HDJC", "summary": "This paper proposes an alternative to RLHF which is effective at learning from a few demonstrations. The paper shows that this method outperforms supervised finetuning and few-shot learning. The paper shows human eval results, qualitative samples, and various quantitative evals to show that DITTO is effective at getting models to adapt to a new task based on a few examples. The paper also discusses the connection between DITTO and imitation learning, explaining why the method might outperform just using supervised learning (as is common in LLM work) to do imitation learning, and why you might even expect to get better performance than the existing examples. The algorithm basically works by using the LLM to generate examples that are assumed to be worse than the demonstrations, then constructing pairwise preferences between the LLM generated samples and the expert demos (and possibly between earlier vs. later LLM checkpoints in the training run), then using DPO to learn from the constructed pairwise ranking.", "questions": "Some questions I had while reading the paper (some might be out of scope for this paper or for the rebuttal period):\n\nDoes this work for 1-shot learning?\nDo all fine-tuning runs use LoRA?\nDoes this work better for highly realistic/plausible synthetic data? Does this look indistinguishable to an LLM from some other real distribution, even after the LLM is fine-tuned? That would be a really compelling use case for this (to help with doing automated red teaming, with realistic looking inputs that closely match the target data distribution)\nDoes it help few-shot to explicitly instruct needed to be very close to few-shot examples in style? Or was that just tried for fitting zero-shot?\nHow do you choose hyperparameters with such a small number of examples? Like SFT/DPO ones? If you were doing any hyperparam selection, you might run into issues like described here: https://arxiv.org/abs/2105.11447\nHow did you pick the 20/80 data mix? How robust is that across datasets/settings?\nHow well does DITTO work in higher data regime? That would be the most compelling result, if it could replace RLHF when using large amounts of data (which is how it's often used in practice)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-The method outperforms few-shot learning, which is surprising/impressive to me, I didn't expect that and it was one of my main doubts about the method from just reading the abstract. I think this could be a pretty compelling method potentially for doing automated red teaming, where you'd want to match some target data distribution as closely as possible, in order to elicit the most representative behavior from the model you're red teaming. This could then help with eliminating backdoors or sleeper agents (https://arxiv.org/abs/2401.05566), which is probably the application of this that I think most stands out to me as different from what is covered from prior work (I'm not that aware of many effective supervised learning alternatives like DITTO)\n-The method seems useful for settings where fine-tuning an existing RLHF model (though I'm a bit less clear how broadly this would work / if this would replace RLHF for finetuning across lots of tasks or just some specific ones related to adapting the model's style or writing)\n-Well-written paper, easy to follow\n- The approach itself is clever, and it's interesting/surprising to me that it works well\n-Nice that there are some human eval results, those helped to convince me that there are real gains with the method over few-shot learning (where it's clear the model hasn't adapted its behavior much).\n-Likewise, the samples in the appendix are quite helpful for the above too\n-Analysis in Table 3 is great for explaining why this might work\n-Section 5 analysis is great/helpful.\nConnecting DITTO to imitation learning is helpful for explaining why this is interesting, and why it would work.\n\nI would give this paper a 7/10 rating, somewhere between marginal accept and accept (but the form would only allow a 6 or an 8).", "weaknesses": "-Would be most compelling if evaluated on higher expertise tasks: like coding complex tasks or forecasting. Seems like one of the main areas of relevance, given that this is where we might expect to be in the low-data regime where we want to get the most of our a small amount of (high-quality or hard to obtain) data. I also expect it to be harder/more impressive to see gains in these domains. Currently, the tasks are fairly basic and all writing related. Enough for a proof-of-concept but probably not complex enough to make me want to use DITTO instead of RLHF everywhere.\n-One of the most interesting applications of the method would be to get generalization beyond what the demos are able to provide, it would be very compelling if this method led to generalization beyond the demos (which seems to be potentially possible if the method is working well, based on the discussion in the paper, if I understand correctly)\n-The paper would ideally compare to Constitutional AI, another popular RLHF-alternative. (Though this could take some time to reimplement, if there aren't publicly available implementations). More generally, I'm unsure if the method outperforms using principles to guide/instruct the model (especially if those principles are derived by an LLM from the few examples, which would be most comparable to the existing method/setting). The results showing that prompting doesn't fix all the issues help here, but more sophisticated methods like Constitutional AI could still outperform DITTO here\n- I'd love to see scaling trends on how well this works across model sizes -- it would be most compelling if the gains in task reward over supervised learning / few-shot learning seem to improve as models grow larger, rather than shrink\n- I'm not sure but it's possible to me that this method partly beats few-shot learning on RLHF models because RLHF models are resistant to adaptation with few-shot examples, but that the method wouldn't outperform few-shot learning if using pretrained LLMs (or maybe even just instruction-tuned/supervised learning finetuned models). That could potentially be a helpful experiment to run (and more compelling if DITTO also outperforms other adaptation techniques when comparing on a pretrained language model)\n\nMinor:\n-Would be nice to show at least 1-2 examples in main paper, to show the sample quality. (Having these in the appendix is helpful though)\n-The method could be explained more clearly sooner in the paper, I think that I didn't understand the actual algorithm until page 4 or so, when it would be nice to understand it from the intro or abstract itself", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)