Aligning Language Models with Demonstrated Feedback

Authors: Omar Shaikh, Michelle S. Lam, Joey Hejna, Yijia Shao, Hyundong Justin Cho, Michael S. Bernstein, Diyi Yang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: personalization, few-shot learning, human computer interaction, alignment

Abstract:
Language models are aligned to emulate the collective voice of many, resulting in outputs that align with no one in particular. Steering LLMs away from generic output is possible through supervised finetuning or RLHF, but requires prohibitively large datasets for new ad-hoc tasks. We argue that it is instead possible to align an LLM to a specific setting by leveraging a very small number ($<10$) of demonstrations as feedback. Our method, Demonstration ITerated Task Optimization (DITTO), directly aligns language model outputs to a user's demonstrated behaviors. Derived using ideas from online imitation learning, DITTO cheaply generates online comparison data by treating users' demonstrations as preferred over output from the LLM and its intermediate checkpoints. We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts. Additionally, we conduct a user study soliciting a range of demonstrations from participants ($N=16$). Across our benchmarks and user study, we find that win-rates for DITTO outperform few-shot prompting, supervised fine-tuning, and other self-play methods by an average of 19\% points. By using demonstrations as feedback directly, DITTO offers a novel method for effective customization of LLMs.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1729451265908

thread_type:
official_review

<thread_object>
creator: reviewer_hdjc

<content>
summary:
This paper proposes an alternative to RLHF which is effective at learning from a few demonstrations. The paper shows that this method outperforms supervised finetuning and few-shot learning. The paper shows human eval results, qualitative samples, and various quantitative evals to show that DITTO is effective at getting models to adapt to a new task based on a few examples. The paper also discusses the connection between DITTO and imitation learning, explaining why the method might outperform just using supervised learning (as is common in LLM work) to do imitation learning, and why you might even expect to get better performance than the existing examples. The algorithm basically works by using the LLM to generate examples that are assumed to be worse than the demonstrations, then constructing pairwise preferences between the LLM generated samples and the expert demos (and possibly between earlier vs. later LLM checkpoints in the training run), then using DPO to learn from the constructed pairwise ranking.

strengths:
-The method outperforms few-shot learning, which is surprising/impressive to me, I didn't expect that and it was one of my main doubts about the method from just reading the abstract. I think this could be a pretty compelling method potentially for doing automated red teaming, where you'd want to match some target data distribution as closely as possible, in order to elicit the most representative behavior from the model you're red teaming. This could then help with eliminating backdoors or sleeper agents (https://arxiv.org/abs/2401.05566), which is probably the application of this that I think most stands out to me as different from what is covered from prior work (I'm not that aware of many effective supervised learning alternatives like DITTO)
-The method seems useful for settings where fine-tuning an existing RLHF model (though I'm a bit less clear how broadly this would work / if this would replace RLHF for finetuning across lots of tasks or just some specific ones related to adapting the model's style or writing)
-Well-written paper, easy to follow
- The approach itself is clever, and it's interesting/surprising to me that it works well
-Nice that there are some human eval results, those helped to convince me that there are real gains with the method over few-shot learning (where it's clear the model hasn't adapted its behavior much).
-Likewise, the samples in the appendix are quite helpful for the above too
-Analysis in Table 3 is great for explaining why this might work
-Section 5 analysis is great/helpful.
Connecting DITTO to imitation learning is helpful for explaining why this is interesting, and why it would work.

I would give this paper a 7/10 rating, somewhere between marginal accept and accept (but the form would only allow a 6 or an 8).

weaknesses:
-Would be most compelling if evaluated on higher expertise tasks: like coding complex tasks or forecasting. Seems like one of the main areas of relevance, given that this is where we might expect to be in the low-data regime where we want to get the most of our a small amount of (high-quality or hard to obtain) data. I also expect it to be harder/more impressive to see gains in these domains. Currently, the tasks are fairly basic and all writing related. Enough for a proof-of-concept but probably not complex enough to make me want to use DITTO instead of RLHF everywhere.
-One of the most interesting applications of the method would be to get generalization beyond what the demos are able to provide, it would be very compelling if this method led to generalization beyond the demos (which seems to be potentially possible if the method is working well, based on the discussion in the paper, if I understand correctly)
-The paper would ideally compare to Constitutional AI, another popular RLHF-alternative. (Though this could take some time to reimplement, if there aren't publicly available implementations). More generally, I'm unsure if the method outperforms using principles to guide/instruct the model (especially if those principles are derived by an LLM from the few examples, which would be most comparable to the existing method/setting). The results showing that prompting doesn't fix all the issues help here, but more sophisticated methods like Constitutional AI could still outperform DITTO here
- I'd love to see scaling trends on how well this works across model sizes -- it would be most compelling if the gains in task reward over supervised learning / few-shot learning seem to improve as models grow larger, rather than shrink
- I'm not sure but it's possible to me that this method partly beats few-shot learning on RLHF models because RLHF models are resistant to adaptation with few-shot examples, but that the method wouldn't outperform few-shot learning if using pretrained LLMs (or maybe even just instruction-tuned/supervised learning finetuned models). That could potentially be a helpful experiment to run (and more compelling if DITTO also outperforms other adaptation techniques when comparing on a pretrained language model)

Minor:
-Would be nice to show at least 1-2 examples in main paper, to show the sample quality. (Having these in the appendix is helpful though)
-The method could be explained more clearly sooner in the paper, I think that I didn't understand the actual algorithm until page 4 or so, when it would be nice to understand it from the intro or abstract itself

questions:
Some questions I had while reading the paper (some might be out of scope for this paper or for the rebuttal period):

Does this work for 1-shot learning?
Do all fine-tuning runs use LoRA?
Does this work better for highly realistic/plausible synthetic data? Does this look indistinguishable to an LLM from some other real distribution, even after the LLM is fine-tuned? That would be a really compelling use case for this (to help with doing automated red teaming, with realistic looking inputs that closely match the target data distribution)
Does it help few-shot to explicitly instruct needed to be very close to few-shot examples in style? Or was that just tried for fitting zero-shot?
How do you choose hyperparameters with such a small number of examples? Like SFT/DPO ones? If you were doing any hyperparam selection, you might run into issues like described here: https://arxiv.org/abs/2105.11447
How did you pick the 20/80 data mix? How robust is that across datasets/settings?
How well does DITTO work in higher data regime? That would be the most compelling result, if it could replace RLHF when using large amounts of data (which is how it's often used in practice)
</content>
</thread_object>

</thread>
<thread>
creation date:
1729988372782

thread_type:
official_review

<thread_object>
creator: reviewer_41at

<content>
summary:
The paper proposes a method called Demonstration Iterated Task Optimization (DITTO), designed to align large language models (LLMs) with user-specific preferences using a minimal number of user-provided demonstrations. This method eliminates the need for large-scale datasets typically required for supervised fine-tuning or RLHF. The paper claims that DITTO can significantly improve the alignment of LLMs for user-driven tasks and offers a practical solution for customizing language models. The paper explains their theoretical insights from online imitation learning with practical implementations, demonstrating effective customization for real-world applications like email writing and author-specific content generation.

I recommend accepting this paper, as it tackles a significant challenge and presents an interesting solution that is well-supported in theory and through empirical evidence. This method can have a strong impact on making LLMs more customizable and accessible. However, I strongly recommend that the author provide further empirical evidence that demonstrate the effectiveness of this method on more tasks/datasets - this would significantly improve the quality of this work.

Comments:
- The theoretical grounding in online learning is well-detailed and provides a clear explanation as to why the method works. The empirical validation further strengthens these theoretical claims.
- The proposed method is designed for practical applications. This is an important factor when applying LLMs in real-world situations.

Suggestions for improvement
- Consider expanding the evaluation to include a wider range of domains. Specifically, investigate tasks tasks that require general alignment rather than user-specific tasks. This would provide a clearer picture of DITTO’s versatility and scalability. I think even negative results would be very informative.
- It would be helpful to include a more detailed analysis of how the quality of demonstrations impacts performance. This could include testing DITTO with intentionally ambiguous or low-quality demonstrations to assess robustness.
- The limitations section could be expanded with a deeper discussion on the trade-offs of using few-shot demonstrations. Exploring scenarios where the approach might fail or require adjustments would strengthen the paper’s transparency.
- A more granular analysis of failure cases would add depth to the evaluation. This could involve detailed case studies highlighting scenarios where the method struggles.

strengths:
- DITTO introduces a new approach to user-specific alignment by using a small set of demonstrations to generate online comparison data. This is innovative and practical for settings where data collection is costly.
- The paper provides a strong theoretical justification for DITTO, grounding it in online imitation learning. The derivation explains why DITTO can outperform traditional methods like SFT in low-data scenarios.
- The paper completes various experiments, demonstrating DITTO’s effectiveness across static benchmarks (e.g. email writing, news articles) and in a user study. The method consistently outperforms traditional techniques like few-shot prompting and SFT, providing convincing empirical support.
- The authors have made the code accessible, allowing for others to reproduce and validate their results

weaknesses:
- Limited exploration is done into how DITTO scales to broader and more diverse tasks that may require a more generalized alignment. This is seen in how the experiments primarily focus on a small number of demonstrations.
- DITTO’s approach heavily relies on the quality of user-provided demonstrations. If demonstrations are unclear or poorly constructed, the alignment could suffer. This could limit DITTO’s real-world applicability when high-quality demonstrations are not readily available.
- The paper primarily focuses on text-based tasks. However, it would be interesting to understand the effectiveness of DITTO’s method in aligning LLMs in other modalities or more complex reasoning situations.

questions:
- How does the method scale with larger LLMs, and are there specific challenges in aligning models that have stronger RLHF priors?
- How does DITTO perform in broader tasks that require more generalized alignment rather than user-specific customization? Could you provide insights into its scalability beyond niche tasks?
- How sensitive is DITTO to the quality of demonstrations? Could you elaborate on strategies to mitigate the impact of poorly constructed or ambiguous demonstrations?
- In terms of computational efficiency, how does DITTO compare with existing approaches when scaling to larger datasets or more complex tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730525694821

thread_type:
official_review

<thread_object>
creator: reviewer_yvim

<content>
summary:
This paper identifies a key issue: current LLMs, aligned to represent the collective voice of many, often fail to align specifically with any individual preference due to contradictions among them. While guiding LLMs toward a general preference is feasible, it requires substantial preference data. The authors propose a method, DITTO, to align LLMs to specific settings using fewer than 10 demonstrations drawn from existing interaction logs or direct edits to LLM outputs. These demonstrations are treated as "golden" examples, while outputs from current and previous LLM checkpoints are rejected. Through author attribution tasks and user studies, they demonstrate the effectiveness and sample efficiency of DITTO.

strengths:
The paper introduces DITTO, a novel method designed to guide LLMs toward specific settings for effective customization, achieving sample efficiency with fewer than 10 demonstrations. DITTO outperforms strong baselines, including SFT and GPT-4 with few-shot prompting. Additionally, a detailed user study further reinforces the reliability of DITTO.

weaknesses:
1. The static experiments in Section 4.1 are not particularly convincing. Have you considered testing additional baselines or employing other automatic evaluation methods, such as calculating sentence embedding similarity to compare styles?
2. Have you evaluated DITTO on more benchmarks or tested its generalization ability? I noticed that only three authors were used for validation or testing. Can the DITTO method generalize to tasks beyond writing?

questions:
1. In Section 3, you introduce the core method of DITTO and compare it with online imitation learning. What is the purpose of Section 3.3?
2. How did you determine the percentage distribution of the paired data, specifically the 70% online data, 20% replay data, and 10% intermodel pair data?
3. In Table 1, for the CMCC dataset, why do the zero-shot and few-shot results from GPT-4 appear the same in column a9, both at 40.28%? Additionally, why do both SFT and DITTO show results of 81.94% without any improvement? How would you comment on this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718404688

thread_type:
official_review

<thread_object>
creator: reviewer_7eqt

<content>
summary:
The paper introduces a novel method, Demonstration Iterated Task Optimization (DITTO), for training large language models (LLMs) with expert demonstration datasets in a more data-efficient manner. Through a mathematical derivation, the authors illustrate how DITTO functions as a form of online imitation learning. They validate the method's effectiveness by utilizing a GPT-4 evaluation scheme and compare it against several other approaches, including Supervised Fine-Tuning (SFT), SPIN, and few-shot prompting. The authors conclude that DITTO is particularly advantageous for training LLMs to adopt specific writing styles or user preference tuning, outperforming other methods in these areas.

strengths:
- The paper proposes a data-efficient training method that enables LLMs to follow expert demonstrations. The Reinforcement Learning from Human Feedback (RLHF) data can be continuously generated by simply comparing expert demonstrations with the intermodel's responses. This approach can also be seen as a blend of Reinforcement Learning from AI Feedback (RLAIF) and RLHF, making it a reasonable and effective method.
- The authors demonstrate the performance improvements of DITTO-trained models using GPT-4 evaluation and validate the method's effectiveness through a large-scale user study.
- They provide a theoretical perspective on the connection between online imitation learning and demonstrate that online imitation learning can outperform Supervised Fine-Tuning (SFT). The mathematical derivation and explanations are clear, and the results are further supported by meticulously designed ablation studies.

weaknesses:
- The authors did not investigate potential side effects, such as performance degradation on other benchmark datasets, after training with DITTO. Since the LLM is fine-tuned exclusively on targeted demonstrations, there’s a risk of significant performance drops in broader tasks. It is essential to preserve the LLM's original knowledge and abilities while adjusting its output to align with specific style and preference.
- Also they overlooks the computational inefficiency of iterative training in an online imitation learning framework. This process requires substantial time and GPU resources, as it involves initializing the policy 𝜋0 (equivalent to SFT), generating responses from 𝜋0, training with DPO, and then iterating to produce 𝜋1, and so forth. These steps are difficult to reproduce and demand more computational power than SFT baseline. Furthermore, achieving faster response generation in the trained LLM would require additional engineering efforts. Although DITTO improves data efficiency, it is also crucial to consider computational efficiency, given the high costs of training and generating responses with LLMs.
- The authors did not explore the limitations of the DPO algorithm or other potential approaches for training LLMs in a Reinforcement Learning from Human Feedback (RLHF) framework. It is known that the DPO algorithm can pose risks when training on preference datasets, as it may forget data from the "winning" side due to inherent mathematical issues.

questions:
- Do you think DITTO would be effective for the coding skills or mathematical problem solving skills of an LLM?
- Have you attempted training the LLM without LoRA, using full fine-tuning instead?
- What kind of source code is used to generate online responses? If you were to train a much larger LLM (such as LLAMA 72B), would it be feasible to apply the online imitation learning method in the same way?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732326348304

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1 / N)

comment:
# General Response

We thank all the reviewers for taking the time to review our work, and for the thoughtful and thorough feedback! In particular, we appreciate that the reviewers valued DITTO’s novel approach to collecting and generating preferences through demonstration, our method’s theoretical grounding, and our focus on the practical customization of LLMs with limited feedback. Additionally, reviewers appreciated our meticulous ablations, our user study, and our qualitative analysis of examples.

While we address each reviewer’s feedback, we want to address shared questions around 1. generalization, 2. RLHF priors, 3. finetuning methods (LoRA v.s. full) and 4. efficiency.  All changes are in the revised manuscript (and are marked in blue).

Please let us know if you have any other questions- we're happy to answer followups!

## 1. Generalization Ability - All Reviewers

All reviewers raised questions regarding the generalization abilities of DITTO, asking if  DITTO-ed models extrapolate beyond the training demonstrations. We focus on three main points. 

1.1 Will DITTO’s approach generalize to coding or reasoning tasks?

1.2 What about forgetting general capabilities / alignment?

1.3 Models must non-trivially generalize to perform well on our static benchmarks.

### 1.1 Will DITTO’s approach generalize to coding or reasoning tasks?

We would not expect DITTO to improve a model’s reasoning abilities. DITTO helps “focus” a model’s output distribution towards a specific demonstrated behavior, relying on online imitation learning for iterative improvement. In our writing evaluations, we suspect the model was trained on samples of writing similar to the author—DITTO iteratively brings this distribution to the fore. If the LLM was not pre-trained on many samples of expert reasoning, we doubt this distribution could surface; however, we don’t expect overall performance to worsen either (see 1.3). 

Unfortunately, we are also unaware of pre-existing datasets with demonstrated feedback over complex reasoning problems (GPQA [1], for example, focuses on MCQ and not open-ended responses). We’ve discussed these concerns more explicitly in our revised limitations / future work.

We do, however, think DITTO would help with style for coding or reasoning problems. Aligned LLMs often generate verbose explanations or highly commented code. Programmers often have a stylistic preference when it comes to writing code. Here’s a qualitative example where we trained DITTO on a handful of coding demonstrations where we minified code.

Prompt
```
Write python code to read a JSON file.
```

Mistral Instruct
```
To read a JSON file in Python, you can use the json module. Here's a simple example:

import json

# Open the JSON file
with open('your_file.json', 'r') as f:
    # Load JSON data from file
    data = json.load(f)

# Now you can work with the data
print(data)

In this example, replace 'your_file.json' with the path to your JSON file. The json.load() function reads the JSON data from the file and returns it as a Python object (e.g., dictionary or list).
```

DITTO:
```
JSON.parse(require('fs').readFileSync(f.json'));
```

Even for coding tasks, we expect DITTO to have some impact on model behavior. **We’ve mentioned these reflections in our updated future work section.**

[1] Rein et al. 2023. GPQA: A Graduate-Level Google-Proof Q&A Benchmark
</content>
</thread_object>

</thread>


Metareview:
metareview:
## Summary
This paper introduces Demonstration ITerated Task Optimization (DITTO), a method for aligning language models to specific tasks using fewer than 16 demonstrations. Unlike methods like RLHF or supervised fine-tuning, which often require large datasets, DITTO leverages ideas from online imitation learning to align models efficiently. The approach constructs pairwise preferences between user-provided demonstrations and outputs generated by the model or its earlier checkpoints. These preferences are then used to guide training through a method like DPO. Experiments across tasks such as writing news articles, emails, and blog posts demonstrate that DITTO significantly outperforms alternatives like few-shot prompting and supervised fine-tuning, with a reported average improvement in win rates of 19 percentage points. A user study further supports the method’s effectiveness in customizing language model behavior.

## Decision 

Overall, the paper provides a compelling contribution to aligning language models efficiently and effectively. The combination of theoretical grounding, empirical evidence, and practical utility justifies its acceptance.

The approach is grounded in online imitation learning, with clear theoretical derivations that explain why DITTO can outperform existing methods like supervised fine-tuning (SFT) in low-data settings. The method's connection to reinforcement and imitation learning is well-explained. 

Extensive experiments demonstrate that DITTO outperforms strong baselines, including few-shot prompting and SFT, across multiple tasks such as email writing, news articles, and blog posts. The method shows an average improvement of 19 percentage points in win rates, validated through GPT-4 evaluations and a large-scale user study.