PAD: Personalized Alignment of LLMs at Decoding-time

Authors: Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, Zuozhu Liu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Alignment, personalization

Abstract:
Aligning with personalized preferences, which vary significantly across cultural, educational, and political differences, poses a significant challenge due to the computational costs and data demands of traditional alignment methods. In response, this paper presents Personalized Alignment at Decoding-time (PAD), a novel framework designed to align LLM outputs with diverse personalized preferences during the inference phase, eliminating the need for additional training. By introducing a unique personalized reward modeling strategy, this framework decouples the text generation process from personalized preferences, facilitating the generation of generalizable token-level personalized rewards. The PAD algorithm leverages these rewards to guide the decoding process, dynamically tailoring the base model’s predictions to personalized preferences. Extensive experimental results demonstrate that PAD not only outperforms existing training-based alignment methods in terms of aligning with diverse preferences but also shows significant generalizability to preferences unseen during training and scalability across different base models. This work advances the capability of LLMs to meet user needs in real-time applications, presenting a substantial step forward in personalized LLM alignment.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730627766615

thread_type:
official_review

<thread_object>
creator: reviewer_s7gt

<content>
summary:
This paper propose personalized alignment at decoding-time, which aligns LLM outputs with personalized preferences during the inference phase, without the need for additional training. Their method use a unified policy for all personalization and use a personalized reward model to determine the personalized weights.

strengths:
1 Personalized alignment and decode-time alignment are both important topics to study.
2 Their method can generalize to customized preferences that are unseen during the training phase.
3.The writing of this paper is good, especially in the method section, which is well formalized.

weaknesses:
1 According to Table 2, Preference Prompting that simpliy add preference information to the prompt without training the model can achieve good performance (GPT4 winrate) comparing to other methods that requires training.
2 I think a case study can strengthen this paper. For example, showing how apply different personalized configurations to the same instruction steers the genenrated responses to varied directions.
3. What is the benefit of decoding-time alignment? It seems to me that PAD still need to train the backbone model, which seems to violate the motivation of decoding-time algorithm.

questions:
Why Best-of-Ndecoding is worse than other decoding strategies in helpful, humor, and diversity? Because BoN is always thought to be an technique that enhances the model performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706020028

thread_type:
official_review

<thread_object>
creator: reviewer_nqpu

<content>
summary:
The paper presents a framework called Personalized Alignment at Decoding-time (PAD) which aligns the outputs of large language models (LLMs) with diverse personal preferences during inference, without the need for additional training. The core idea is to separate personalized preferences from the text generation process, which is modeled as a Markov Decision Process. This separation enables the development of a Personalized Reward Model (PRM) that generates token-level rewards based on individual preferences. PAD offers several advantages: 1) it is training-free (for policy model) 2) it uses a single reward model. 3) it can generalize to preferences not seen during training.

strengths:
1. The paper is well-written and easy to follow. The authors build up the concept of PAD clearly.
2. Decoupling personalized preferences from the Markov Decision Process and the idea of the generation of generalizable token-level personalized reward is unique.
3. It does not need to further train the policy nor need to train multiple reward models unlike many existing works.

weaknesses:
1. The paper lacks a Pareto front analysis for scenarios involving multiple rewards. For experiments that combine multiple objectives (like "Harmless and Humor" or "Expert, Informative, and Creative" in Figure 3), including such an analysis would help illustrate the trade-offs between different preferences.
2. Table 3 only displays PAD's performance. Including results from other baseline methods across different models would be nice.
3. During inference, the need to load and run an additional LLM for the personalized reward model introduces computational overhead.
4. Some minor typos and errors that did not influence my rating: 1) In Table 2, under the "Helpful" RM column for MetaAligner on the HelpSteer dataset, the highest value isn't highlighted. 2) Line 322: "LLama" should be "Llama."

questions:
1. Could you include pseudocode or an algorithm box for PAD? This would be helpful for the audience to understand the training and inference details.
2. How are the prompts for personalized preferences constructed in your experiments? Have you tested the robustness of PAD by using paraphrased versions of the prompts used during training to see if it maintains performance with variations in preference expressions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712684115

thread_type:
official_review

<thread_object>
creator: reviewer_6ie9

<content>
summary:
This paper proposes a method for achieving personalized alignment by modifying logits, named Personalized Alignment at Decoding-time (PAD). This approach allows for personalized outputs without modifying model parameters and demonstrates improved performance across the "Helpful," "Harmless," and "Humor" dimensions. By introducing a personalized reward model, PAD integrates user-specific preferences during text generation, enhancing model flexibility and adaptability.

strengths:
- **Theoretical and empirical validation**: PAD combines theoretical innovation with empirical evidence, reinforcing its effectiveness. Experimental results align with theoretical assumptions, showcasing PAD’s strengths in accommodating personalized preferences.
- **Superior performance over existing personalized alignment methods**: Compared to other approaches, PAD performs better across multiple metrics and significantly reduces computational costs, making it more feasible for practical applications.
- **Comprehensive ablation studies**: The paper includes detailed ablation studies, exploring PAD's performance across various hyperparameters settings and decoding strategies. This supports the model’s robustness and generalization capability.

weaknesses:
- **Unclear description of training process**: While the paper focuses on theoretical derivations, the description of the actual training process lacks clarity, omitting key implementation details. Adding specific explanations of the training process would improve reader comprehension.
- **Inconsistent notation and terminology**:
  - The definition of variable $a$ in line 287 is unclear, and the inconsistent use of subscript $t$ could lead to confusion.
  - The term PRM is commonly understood to mean "Process Reward Model," which differs from its use in this paper. Clarification on this term would help avoid ambiguity.
- **Inaccuracy in "Model Agnostic" claim**: The term "Model Agnostic" in Section 4.4 is somewhat misleading. Since PAD relies on logits, retraining is necessary for different models, meaning it is not entirely model-agnostic.
- **Lack of comparisons**: There are other methods that enhance model performance by modifying logits[1] or directly correct the response[2]. Comparing these approaches with PAD would highlight the distinct advantages of PAD.

[1] Konen K, Jentzsch S, Diallo D, et al. Style Vectors for Steering Generative Large Language Model[J]. arXiv preprint arXiv:2402.01618, 2024.

[2] Ji J, Chen B, Lou H, et al. Aligner: Achieving efficient alignment through weak-to-strong correction[J]. arXiv preprint arXiv:2402.02416, 2024.

questions:
1. During the first training phase, is personalized data $p$ included? If so, in what form is it integrated? Could you provide an example?
2. Is there a distinction between the training data used in the two stages?
3. Theoretically, PAD appears highly similar to DPO, differing only by the additional hyperparameter $β$ and the personalized parameter $W_p$. Can the first training stage be understood as DPO, with the second training stage merely training a value head?
4. What does $\hat{\pi}_{\text{ref}}$ represent in Equation (13)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731474652856

thread_type:
official_review

<thread_object>
creator: reviewer_lds8

<content>
summary:
The paper introduces a novel framework for aligning large language model (LLM) outputs with diverse personalized preferences during the inference phase, eliminating the need for additional training and different models for different preferences. The authors theoretically derive a different formulation of the personalized preference model policy with unique personalized reward modeling strategy, which decouples the text generation process from personalized preferences, allowing for generalizable token-level personalized rewards. This decoupling allows to have a single model trained once and use test-time compute and reward computation at each generation step to achieve the preference stated.

strengths:
1. The paper discusses that PAD requires only a single policy model aligned with general preferences, eliminating additional training. This means that the algorithm operates efficiently without the need for the creation of multiple specialized models. The contrast with previous works in these aspects is detailed very well in Table 1.

2. Great presentation of the theoretical aspects of the algorithm. The experiment section a bit lacking in details though. Yet the analysis of different baselines and different datasets is commendable.

weaknesses:
1. Not much details are shared on how the model can generalize to unseen preferences not seen during the training. The w_p is dependent on p being specified so new preferences would need some retraining.

2. While the experiment section compares the performance of the model against the baselines, the latency and cost is not compared which will have practical implication in real world use.

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732266323510

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
Dear reviewers,

**We sincerely appreciate the time and effort you devoted to reviewing our paper. Your detailed feedback has been invaluable, and we have made comprehensive revisions based on your insightful comments.**

We appreciate that the reviewers acknowledge the **advantages** of our work:

1. **Technical Novelty**: Our approach uniquely decouples personalized preferences from the Markov Decision Process and introduces the generation of generalizable token-level personalized rewards (Reviewer NqpU). This innovative concept stands out because it does not require further training of the policy or multiple reward models, which is a common necessity in many existing works (Reviewer NqpU). The paper provides a detailed contrast with previous work, highlighting the novel aspects of our methodology (Reviewer LdS8). 

2. **Empirical effectiveness**: The combination of theoretical innovation and empirical evidence significantly reinforces the effectiveness of our method. The paper conducts extensive ablation studies to explore PAD’s robustness and its capability to generalize across different conditions, underscoring the practical utility of our approach (Reviewer 6ie9).

3. **Clear presentation**: The paper is well-received for its excellent presentation of the theoretical aspects of the algorithm (Reviewer LdS8). It is well-written, organized, and easy to follow, effectively building up the concept of PAD (Reviewer NqpU). The methods section, in particular, is noted for its clarity and formalization, making complex ideas accessible and understandable (Reviewer s7Gt).

On the other hand, we have diligently addressed all the **concerns** by conducting extensive experiments and providing detailed clarifications. Additional experiments and clarifications have been incorporated into the revised manuscript. We have made substantial updates to address the concerns:

1. **Enhanced Experimental Results:**

 - **Computational Cost Analysis** (Reviewers LdS8 and NqpU): Added a new paragraph in Section 4.3 specifically discussing the computational cost associated with PAD. 
-  **Expanded Baseline Comparative Analysis** (Reviewer 6ie9): Included two additional baselines in the revised manuscript's Table 2 to enhance comparative analysis.
-  **Pareto Front Analysis** (Reviewer NqpU): Introduced a new Section C.1 discussing the Pareto front analysis on the trade-offs between different preferences for PAD.
- **Expanded Base Model Comparison** (Reviewer NqpU): Added comparisons of PAD performance across various base models to demonstrate its applicability and effectiveness.
- **Robustness Analysis** (Reviewer NqpU): Conducted new experiments on the robustness of PAD to variations in preference expressions, underscoring its stability.
-  **Case Studies** (Reviewer s7Gt): Included case studies illustrating how our method can guide the generation of responses in various directions.

2. **Enriched Implementation Details:**
- **Detailed implementation Details** (Reviewers 6ie9 and NqpU): Added two algorithm boxes that provide detailed pseudocode for the practical implementations of PAD, enhancing the manuscript's clarity.
- **Extended Experimental Explanations** (Reviewer 6ie9): Provided additional explanations of the motivation and experimental setup in Section 4.4.
- **Personalized Prompt Template Disclosure** (Reviewers 6ie9 and s7Gt): Provided the prompt template to elucidate how the prompts for personalized preferences are constructed.

**We hope that these revisions adequately address your concerns. Since the discussion phase is concluding, we would be grateful if you would kindly let us know of any other concerns and if we could further assist in clarifying any other issues.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Personalized Alignment at Decoding-time (PAD) which aims to to align LLM outputs with personalized preferences during inference. The reviewers generally found the proposed personalized reward modeling strategy to be novel and this decoding stage alignment problem to be interesting and important. The authors have well addressed the reviewers' concerns in the rebuttal.

A minor note: The authors should update their references with the conference proceeding version (if accepted) instead of the arXiv version. 
For example:

@inproceedings{mudgalcontrolled,
  title={Controlled Decoding from Language Models},
  author={Mudgal, Sidharth and Lee, Jong and Ganapathy, Harish and Li, YaGuang and Wang, Tao and Huang, Yanping and Chen, Zhifeng and Cheng, Heng-Tze and Collins, Michael and Strohman, Trevor and others},
  booktitle={Forty-first International Conference on Machine Learning},
year={2024},
}

@inproceedings{meng2024simpo,
title={Sim{PO}: Simple Preference Optimization with a Reference-Free Reward},
author={Yu Meng and Mengzhou Xia and Danqi Chen},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
}