On-the-fly Preference Alignment via Principle-Guided Decoding

Authors: Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, Zhendong Mao
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: preference alignment, tuning-free alignment, principle-based decoding

Abstract:
With the rapidly expanding landscape of large language models, aligning model generations with human values and preferences is becoming increasingly important. Popular alignment methods, such as Reinforcement Learning from Human Feedback, have shown significant success in guiding models with greater control. However, these methods require considerable computational resources, which is inefficient, and substantial collection of training data to accommodate the diverse and pluralistic nature of human preferences, which is impractical. These limitations significantly constrain the scope and efficacy of both task-specific and general preference alignment methods. In this work, we introduce On-the-fly Preference Alignment via Principle-Guided Decoding (OPAD) to directly align
model outputs with human preferences during inference, eliminating the need for fine-tuning. Our approach involves first curating a surrogate solution to an otherwise infeasible optimization problem and then designing a principle-guided reward function based on this surrogate. The final decoding policy is derived by maximizing this customized reward, which exploits the discrepancy between the
constrained policy and its unconstrained counterpart. OPAD directly modifies the model’s predictions during inference, ensuring principle adherence without incurring the computational overhead of retraining or fine-tuning. Experiments show that OPAD achieves competitive or superior performance in both general and personalized alignment tasks, demonstrating its efficiency and effectiveness compared to state-of-the-art baselines.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730453057140

thread_type:
official_review

<thread_object>
creator: reviewer_dqvo

<content>
summary:
This work introduces a decoding method called OPAD that aligns model outputs with human preferences during inference. OPAD modifies model predictions to ensure compliance with human preferences. Compared to algorithms like DPO/PPO, OPAD achieves better performance without requiring additional training computational costs, which is very exciting.

strengths:
The authors introduce an alternative objective that maximizes the KL divergence between constrained and unconstrained policies during decoding. This approach quantifies the model's adherence to target preferences, thereby determining the reward in the alignment process. This method adjusts token prediction probabilities to promote preference compliance.

The major advantage of this paper is that OPAD outperforms DPO/PPO algorithms and Best-of-N methods with significantly fewer computational resources and requires no additional training.

weaknesses:
Recent research has focused on alignment during the decoding phase, such as [1][2][3][4], and the authors should discuss these works.

Some content lacks clarity, such as what constitutes a Principle in Alignment?  Additionally, to my knowledge, for models like Llama-3.1, if Principles are set in the system prompt, they can follow the described text. Therefore, experimenting with newer SOTA open-source models would help strengthen the paper's robustness.

The paper lacks theoretical explanation for why the proposed method is superior to placing Principles in ICL (In-Context Learning).

---

[1] DeAL: Decoding-time Alignment for Large Language Models

[2] Decoding-Time Language Model Alignment with Multiple Objectives

[3] Reward Steering with Evolutionary Heuristics for Decoding-time Alignment

[4]  Personality Alignment of Large Language Models

questions:
How are Principles set in Summarization and HH-RLHF?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643453829

thread_type:
official_review

<thread_object>
creator: reviewer_se9h

<content>
summary:
This paper introduces On-the-fly Preference Alignment via Principle-Guided Decoding (OPAD), addressing the challenge of aligning language models with human preferences without requiring expensive fine-tuning or extensive data collection. The method works by designing a principle-guided reward function based on a surrogate optimization solution and directly modifying model predictions during inference to ensure adherence to target principles. Experimental results demonstrate that OPAD achieves competitive or superior performance compared to existing baselines across both general and personalized alignment tasks.

strengths:
- The paper introduces a novel tuning-free alignment approach that uniquely leverages KL divergence between constrained and unconstrained policies during inference.
- The method demonstrates robust empirical validation through comprehensive experiments across multiple datasets and strong performance against RLHF, DPO, and other baseline methods.

weaknesses:
- The derivation of the surrogate optimization solution relies on an overly strong condition, namely 'The constraint c aligns well with the data distribution $P_{\text{data}}$'. In lines 214-215, there is a statement saying 'Direct optimization is infeasible since we have no access to $P_{\text{data}}$'. However, the authors did not explain how to design principles without access to $P_{\text{data}}$, and how to verify and guarantee this condition holds.
In fact, even if the constraint $c$ partially aligns with $P_{\text{data}}$, it could still lead to bad alignment. For example, let $P_{\text{data}}$ be 'professional medical advice', $P$ generate health-related statements with broad coverage but low accuracy, and $c$ be 'use formal medical terminology'. Maximizing $\text{KL}(P_c||P)$ might lead to technically-worded but meaningless or incorrect medical statements.
Therefore, the alignment quality of the principle is crucial. However, designing well-aligned principles without access to $P_{\text{data}}$ seems almost impossible; at least a surrogate $P_{\text{data}}$ is needed for guidance and principle evaluation. I believe the authors' experiments actually had access to $P_{\text{data}}$ (i.e., we clearly know we're dealing with the harmless and helpful aspects of hh-rlhf), so the designed principles could be well-aligned. But this contradicts the assumption of having no access to $P_{\text{data}}$.

- The proposed method lacks computational efficiency, running twice as slow as standard model decoding. While the tuning-free approach eliminates training costs, these costs are typically one-time investments, whereas inference requires continuous computational overhead. This computational burden limits the method's practical applications. Additionally, I recommend including computational efficiency comparisons with other in-context alignment methods to demonstrate whether OPAD offers advantages over similar approaches

-  For a paper focusing on decoding-time alignment, the Methodology section lacks both a complete description of the decoding steps and clear connections to the method overview presented in Figure 2, which reduces the paper's clarity. I recommend relocating the 'Relation with the residual EBMs' section to the later analysis and discussion portion, as it isn't a direct methodology description, and enhancing the method section with either a complete description of the decoding process or pseudocode to improve clarity.

questions:
- Why do more steps hurt performance? Theoretically, global rewards are necessary because the text that aligns locally can violate principles globally. For example, for a medical advice principle, 'Take aspirin for headaches. Then take ibuprofen for fever.' While each 2-token window appears medically sound, this advice globally violates medical safety.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719043997

thread_type:
official_review

<thread_object>
creator: reviewer_md8g

<content>
summary:
This paper introduces On-the-fly Preference Alignment via Principle-Guided Decoding (OPAD), a novel approach for aligning large language model outputs with human preferences without requiring resource-intensive fine-tuning or retraining. Unlike RLHF, which rely on substantial data and computational resources, OPAD achieves preference alignment directly during inference. The method leverages a principle-guided reward function derived from a surrogate solution to an otherwise intractable optimization problem, allowing it to steer model predictions towards desired behaviors dynamically. OPAD demonstrates efficiency and effectiveness across general and personalized alignment tasks, often outperforming established baselines.

strengths:
- OPAD offers a tuning-free alignment method inspired by the objectives of reinforcement learning, allowing it to align model outputs with human preferences without the need for additional tuning.
- The framework  can be applied to various tasks with minimal adjustments, making it a flexible solution for different alignment needs.
- Despite not requiring any model training, OPAD demonstrates competitive performance, often rivaling or surpassing traditional, training-intensive alignment methods.

weaknesses:
- In some cases, there is a high percentage of ties instead of clear wins or losses. How should we interpret this outcome?
- Since your method does not involve any model training, it would be valuable to demonstrate its performance on larger, more capable models to assess the trend and generalizability of your approach across different model scales.
- How does your method perform when LLMs do not achieve sufficient performance, such as smaller LLMs?

questions:
- Would it be more appropriate to rename Section 5 from 'Discussion' to 'Conclusion' for clarity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731852690175

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Response to All Reviewers

comment:
We thank all the reviewers for their insightful comments and helpful suggestions. We hope our responses and paper updates alleviate the concerns raised.

We are happy that our work is conceived as a “novel” (Se9h/mD8g), “flexible” (mD8g), and “exciting” (dQvo) tuning-free alignment approach to “various” tasks (mD8g) and “multiple datasets” (Se9h). We appreciate the reviewers perceive our experimental study as “comprehensive/robust” (Se9h) with “competitive/superior” (mD8g) and “strong” (Se9h) performance.

Following the reviewers’ feedback, we have updated our manuscript mainly with the following contexts:

- **More related works** (Section 2.2-Line149): Following the suggestion from dQvo, we discussed more related works that focus on decoding-phase alignment in the updated manuscript.
- **Detailed decoding pipeline** (Section 3.3-Line254): Following the suggestion from Se9h, we added a complete description of the decoding process to improve clarity.
- **Principle curation analysis** (Section 4.1-Line305): Following the feedback from Se9h and dQvo, we demonstrate how to design the task-specific principles in detail.
- **Method generalization** (Section 4.4-Line 426): Following the suggestion from mD8g, we test the proposed method with smaller and larger, more capable models. The results indicate that  OPAD can generalize well and is very effective on a wide range of mid-scale models (e.g., Vicuna-7B and Vicuna-13B), where it effectively “reinforces” principles to improve alignment.
- **Experiments with Llama 3 family** (Section 4.4-Line 438): Following the suggestion from dQvo, we have directly incorporated the experimental results with the Llama 3 family into the manuscript to enhance the robustness of the method and contribution to the field. 

We have highlighted the corresponding modifications in the manuscript with color blue for your convenience. Again, thank you for your hard work. We believe your input has already helped improve the paper and look forward to further engaging with you during the discussion. Please see our replies to each of you below.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel tuning-free approach for aligning language model outputs with human preferences during inference time. The key scientific contribution is a principle-guided reward function derived from a surrogate optimization solution that modifies model predictions to ensure adherence to target principles. Through comprehensive experiments, the authors demonstrate that their method achieves competitive or superior performance compared to existing baselines across both general and personalized alignment tasks. The main strengths are: 1) elimination of expensive fine-tuning/data collection requirements, 2) flexibility across various tasks, and 3) strong empirical results against RLHF/DPO baselines. Key weaknesses include: computational overhead during inference, lack of theoretical justification for superiority over in-context learning, and initial unclear principle design methodology. I would recommend acceptance primarily due to its novel technical approach and strong empirical validation, while addressing an important practical challenge in LLM alignment.