Discovering Factor Level Preferences to Improve Human-Model Alignment

Authors: Juhyun Oh, Eunsu Kim, Jiseon Kim, Wenda Xu, Inha Cha, William Yang Wang, Alice Oh
Primary area: interpretability and explainable AI
Keywords: human alignment, Large Language Model, explainability, generation, evaluation

Abstract:
Despite advancements in Large Language Model (LLM) alignment, understanding the reasons behind LLM preferences remains crucial for bridging the gap between desired and actual behavior. LLMs often exhibit biases or tendencies that diverge from human preferences, such as favoring certain writing styles or producing overly verbose outputs. However, current methods for evaluating preference alignment often lack explainability, relying on coarse-grained comparisons. To address this, we introduce PROFILE (PRObing Factors of InfLuence for Explainability), a novel framework that uncovers and quantifies the influence of specific factors driving preferences. PROFILE's factor level analysis explains the "why" behind human-model alignment and misalignment, offering insights into the direction of model improvement. We apply PROFILE to analyze human and LLM preferences across three tasks: summarization, helpful response generation, and document-based question-answering.  Our factor level analysis reveals a substantial discrepancy between human and LLM preferences in generation tasks, whereas LLMs show strong alignment with human preferences in evaluation tasks. We demonstrate how leveraging factor level insights, including addressing misaligned factors or exploiting the generation-evaluation gap, can improve alignment with human preferences. This work underscores the importance of explainable preference analysis and highlights PROFILE's potential to provide valuable training signals, driving further improvements in human-LLM alignment.

Decision: reject
Reviewer scores: 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730700668960

thread_type:
official_review

<thread_object>
creator: reviewer_vtjd

<content>
summary:
This paper introduces PROFILE, a framework designed to uncover and quantify the specific factors influencing both human and LLM preferences in language generation tasks. It addresses the problem of misalignment between LLM outputs and human preferences by providing a granular, factor-level analysis rather than relying on coarse-grained comparisons. The main contributions include the development of PROFILE, its application across three tasks (TLDR summarization, helpful response generation, and WebGPT document-based QA), and demonstrating how factor-level insights can improve human-LLM alignment

strengths:
● The paper presents an explainable framework that enhances understanding of human-model preference alignment at a granular level.
● It addresses a significant gap in current methods by focusing on specific factors influencing preferences, which can guide improvements in LLM training.
● Demonstrating that leveraging factor-level insights can improve alignment has practical implications for developing more human-aligned LLMs.

weaknesses:
● The paper might not thoroughly compare with existing methods, leaving questions about its relative advantages.
● The paper may lack sufficient empirical validation due to limited experiments or datasets, potentially affecting the generalizability of its conclusions. 
● There might be concerns about the scalability of the proposed framework without fine-grained human annotations, impacting its practicality.

questions:
1. How does your approach compare quantitatively and qualitatively with existing methods in preference alignment? Such as all kinds of llm-as-a-judge methods / G-Eval / ... etc

2. Can you provide more details on how the framework performs when applied to tasks beyond the three studied, and are there limitations to its generalizability? Such as creative writing, role-playing, and coding.

3.  The boundary between Receptiveness / Intent Align / Helpfulness is vague and not independent of each other.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718731923

thread_type:
official_review

<thread_object>
creator: reviewer_zmm8

<content>
summary:
This paper introduces a framework called PROFILE, designed for fine-grained factor analysis of LLM alignment. PROFILE reveals the discrepancies between human and AI preferences, pinpointing specific areas of divergence. By quantifying the influence of various factors, this framework provides new insights into the interpretability and improvement of models.

strengths:
- **Comprehensive Analysis from a Unique Perspective**: The paper conducts a detailed analysis of the underlying factors affecting human and AI preferences, providing a comprehensive view into the mechanisms of preference alignment.
- **Interpretability**: By performing factor-level preference analysis, PROFILE helps identify specific reasons for human-model preference divergence, offering clear directions for model optimization.

weaknesses:
- **Unclear Justification for Multi-level Factor Classification**: While the paper proposes a three-level factor classification system, it does not sufficiently explain the basis for each factor level, the scientific soundness of the classification, or its handling of task complexity. This raises concerns about whether the framework can accurately reflect human-model preference differences.
- **Potential Issues in the Analytical Approach**: The study uses GPT-4o for factor quantification while examining human-AI preference alignment, which may introduce new biases, potentially affecting the objectivity of the analysis.
- **Limitations in Experimental Design**: The paper validates the PROFILE framework on a limited set of public datasets, which restricts its demonstration of applicability to other tasks. Moreover, it lacks sufficient ablation studies to analyze the contribution of each factor, making it difficult to understand their impact across tasks.
- **Weak Correspondence between Results and Conclusions**: Although the experiments showcase preference alignment in some tasks, they lack clear methodological and empirical support for guiding improvements in human-AI alignment. The paper does not clarify how PROFILE contributes to enhancing model performance or its impact on generation quality in practical applications.
- **Over-reliance on Quantitative Metrics in Analysis**: The paper's analysis mostly depends on correlation and quantitative scores, lacking qualitative insights into why the model exhibits inconsistencies with human preferences for certain factors. This approach results in a somewhat superficial view that fails to reveal the deeper reasons behind the observed divergences.

questions:
1. I am curious about the correlations between different factors. Could you provide an analysis on this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730930961685

thread_type:
official_review

<thread_object>
creator: reviewer_gjul

<content>
summary:
This paper studies the preference discrepancy between human judgment and model judgment. Specifically, it investigates three RQs using a unified framework, PROFILE, to understand and enhance preference alignment at a fine-grained level (length, hallucination, etc). The authors conduct experiments on three datasets across two settings, and the results suggest misalignment in the generation setting. Their analysis highlights the potential of the proposed model to further improve alignment.

strengths:
The idea of `enhancing' alignment by better understanding the preference discrepancy between human and model judgment through more fine-grained factors is both interesting and important.


Their findings and insights could be valuable for researchers interested in human alignment, LLMs, and explainable AI.


I liked the overall organization of the paper, which consistent with the state of the field.

weaknesses:
The paper would benefit from more precise notation to improve clarity. I found the notation to be inconsistent and, at times, confusing, which impacts readability. For example:

- L134-135: The notation of score level s and Score(r) is confusing. If Score(r) already equals s, what is the purpose of having the model assign a score again?

The writing quality could also be improved. The core value of this paper lies in the exploration of the preference discrepancy between human and model judgment, yet the novelty and key ideas are not clearly articulated. Some concepts and terms are introduced without sufficient explanation, leading to confusion. For example:

- L40-41: “considering their alignment not only as generators but also as evaluators becomes crucial”

A better discussion and analysis are needed. Some findings and conclusions lack depth and specificity. For example:

- L471: "… engage in reward hacking by generating overly lengthy outputs…"  it is unclear how these conclusions were reached. The proposed method does not appear to involve RLHF/DPO training (correct me if I’m wrong).

The so-called "generalizability" conclusion seems to be derived solely from the summarization experiment, which may not provide sufficient support.”

questions:
What factors should we consider regarding preference discrepancies in other tasks, such as math and coding?

Aside from reward models, how do you think RLHF/DPO contributes to alignment?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732345431857

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We sincerely appreciate the meaningful and constructive feedback from all reviewers. The insights and questions provided by the reviewers were incredibly helpful in improving our paper during the review process. 

We are especially grateful to the reviewers for recognizing our framework's **explainability** (Reviewer vTJd) and **interpretability** (Reviewer ZMM8), as well as for finding it both **interesting and important** (Reviewer gJUL). Furthermore, we are pleased that our findings and insights are considered **valuable for researchers interested in human alignment** (Reviewer gJUL) and are seen as offering **practical implications for developing more human-aligned LLMs** (Reviewer vTJd). We sincerely hope that our contributions will help advance the alignment of LLMs with human preferences.

We have addressed the majority of the reviewers' concerns and would like to summarize our main contributions and the key points addressed as part of our general response.
## **Main Contribution**
- We present **PROFILE**, a framework for analyzing factor-level preferences in human-LLM alignment. Using PROFILE, it is possible to analyze **why** humans and models prefer certain responses **across various tasks** and identify **which factors** contribute to misalignment between humans and models.
- Using PROFILE, we applied our framework to analyze human-model alignment across three commonly used alignment tasks: summarization, helpful response generation, and document-based QA. Our analysis revealed that in all three tasks, models tend to prioritize "length" as a key factor in their generation output, whereas human preferences vary depending on the task.
- The explainable analyses enabled by PROFILE can facilitate **improvements in human-LLM alignment**. For example, our experiments illustrate that misalignments can be improved by providing direct instructions to models about the factors on which they are misaligned. Additionally, these insights could be used as training signals to improve alignment further.

## **Summary of the Discussion Phase and Draft Revision**
We have addressed the majority of the concerns raised by the reviewers and have revised the paper accordingly. These revisions have been highlighted in red for easy identification.

- **Clarification**: We acknowledge that certain parts of the paper, particularly **Section 2: Problem Definition** and **Section 3: Profile**, may have caused confusion, as raised by Reviewer gJUL. To address this, we have revised the ambiguous explanations and made significant adjustments to the writing in Sections 2 and 3 to enhance clarity and improve reader comprehension. 

- **New Analysis**:
In response to Reviewer ZMM8's inquiry about the **correlation between factors**, we have conducted and included a detailed analysis (Appendix G.3). We also included a **qualitative analysis** of the models' factor level preferences, including the examples (Table 5 in Appendix). To address Reviewer gJUL's concerns regarding the generalizability of one of our experiments, We have extended our evaluation by incorporating an **analysis of two additional tasks alongside summarization**, as presented in Table 10 of the Appendix.

Lastly, thank you to the reviewers for their invaluable feedback and thoughtful consideration.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper explores the differences between human and model judgments, focusing on preference discrepancies. It addresses three research questions using a unified framework, PROFILE, to examine and enhance preference alignment at a granular level, including aspects such as length and hallucination. The authors conduct experiments on three datasets across two settings, revealing misalignment in the generation setting. Their analysis underscores the potential of the proposed model to further improve alignment.

Pros:
1. The idea is both interesting and important. The paper is easy to follow.
2. The findings and insights is useful for researchers interested in human alignment, LLMs, and explainable AI.

Cons:
1. The presentation quality needs further improvement. Some notations are inconsistent and confusing, reducing the readability. The novelty and key ideas are not clearly articulated.
2. The findings and conclusions lack depth and specificity. A better discussion and analysis are needed.
3. Lack sufficient evaluation. The paper might not thoroughly compare with existing methods, leaving questions about its relative advantages. The paper's analysis mostly depends on correlation and quantitative scores, lacking qualitative insights into why the model exhibits inconsistencies with human preferences for certain factors. This approach results in a somewhat superficial view that fails to reveal the deeper reasons behind the observed divergences.
4. There are concerns about the scalability of the proposed framework without fine-grained human annotations, impacting its practicality.
5. Potential issues in the approach. The study uses GPT-4o for factor quantification while examining human-AI preference alignment, which may introduce new biases, potentially affecting the objectivity of the analysis.

I agree with most of the reviewers that this paper would benefit from another round of major revision.