Dissecting Human and LLM Preferences (2024)

Abstract: As a relative quality comparison of model responses, human and Large Language Model (LLM) preferences serve as common alignment goals in model fine-tuning and criteria in evaluation. Yet, these preferences merely reflect broad tendencies, resulting in less explainable and controllable models with potential safety risks. In this work, we dissect the preferences of human and 32 different LLMs to understand their quantitative composition, using annotations from real-world user-model conversations for a fine-grained, scenario-wise analysis. We find that humans are less sensitive to errors, favor responses that support their stances, and show clear dislike when models admit their limits. On the contrary, advanced LLMs like GPT-4-Turbo emphasize correctness, clarity, and harmlessness more. Additionally, LLMs of similar sizes tend to exhibit similar preferences, regardless of their training methods, and fine-tuning for alignment does not significantly alter the preferences of pretrained-only LLMs. Finally, we show that preference-based evaluation can be intentionally manipulated. In both training-free and training-based settings, aligning a model with the preferences of judges boosts scores, while injecting the least preferred properties lowers them. This results in notable score shifts: up to 0.59 on MT-Bench (1-10 scale) and 31.94 on AlpacaEval 2.0 (0100 scale), highlighting the significant impact of this strategic adaptation. We have made all resources of this project publicly available.123

heading: 1 Introduction
text: Human and LLM preferences have played a crucial role in the development pipeline of recent advanced
* Corresponding author 1 Interactive Demo: https://huggingface.co/spaces/ GAIR/Preference-Dissection-Visualization 2 Dataset: https://huggingface.co/datasets/GAIR/ preference-dissection 3 Github Repository: https://github.com/GAIR-NLP/ Preference-Dissection
language models. Preference-based training, such as Reinforcement Learning from Human/AI Feedback (RLHF/RLAIF) (Ouyang et al., 2022; Lee et al., 2023) and Direct Preference Optimization (DPO) (Rafailov et al., 2023), are widely used to fine-tune models to align with more practical needs. On the other hand, the preferences of human and LLMs, in the form of LLM-as-a-judge or Elo ratings, have become the de facto judging criteria for assessing the quality of model outputs (Li et al., 2023b; Zheng et al., 2023) as the tasks are becoming increasingly diverse and complex.
Unlike the widely applied preference-based methods above, the preferences themselves lack thorough research. In most cases, they are only binary labels indicating which response is preferred as a vague form of expression, and we are unable to understand the preferences in an explainable and quantitative way. As a result, optimizing models towards such goals inevitably leads to certain issues (Casper et al., 2023). These include the trained models engaging in over-optimization (Gao et al., 2023) and reward hacking (Sun et al., 2023), manifesting in undesired ways such as producing overly verbose answers (Singhal et al., 2023) or demonstrating sycophancy (Sharma et al., 2023), which hinder the building of more reliable AI systems.
In this work, we build a systematic framework to dissect the overall preferences into a quantitative combination of multiple clearly defined properties. To pursue understanding in realistic settings, we sample real-world user conversations with a balanced distribution of different scenarios from ChatBot Arena Conversations (Zheng et al., 2023), where each sample is a pair of model responses to a query. We adopt an elaborate yet automated pipeline to annotate the data with regard to our pre-defined properties (e.g., harmless or admit limits). Based on the annotations, we determine how a pair of responses differ from each other on all properties. These distinctions are then used to fit
1790
Bayesian logistic regression models, which help us quantitatively decompose preferences based on different properties by examining their weights.
Leveraging the above framework, we analyze the human preferences of real users and the preferences of numerous LLMs we collect. The analysis is conducted separately on different scenarios to avoid the mixing of preferences and achieve clearer conclusions (see Figure 1 for an example). We summarize the key findings as follows:
1. Humans are less sensitive to errors, clearly dislike a model when it admits its limits, and prefer a response that supports their stances (§4.1). 2. Advanced LLMs like GPT-4-Turbo prefer correctness, clarity, and harmlessness more (§4.1). 3. LLMs of similar sizes exhibit similar preferences irrespective of training methods, and the preference of a pretrained-only LLM is largely unchanged after alignment (§4.2).
Finally, we reveal that benchmarks with LLMas-a-judge are easy to manipulate (§4.3). Our experiments on AlpacaEval 2.0 (Li et al., 2023b) and MT-Bench (Zheng et al., 2023) show that aligning models with the judges’ preferences increases scores, whereas diverge from these preferences leads to lower scores. This is achievable across both training-free and training-based methods, with score variations up to 0.59 on MT-Bench (1-10 scale) and 31.94 on AlpacaEval 2.0 (0-100 scale). The manipulation highlights the urgent need for more robust benchmarks and further underscores the importance of understanding preference.

heading: 2 Related Work
text: The Application of Preferences In training, preference data through pairwise comparisons is used to build the reward model in Reinforcement Learning from Human/AI Feedback (RLHF/RLAIF) (Ouyang et al., 2022; Bai et al., 2022; Lee et al., 2023; Anthropic, 2023) or as the learning target in Direct Preference Optimization (DPO) (Rafailov et al., 2023; Tunstall et al., 2023; Ivison et al., 2023). In evaluation, comparing model outputs to references (Li et al., 2023b; Dubois et al., 2023; Zhou et al., 2023), directly rating individual responses (Zheng et al., 2023) or using Elo ratings from human votes (Zheng et al., 2023) have become common ways to assess the aligned models.
Challenges in Preference-based Methods The preferences of human and LLMs are greatly affected by features like the length of a response (Singhal et al., 2023), sycophancy (Sharma et al., 2023), or certain writing styles (Gudibande et al., 2023) due to limited annotation time (Chmielewski and Kucker, 2020), cognitive biases of annotators (Pandey et al., 2022), limited reasoning ability and self-enhancement bias (Zheng et al., 2023). Consequently, the reward model trained as the proxy of preferences is vulnerable to over-optimization (Gao et al., 2023) and reward hacking (Sun et al., 2023) in RLHF. Moreover, this can lead to instability in preference-based evaluation, reducing the credibility and reliability of the assessment results (Wang et al., 2023; Hosking et al., 2023).
Understanding and Demonstrating Preferences Perez et al. (2022) generate multiple-choice evaluations with LMs to discover model preferences across various personas. Turpin et al. (2023) and Wei et al. (2023) find that models may agree with incorrect answers if they are deliberately suggested in queries. Similar to our work, Sharma et al. (2023) and Hosking et al. (2023) conduct regression-based analyses to detect key factors in human preferences, such as sycophancy, assertiveness, and formatting. However, most of these works only offer basic analyses on limited synthetic data. In contrast, we thoroughly analyze human and LLM preferences in various real-world scenarios within a unified framework, and further explore potential applications arising from a deeper understanding of preferences.

heading: 3 Preference Dissection

heading: 3.1 High-level Methodology
text: In this work, we analyze preferences using pairwise comparison data, which has clearer and more consistent results than individual ratings (Ziegler et al., 2019). We start with a raw dataset 𝐷, where each sample contains a pair of responses (𝑟𝐴, 𝑟𝐵) to a query. Then, we collect the preference label 𝑙 ∈ {𝐴, 𝐵} from a judge 𝑗 (human or an LLM)4 to indicate the preferred one in each pair, forming the preference dataset 𝐷 𝑗 . A set of properties 𝑃 = {𝑝1, ..., 𝑝𝑁 } is defined to guide the analysis. Our objective is to decompose the overall preference in 𝐷 𝑗 into quantifiable contributions of each
4We treat preferences from different human annotators collectively as “human” and view each LLM as separate.
property in 𝑃: 𝐷 𝑗 ⇒ ⊕𝑁 𝑖=1 𝛼𝑖 ⊙ 𝑝𝑖. Here, ⊕ represents the composition concept, and 𝛼𝑖 ⊙ 𝑝𝑖 shows the contribution of property 𝑝𝑖 to overall preference with effect strength as 𝛼𝑖 .
To elaborate, as shown in Figure 2, we first annotate how each property in 𝑃 is satisfied in a response using a Likert scale rating, which is applied to both 𝑟𝐴 and 𝑟𝐵 in a sample. Then, we compare the ratings of two responses across these properties, resulting in a set of +1/0/-1 outcomes, indicating if 𝑟𝐴 is better, equal, or worse than 𝑟𝐵 per property.5 These outcomes create the “comparison feature” of a sample. Next, we learn a Bayesian logistic regression model to predict preference labels using the “comparison feature”, and the fitted weights represent the effect strengths for each property.

heading: 3.2 Dataset
text: To pursue analysis in realistic settings, we choose Chatbot Arena Conversations (Zheng et al., 2023) as the raw dataset 𝐷. It is collected from a public platform where users can freely converse with two models simultaneously and select a preferred one. We filter out samples with “Tie/Both Bad” labels and multi-turn conversations for future work to streamline annotation and analysis.
We notice that preferences of different scenarios vary a lot, so we take a scenario-balanced sampling. We first use the OpenAI moderation6 and toxicity tags in the dataset to identify 400 samples with unsafe queries. We then utilize the classifier in Li et al. (2023a) to categorize the rest samples into 10 scenarios: Exam Questions, Code, Creative Writing, Functional Writing, Communication, Knowledgeaware, Advice, Daily Tasks, NLP Tasks, and Others (see Table 8 in Appendix B for detailed descriptions). Since the Knowledge-Aware and Others scenarios have a notably higher proportion in the data, we randomly select 820 samples for each, while taking 400 ones from the other scenarios.

heading: 3.3 Collecting Preferences
text: Since human preferences already exist in the raw dataset 𝐷, we can directly get the preference dataset 𝐷ℎ𝑢𝑚𝑎𝑛, and only need to collect LLM preferences additionally. We select 2 proprietary LLMs: GPT-4-Turbo (gpt-4-1106-preview) and
5Directly annotating comparison outcomes is feasible, but early experiments indicate it yields an excessive number of slightly different properties, which hurts analysis reliability.
6https://platform.openai.com/docs/guides/ moderation
GPT-3.5-Turbo (gpt-3.5-turbo-1106), and 30 open-source LLMs (see Table 7 in Appendix C) to collect LLM preference datasets 32 × 𝐷𝐿𝐿𝑀 .
To minimize prompt bias in model preference assessment, we use a straightforward one: “Between Response A and Response B, which better addresses the user’s query? The better response is Response”, and measure preferences by the output log-probability of “A” or “B”. Acknowledging a positional bias in LLMs (Wang et al., 2023), where they prefer either the first or second response irrespective of content, we alternate response order and average log-probabilities for an accurate preference rating. This method yields a binary label with the same format as human preferences.

heading: 3.4 Pre-defined Properties and Annoatation
text: We design 29 properties for our analysis, referring to criteria from Li et al. (2023a); Sharma et al. (2023); Hosking et al. (2023). These properties, categorized into Basic, Query-Specific, and Error Detection groups, are listed in Table 1. For automated annotation, we employ GPT-4-Turbo to annotate a pair of responses simultaneously in one prompt to keep a consistent standard. We provide a fully annotated sample in Table 11 in Appendix D.
Basic Properties We define 21 basic properties, including stylistic ones like funny, and contentbased ones like admit limits. GPT-4-Turbo is required to rate responses from 0 to 3 for each property in a single prompt. We find that including the query in the prompt disturbs the annotation as most of the basic properties are query-independent. Thus, we only use the query for relevant and novel, the two query-aware ones. For lengthy, we directly measure word count using NLTK (Loper and Bird, 2002). Detailed annotation prompts and descriptions are in Figure 6, 7 and 8 in Appendix A.
Query-specific Properties We defined 5 queryspecific properties that are annotated based on the user query, e.g., support stances is inapplicable for queries with no subjective stance. Therefore to improve accuracy, we adopt a two-round annotation process. The first round determines if a query
meets the prerequisites for these properties, and we list the number of samples satisfying the prerequisites in Table 2. In the second round, annotation focuses only on applicable properties. The detailed prompts and the specific annotation goal for each property are illustrated in Figure 9, 10, 11 and 12 in Appendix A. The annotation results are also converted into a rating from 0 to 3 (see Appendix F).
Error Detection Although GPT-4-Turbo typically identifies errors in most samples accurately, it may fail with content beyond its training data. Therefore, we first ask it to evaluate whether it can reliably detect errors in a response, outputting an “applicable/not applicable” tag. Samples tagged as “not applicable” are excluded. For clearer annotation, we limit error types to four: Factual Error, Information Contradiction, Math Operation Error, and Code Generation Error. We also define three severity levels for errors: Minor, Moderate, and Severe, based on their impact on response correctness. Additionally, a reference answer generated independently by GPT-4-Turbo is included in the prompt, which has proven to help identify errors correctly (Zheng et al., 2023; Saunders et al., 2022; Sun et al., 2024). We ask GPT-4-Turbo to list errors by type and severity, creating 3 properties based on the error count per severity level. The complete prompt is shown in Figure 13 in Appendix A.
Annotation Quality Check We ask four of our authors to check the annotation quality. They are required to determine if they agree with the annotations, using the same guidelines for GPT-4-Turbo. The agreement rates are 93.1% for Basic property ratings, 85.1% for Detected Errors, with 90% of responses having all errors identified without missing. For Query-specific properties, the agreement is 94.8% in the first-round prerequisite questions
Table 3: The top/last 3 preferred properties of human and GPT-4-Turbo on different selected groups of samples, ranked by the degree of preference 𝑃(𝑝𝑖) = 𝜎(𝛼𝑖), the probability a response is preferred over the other when the former satisfies only one property 𝑝𝑖 better than the latter and all else equal.
ScenarioWise Advice Communication Creative Writing
Human
contain rich info lengthy well formatted harmless admit limits no minor errors
48 52 56 60 64 Top 3 Last 3 Human GPT-4-Turbo
lengthy friendly no severe errors polite correct mistakes admit limits
40
50
60
70
80 Top 3 Last 3 Human
GPT-4-Turbo
novel no severe errors satisfy constraints harmless interactive admit limits
48
56
64
72 80 Top 3 Last 3 Human GPT-4-Turbo
GPT-4Turbo
no severe errors clear contain rich info no minor errors harmless complex word & sent.
50
55
60
65 Top 3 Last 3 Human GPT-4-Turbo
no severe errors contain rich info no moderate errors step by step no minor errors persuasive
48
56
64
72
80 Top 3 Last 3 Human
GPT-4-Turbo
novel no severe errors satisfy constraints interactive admit limits informal expressions
48
56
64
72 80 Top 3 Last 3 Human GPT-4-Turbo
ScenarioWise Daily Tasks Knowledge-Aware Unsafe Query
Human
satisfy constraints novel no severe errors harmless admit limits no minor errors
48 54 60 66 72 Top 3 Last 3 Human GPT-4-Turbo
no severe errors lengthy relevant interactive admit limits harmless
50
60
70
80 Top 3 Last 3 Human GPT-4-Turbo
no severe errors relevant satisfy constraints polite harmless admit limits
30
45
60
75 Top 3 Last 3 Human GPT-4-Turbo
GPT-4Turbo
satisfy constraints no severe errors novel harmless show empathetic support material
48
54
60
66
72 Top 3 Last 3 Human GPT-4-Turbo
no severe errors clear no moderate errors harmless interactive show empathetic
50
60
70
80 Top 3 Last 3 Human GPT-4-Turbo
no severe errors correct mistakes authoritative step by step informal expressions no minor errors
48
56
64
72 80 Top 3 Last 3 Human GPT-4-Turbo
QuerySpecific Unclear Intent Express Subjective Stances Contain Mistakes/Bias
Human
relevant clarify intent no severe errors no moderate errors polite admit limits
40 50 60 70
Top 3 Last 3 Human GPT-4-Turbo
support stances contain rich info well formatted harmless no minor errors admit limits
45
50
55
60 65 Top 3 Last 3 Human GPT-4-Turbo
relevant no severe errors contain rich info no moderate errors admit limits harmless
40
48
56
64 72 Top 3 Last 3 Human GPT-4-Turbo
GPT-4Turbo
clarify intent no severe errors novel persuasive interactive admit limits
40 50 60 70
Top 3 Last 3 Human GPT-4-Turbo
no severe errors clear contain rich info rhetorical devices persuasive no minor errors
50 55 60 65 Top 3 Last 3 Human GPT-4-Turbo
correct mistakes no severe errors clear funny informal expressions no minor errors
48 54 60 66 72 Top 3 Last 3 Human GPT-4-Turbo
and 85.5% in the second-round annotation. More details can be found in Table 12 in Appendix G.

heading: 3.5 Calculating Effect Strengths of Properties
text: Building Comparison Features For each sample, we assign +1/-1/0 to each property based on whether the rating of 𝑟𝐴 is higher/lower or equal to that of 𝑟𝐵. For Error Detection properties, a lower count of errors in 𝑟𝐴 yields +1. The detailed comparison strategy is in Appendix H. These outcomes form a comparison feature, 𝜙 ∈ {+1, 0,−1}𝑁 , where 𝑁 = 29 is the total number of properties. Bayesian Logistic Regression We use Bayesian logistic regression to predict the preferences from comparison features (𝜎(·) is the sigmoid function):
𝑃(𝑙 = 𝐴|𝜙, 𝛼) = 𝜎 (∑𝑁
𝑖=1 𝛼𝑖𝜙𝑖 ) We place a prior 𝛼𝑖 ∼ Laplace(𝜇 = 0, 𝑏 = 0.1)
over the weights 𝛼𝑖 with zero mean and scale 𝑏 = 0.1. This prior encodes the belief each property is equally likely to increase or decrease the probability a response is preferred. We perform approximate Bayesian inference with the No-U-Turn Sampler (Hoffman et al., 2014) with Hamiltonian Monte Carlo (Neal et al., 2011) to collect 6,000 posterior samples across four independent Markov Chain Monte Carlo (MCMC) chains (each chain contains 500 warmup samples and 1,500 collected samples) , and take the mean value of all samples as the results in one fitting. To reduce instability in fitting, we divide the data into 10 parts, using 9 for
fitting in each iteration. The final weights 𝛼 are the average of the results from 10 iterations. To pursue a fine-grained analysis, we separately fit individual models for subsets with different scenarios (or meet certain Query-specific prerequisites) in each 𝐷 𝑗 . We see that the fitted models reach about 80% accuracy for most 𝐷 𝑗 (Table 10 in Appendix J).

heading: 4 Analysis and Application

heading: 4.1 Which Properties are Most or Least Preferred by Human and GPT-4-Turbo?
text: As the most common sources of preference data, we thereby analyze two specific cases: human and one of the most advanced LLMs, GPT-4-Turbo.

heading: 4.1.1 Quantifying Preferences for Properties
text: For a property 𝑝𝑖 , we calculate the degree of preference 𝑃(𝑝𝑖) as 𝜎(𝛼𝑖). This value corresponds to the probability that a response is preferred over another if it only satisfies 𝑝𝑖 better and all else equal. A higher value indicates a stronger preference, while a value less than 50% signifies dislike.

heading: 4.1.2 Results
text: We sort the properties by 𝑃(𝑝𝑖) for human and GPT-4-Turbo, and show the top/last 3 preferred ones on 9 selected groups of samples in Table 3, and the rest are shown in Table 13 in Appendix K.
Commonalities We first observe that under different scenarios or query-specific cases, the compositions of preferences vary greatly for both humans and GPT-4-Turbo. We also find human and GPT4-Turbo share some similarities like they have the same set of top 3 preferred properties for the Creative Writing and Daily Tasks, and in most cases, they both prefer responses with fewer severe errors and satisfy the explicit constraints in queries.
Disparities There are also many disagreements between them. We calculate the average degree of preference on the 3 Error Detection features across all scenarios in Table 4 and find humans are significantly less sensitive to severe errors, and do not show a clear preference/dislike to responses with
fewer moderate and minor errors. Besides, we see humans clearly dislike a model when it admits its limit in addressing the query, especially for Unsafe Query and Communication scenarios, indicating that human users in real settings have an urgent desire to have all their queries addressed even if they are unsafe. Humans also prefer responses that support their subjective stances (known as sycophancy), and pay little attention to how well a response corrects the mistakes or biases in queries.
On the contrary, GPT-4-Turbo emphasizes correctness, clarity, and harmlessness more. It tends to have much larger degrees of preference for properties it prefers than humans. It also likes the responses that help clarify the unclear intent in queries and correct the mistakes for queries that are unsafe or contain mistakes/biases, which highlight the Helpfulness, Harmlessness and Honesty (HHH) goals it has been aligned to.

heading: 4.2 How similar are the preferences of different LLMs?

heading: 4.2.1 Definition of Similarity
text: The similarity between preferences of two LLMs is defined as the average Pearson coefficient of the weights of their fitted Bayesian logistic regression models across all scenarios:
𝜌𝑠𝑀𝑁 = ∑𝑁 𝑖=1 (𝛼𝑠𝑀𝑖 − ¯𝛼𝑠𝑀 ) (𝛼𝑠𝑁𝑖 − ¯𝛼𝑠𝑁 )√︃∑𝑁
𝑖=1 (𝛼𝑠𝑀𝑖 − ¯𝛼𝑠𝑀 )2 √︃∑𝑁 𝑖=1 (𝛼𝑠𝑁𝑖 − ¯𝛼𝑠𝑁 )2
similarity(𝑀, 𝑁) = 1|𝑆 | ∑︁ 𝑠∈𝑆 𝜌𝑠𝑀𝑁
where 𝛼𝑠𝑀 denotes the weights of the fitted Bayesian logistic regression model for an LLM (or human) 𝑀 on scenario 𝑠, 𝑆 is the set of all scenarios, 𝑁 = 29 is the number of defined properties.

heading: 4.2.2 Results
text: We show the similarity in preferences between different LLMs (or human and an LLM) in Figure 3. We find that the preferences of human and GPT-4Turbo are significantly different from other LLMs, and most similar to Qwen-72B (-Chat). We also find LLaMA-2-7B is quite different from all other LLMs. This may be because it does not exhibit a consistent preference, preventing us from fitting a good model (only 63% accuracy, Table 10).
Implications of LLM Size on Preferences We divide the LLMs into two groups by their size: less than 14B (<14B) and larger than 30B (>30B). We calculate the intra- and inter-group similarities:
Intra(𝐴) = 1(𝑛 2 ) 𝑛−1∑︁
𝑖=1 𝑛∑︁ 𝑗=𝑖+1 similarity(𝑎𝑖 , 𝑎 𝑗 )
Inter(𝐴, 𝐵) = 1 𝑛𝑚 𝑛∑︁ 𝑖=1 𝑚∑︁ 𝑗=1 similarity(𝑎𝑖 , 𝑏 𝑗 )
where 𝐴, 𝐵 are groups of LLMs and 𝑎𝑖 , 𝑏𝑖 are the components. We find the intra-group similarities (0.83 for <14B and 0.88 for >30B) are much higher than the inter-group similarity (0.74). We also divide the models by the series mentioned in Section 3.3, and find the average intra- and inter-group similarities are very close (both 0.81, a complete version in Figure 5 in Appendix K). This further suggests that LLMs of similar sizes often have alike preferences, regardless of their training methods.
Effects of Alignment on Preferences Another question we care about is how the preference of an LLM changes after it is fine-tuned for alignment, which is calculated as the average similarity between a pretrained-only LLM and all its aligned variants. We also calculate the average log-probability difference between “A” and “B” as
the first output token in collecting preference labels. Results in Table 5 show that the preferences tend to remain largely unchanged after fine-tuning for alignment (except for the outlier LLaMA-2-7B), but the difference in log-probability increases a lot. This can be seen as a signal that alignment does not change the tendency of LLM preference, but greatly changes the intensity of expressing it.

heading: 4.3 Can Preference-based Evaluation be Intentionally Manipulated?
text: As an application of dissecting the preferences, we show that results on popular benchmarks with LLM-as-a-judge can be intentionally manipulated by adapting the responses of a model to more closely align with or deliberately diverge from the identified preferences of the judge.

heading: 4.3.1 Benchmarks
text: We take two benchmarks and use GPT-3.5-Turbo and GPT-4-Turbo as judges for both of them with their official evaluation prompts: (1) AlpacaEval 2.0 (Li et al., 2023b) has 805 queries where each of them has a reference response generated by GPT4-Turbo. The metric is the pairwise comparison win rate against the reference responses. (2) MTBench (Zheng et al., 2023) has 80 two-round conversations across various domains. The metric is the average response rating over all conversations. Since we focus on single-turn interaction, we only report the scores on the first rounds of all samples.

heading: 4.3.2 Model Adaptation to Judge Preferences
text: Training-free We utilize four models proficient in handling system messages (LLaMA-2-70B-Chat, Qwen-72B-Chat, GPT-3.5-Turbo, GPT-4-Turbo) and configure system messages to prompt them to adhere to the judge’s Top 3 or Last 3 preferred properties in the scenario of a query. In this setting, we also sample a subset of AlpacaEval 2.0 with 80 queries and require two authors as human judges.
Training-based We use the fitted Bayesian logistic regression models for our target judges to annotate the preference labels on our collected dataset. We exclude samples where the final preference probability is within 50 ± 15% to emphasize the preferences. We then use the remaining data (4,022 for GPT-3.5-Turbo and 3,991 for GPT-4-Turbo) to train 2 Alpaca models7 with DPO towards the preferences of judges (GPT-3.5-Turbo and GPT-4Turbo, respectively). For training against preferences, we simply invert labels from the regression models. We set batch size as 64, learning rate as 1e5 with a cosine scheduler, and train three epochs.

heading: 4.3.3 Results
text: The results are presented in Table 6. Generally, we find that in both training-free and training-based settings, adapting model responses to align with or diverge from judge preferences results in corresponding improvements or reductions in scores. Compared to GPT-4-Turbo, the effect of adaptation is more noticeable when targeting GPT-3.5-Turbo, possibly due to its less robust inferential abilities. While the effect is somewhat less pronounced with human judges, the manipulation remains effective, demonstrating its generality.
7https://huggingface.co/chavinlo/alpaca-7b (13b)
It is also essential to highlight that the prompts for LLM-as-a-judge in both benchmarks differ notably from those we employed for collecting preferences. These prompts included extra instructions, and MT-Bench even uses a single-response rating system. Despite the differences, our approach, which relies on pairwise comparisons to gather preferences, shows substantial and predictable effectiveness. This outcome underscores the robustness of our method in preference analysis, demonstrating its reliability even under varied conditions.

heading: 5 Conclusion
text: In this work, we conduct a thorough analysis to dissect how the preferences of human and LLMs can be quantitatively decomposed into different properties. We find that humans prefer responses that can directly address their queries but are less sensitive to errors in responses, while advanced LLMs like GPT-4-Turbo emphasize more on correctness, clarity, and harmlessness. We then find that model size can be a distinguishing factor in the preferences where LLMs of similar sizes share similar preferences, and fine-tuning for alignment does not bring about significant change to LLM preferences. As an application of our analysis, we also show the results of current benchmarks with LLM-as-a-judge can be intentionally manipulated, indicating the
vulnerability of preference-based evaluation. Last but not least, we publicly release all the collected resources to facilitate future research.

heading: 6 Acknowledgements
text: We thank Yuan Guo, Yiheng Xu, Yuqing Yang, Zhoujun Cheng, Zhihui Xie for their valuable feedback and suggestions on the writing of this paper. We also thank the anonymous reviewers for their valuable feedback and helpful suggestions. This project is supported by Qingyuan Research Project and Shanghai Artificial Intelligence Laboratory.

heading: 7 Limitation and Future Work
text: To obtain a binary preference label without positional bias, our method is not applicable to proprietary LLMs like Gemini (Team et al., 2023) or Claude (Anthropic, 2023), which do not return logprobabilities. Additionally, our use of Bayesian logistic regression for preference dissection assumes that all predefined properties independently influence the final preference, possibly missing complex interactions between them. Designing predefined properties and corresponding prompts for GPT-4Turbo is still challenging and laborious. A future improvement could be the automatic discovery of these properties, avoiding the trial and error in creating effective prompts for data annotation. This work focuses only on single-turn conversations, leaving the extension to more complex, multi-turn interactions for future research.

heading: 8 Ethics
text: This study primarily investigates the preferences of human and LLMs. While the analysis results are neutral, they could potentially be used to deliberately train and build models with potential risks and undesired behaviours. Thus, an important future task is to use these insights, based on dissecting and understanding preferences, to construct more reliable AI systems. The original dataset we use is licensed under CC-BY-NC-4.0, and we follow it to set the license or our annotated dataset as CC-BYNC-4.0. Besides, the original dataset we use have removed all conversations that contain personally identifiable information (PII).

heading: A Detailed Properties and Prompts for GPT-4-Turbo Annotation
text: The prompt for annotating the Basic properties is in Figure 6. The detailed descriptions for the basic properties are in Figure 7 and 8. The prompt for the first-round annotation, i.e. the prerequisite questions of the Query-specfic properties is in Figure 9. The prompt for the second-round annotation for query-specific properties is in Figure 10, and the detailed contents for the placeholders {questions str} and {output format str} in it are in Figure 11 and 12 respectively. The prompt for Error Detection is shown in Figure 13.

heading: B Defined Scenarios
text: We show how we merge the scenarios from the classifier in Li et al. (2023a) into our newly-defined 10 scenarios in Table 8.

heading: C Details of Selected Open-Source LLMs
text: We select 30 recent open-source LLMs from different series: LLaMA-2 (Touvron et al., 2023), Mistral (Jiang et al., 2023, 2024), Vicuna (Chiang et al., 2023), WizardLM (Xu et al., 2023), Tulu-2 (Ivison et al., 2023), Yi (01.AI, 2023), Zephyr (Tunstall et al., 2023), Qwen (Bai et al., 2023). The details of all selected open-source LLMs are listed in Table 7, including the base model they are trained on, and whether they are only pre-trained or aligned via Supervised Fine-tuning (SFT), RLHF, or DPO.

heading: D Annotation Example
text: We present an example in Table 11 to illustrate the contents of a fully annotated sample in our collected dataset.

heading: E More Statistics of Collected Dataset
text: We show more statistics of the collected dataset, including the average score/count for each property in Table 9 and how different properties correlate with each other in Figure 4.

heading: F Converting Annotations of Query-Specific Properties to Ratings
text: For clarify intent and show empathetic, we directly annotate a score of 0,1,2,3 to measure how much a response matches the property.
For satisfy constraints, we annotate 0,1,2,3 for each identified constraint on how each of them is
satisfied and take the average value over all constraints as the final rating.
For support stances, we use this mapping to convert the collected labels to scores: {strongly supported: 3, weakly supported: 2.25, neutral: 1.5, weakly opposed: 0.75, strongly opposed: 0}. The final rating is the average of these scores across all stances.
For correct mistakes, we use this mapping to convert the collected labels to scores: {pointed out and corrected: 3, corrected without being pointed out: 2, pointed out but not corrected: 1, neither pointed out nor corrected: 0}. The final rating is the average of these scores across all mistakes/biases.

heading: G Detailed Statistics of Annotation Quality Check by Human
text: We show the detailed statistics of the quality check of the 50 randomly selected annotated samples by human evaluators in Table 12, where we ask the evaluators to choose whether they agree with each annotation item or not in a sample. These annotation items include the ratings of Basic properties, the answers to the prerequisite questions and the optional second-round annotation result for QuerySpecific properties, and the detected errors for Error Detection.

heading: H Comparison Strategy
text: For most of the properties, we mark the comparison feature as +1/-1/0 if the rating of response A is larger/smaller than or equal to response B. For
properties of Error Detection of different severity (i.e. no severe/moderate/minor errors), we mark the comparison feature as +1/-1/0 if response A contains less/more/equal number of errors than response B. We also have two special cases: For relevant, we only mark a +1/-1 when one response has a rating of 0 and the other >0, otherwise we mark it as 0; for lengthy, we mark +1/-1 when the word counts of the two responses have a significant difference, i.e., the shorter one has fewer than 70% words of the longer one, otherwise we mark it as 0.

heading: I Details of Fitting Bayesian Logistic Regression
text: We perform approximate Bayesian inference with the No-U-Turn Sampler (Hoffman et al., 2014) with Hamiltonian Monte Carlo (Neal et al., 2011) implemented using numpyro (Phan et al., 2019), collecting 6,000 posterior samples across four independent Markov Chain Monte Carlo (MCMC) chains (each chain contains 500 warmup samples and 1,500 collected samples). The scale 𝑏 = 0.1 of the Laplace prior is determined using the remaining part as the validation set in each iteration of our 10-fold aggregation.

heading: J Accuracy of the Fitted Bayesian Logistic Models
text: We report the prediction accuracy of the fitted Bayesian Logistic models for a preference 𝐷 𝑗 :
acc(𝐷 𝑗) = 1|𝑆 | ∑︁ 𝑠∈𝑆 acc(𝐷𝑠𝑗)
, where 𝐷 𝑗 is the preference dataset of a judge 𝑗 , 𝐷𝑠𝑗 is the subset of it with scenario as 𝑠, and 𝑆 is the set of all scenarios. The results are in Table 10.

heading: K Additional Analysis Results
text: The top/last 3 preferred properties of human and GPT-4-Turbo on the rest 7 groups of samples (5 scenario-wise and 2 query-specific) that are not included in Section 4.1 are shown in Table 13.
We show the complete intra- and inter-group similarities when dividing LLMs by their series (mentioned in Section 4.2) in Figure 5.
Table 11: An example of a fully annotated sample in our collected dataset. For each property, we show the annotation results for of the two responses: first Response A, then Response B.
Query (Scenario: Daily Tasks)
give me an itinerary for a day in paris on june 11th saturday from 12pm to 10pm. no museums or other things that take too long. include lunch and dinner (vegan options preferred)
Response A (102 words), Preferred by User
12pm: Have lunch at a vegan restaurant in the Latin Quarter, such as Sage. 2pm: Take a stroll down the Rue Mouffetard, a bustling market street. 4pm: Visit Notre Dame Cathedral and take a look around the grounds. 6pm: Head to the Eiffel Tower and take in the views from the observation deck. 8pm: Enjoy dinner at a cozy Italian restaurant in the Marais, such as La Pizza Frites. 10pm: Wrap up the day with a stroll along the Seine, taking in the beautiful city lights.
Response B (346 words)
Sure, here is a suggested itinerary for a day in Paris on June 11th, 2021, that includes lunch and dinner options and keeps the activities relatively short: 12:00pm: Arrive at the Sainte-Chapelle, a beautiful Gothic chapel that was built in the 13th century. The chapel is famous for its stunning stained glass windows, which are considered some of the most beautiful in Europe. 1:00pm: Walk along the River Seine and take a Seine River cruise. This allows you to see the city from a different perspective and also visit some of the city’s most famous landmarks such as the Eiffel Tower, the Notre-Dame and the Île de la Cité. 2:30pm: Have lunch at Chez L’Ami Jean, a popular vegan bakery located in the Montmartre neighborhood. The bakery offers a variety of vegan pastries and sandwiches. 3:30pm: Visit the Arc de Triomphe, a famous landmark that was built in the early 19th century. The arc is located at the top of the Champs-Élysées, one of the most famous streets in Paris. 4:30pm: Walk to the Musée du quai Branly - Jacques Chirac, a museum that showcases the art and culture of Africa, Oceania, and the Americas. The museum’s stunning architecture and unique collections make for an interesting visit. 6:30pm: Have dinner at Le Comptoir du Relais, a vegan bakery located in the Montmartre neighborhood. The bakery offers a variety of vegan pastries and sandwiches. 7:30pm: Visit the Eiffel Tower, a famous landmark that was built in the late 19th century. The tower offers stunning views of the city and is a must-see for any first-time visitor to Paris. 9:30pm: End the evening with a casual dinner at Chez L’Ami Jean, a popular vegan bakery located in the Montmartre neighborhood. The bakery offers a variety of vegan pastries and sandwiches. Note: This itinerary is just
Annotations For Basic Features (ratings, higher value means satisfying the property better)
harmless 3 3 grammarly correct 3 2 friendly 2 2 polite 3 3 interactive 0 0 authoritative 2 2 funny 0 0 use rhetorical devices 0 0 complex word & sentence 1 1 use supporting materials 0 0 well formatted 2 2 admit limits 0 0 persuasive 0 0 step-by-step 0 0 use informal expressions 0 0 non-repetitive 3 2 clear 3 2 relevant 3 1 novel 1 1 contain rich information 2 3
Annotations for Query-Specific Features, Both Rounds.
Clear Intent Yes Explicitly express feeling No Mistakes or Bias None Explicit Constraints 5 Subjective Stances 2
5 Constraints 2 Stances Itinerary for a day in Paris on June 11th, Saturday: 3 3 Preference for a day without visiting museums or lengthy Time frame from 12pm to 10pm: 3 2 activities: Strongly supported Weakly opposed No museums or other activities that take too long: 3 1 Preference for vegan meal options: Include lunch and dinner: 3 3 Strongly supported Weakly supported Vegan options preferred for meals: 3 2
Annotations for Error Detection
Response A: Moderate (Information contradiction to the query) - La Pizza Frites is not known as a vegan restaurant, which contradicts the user’s preference for vegan dinner options.
Response B: Severe (Factual error) - Le Comptoir du Relais is not a vegan bakery, and it is incorrectly listed as such. Moderate (Information contradiction to the query) - Sainte-Chapelle visit contradicts the ’no museums or other things that take too long’ request. Moderate (Information contradiction to the query) - Chez L’Ami Jean is not a vegan bakery, and it is listed for both lunch and dinner, which contradicts the vegan preference. Moderate (Information contradiction to the query) - Musée du quai Branly - Jacques Chirac visit contradicts the ’no museums or other things that take too long’ request. Moderate (Information contradiction to the query) - The itinerary suggests ending the evening with a casual dinner at Chez L’Ami Jean after already having dinner there at 6:30pm. Minor (Information contradiction to the query) - The date June 11th, 2021, is incorrect as the query asks for an itinerary for June 11th without specifying a year.

heading: Future Work
text: Thus, an important future task is to use these insights, based on dissecting and understanding preferences, to construct more reliable AI systems. The original dataset we use is licensed under CC-BY-NC-4. 0, and we follow it to set the license or our annotated dataset as CC-BYNC-4. 0. Besides, the original dataset we use have removed all conversations that contain personally identifiable information (PII)