Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
Authors: Siyan Zhao
Mingyi Hong
Yang Liu
Devamanyu Hazarika
Kaixiang Lin
Venue: ICLR 2025 Oral (2025)
Keywords: personalization
benchmark
Large language models
conversational llm
chatbots
Primary area: datasets and benchmarks

Abstract: Large Language Models (LLMs) are increasingly deployed as chatbots, yet their ability to personalize responses to user preferences remains limited. We introduce PrefEval, a benchmark for evaluating LLMs' ability to infer, memorize and adhere to user preferences in long-context conversational setting.
PrefEval comprises 3,000 manually curated user preference and query pairs spanning 20 topics. PrefEval contains user personalization or preference information in both explicit and implicit preference forms, and evaluates LLM performance using a generation and a classification task. With PrefEval, we have evaluated 10 open-sourced and
proprietary LLMs in multi-session conversations with varying context lengths up to 100k tokens. We benchmark with various prompting, iterative feedback, and retrieval-augmented generation methods. 
Our benchmarking effort reveals that state-of-the-art LLMs face significant challenges in following users' preference during conversations. In particular,  in zero-shot settings, preference following accuracy falls below 10\% at merely 10 turns (~3k tokens) across most evaluated models. Even with advanced prompting and retrieval methods, preference following still deteriorates in long-context conversations. Furthermore, we show that fine-tuning on PrefEval significantly improves performance. We believe PrefEval serves as a valuable resource for measuring, understanding, and enhancing LLMs' proactive preference following abilities, paving the way for personalized conversational agents.

Published as a conference paper at ICLR 2025
Do LLMS RECOGNIZE YOUR PREFERENCES? EVAL- UATING PERSONALIZED PREFERENCE FOLLOWING IN LLMs
Siyan Zhao"*, Mingyi Hong'.>, Yang Liu', Devamanyu Hazarika', Kaixiang Lin' Amazon AGI, UULA, University of Minnesota
siyanzØcs www.allaaaaoo edu,, mhong@umn. edu, devamanyugu. nus edu {yangliud, kaixian] ammazon. com
ABSTRACT
Large Language Models (LLMs) are increasingly used as chatbots, yet their abil- ity to personalize responses to user preferences remains limited. We introduce PREFEVAL, benchmark for evaluating LLMs' ability to infer, memorize and adhere to user preferences in a long-context conversational setting. PREFEVAL comprises 3,000 manually curated user preference and query pairs spanning 20 topics. PREFEVAL contains user personalization or preference information both explicit and implicit forms, and evaluates LLM performance using gen- eration and classification task. With PREFEVAL, we evaluated the aforemen- tioned preference following capabilities of 10 open-source and proprietary LLMs in multi-session conversations with varying context lengths up to 100k tokens. We benchmark with various prompting, iterative feedback, and retrieval- retrieval-aummented generation methods. Our benchmarking effort reveals that state-of of-the-art LLMs face significant challenges in proactively following users' preferences during con- versations. In particular, in zero-shot settings, preference following accuracy falls below 10% at merely 10 turns (~33 tokens) across most evaluated models. Even with advanced prompting and retrieval methods, preference following still dete- riorates in long-context conversations. Furthermore, we show that fine-tuning on PREFEVAL significantly improves performance. We believe PREFEVAL serves a valuable resource for measuring, understanding, and enhancing LLMs' prefer- ence following abilities, paving the way for personalized conversational agents. Our code and dataset are available at https: /ttp//preeeeeee g tit.....tt..... io/.
INTRODUCTION
The pursuit of personal chatbots that can remember your favorite cuisine, propose binge-worthy se- ries tailored to your tastes, and avoid suggestions that conflict with your dietary restrictions has been a longstanding desire for many. While LLMM-bseed chatbots such as Claude (Bai et al., 2022) and GPT-4 (Achiam et al., 2023) have substantially advanced natural language processing capabilities, their ability to proactively provide personalized interactions that are scalable to millions of users remains limited (Salemi et al., 2023; Li et al., 2023a; Jang et al., 2023; Tan et al., 2024; Liu et al., 2024a; Zhuang et al., 2024; Li al., 2024a; Shaikh et al., 2024; Lee al., 2024). For example, if user says, "I don't 't like jazz." and later asks for travel recommendations alized chatbot should avoid suggesting jazz-related attractions, which are popular there. Achieving this level of proactive personalization poses a challenge when scaled to millions of users with diverse real-life preferences. Rather than building separate models for each, it's more scalable to create single adaptable chatbot that can dynamically understand and accommodate these preferences in real-time. This brings us to the central evaluation goal of our benchmark:
New Orleans, a person-
Can LLMS infer, remember and follow personalized preferences?
This ability is crucial for user satisfaction and engagement during conversations. Current LLMs, however, are primarily optimized and evaluated for general-purpose tasks. Our study will reveal that we lack comprehensive understanding of their ability to proactively follow and apply user
Work done while as an intern at Amazon.
1 Corresponds siyanz&cs ucl edu, kaixi @amazon.com

---

Published as a conference paper at ICLR 2025
-ulsi-Session
Preferences and Queries
Tasks
Conversation Setup
User Preference Forms
Generation:
Classification:
Multi-Session
Explicit Preference didlik eating
Sugpestions, Advice,
Cuolie--Chlice
Conversations
allergy"
Evaluation Protocols
Conversation Flow
Implicit Preference
User Preference
Assistant:
Hore are a few dinner options to consider:
Preference
User Query
Response
Vegetable sti-try stroganoit noodles.
Multi-Session Conversations
3. A seafood plaster with shnmp and crab.
4. Grilled chicken Caeser salad.
Which one you
Evaluator
...
Useri
Optin ls too seafapd-heaw for my liking.
LLM-based
MCQ
Query related to the preference
but othe work.
Error Types
Query related the preference
(1) Hallucination; (2) Unhelpful;
[3) Incorsistency: (4) Preference-
Classiication
LLM Response
"What are zome Japanese casioes dishes
Unswware Vibletion.
Accuracy
Figure I: PREFEVAL setup overview. Key components from left to right: 1) Multi-Session Conver- sation Setup: PREFEVAL evaluates LLMs' ability to follow user preferences in multi-session con- versation, challenging LLMs to handle preference inference, long-range retrieval, and context-aware preference following simultaneously. 2) Preferences and Queries: User preferences can be ex- pressed through both explicit and implicit forms. Queries are designed such that non-personalized answer would inadvertently conflict with user preferences, testing the LLM's adherence. 3) Tasks and Evaluations: PREFEVAL includes generation and classification tasks. Generation tasks are evaluated using an LLLM-based evaluator to measure preference following accuracy and analyze er- types. Classification tasks enable quicker evaluation through multiple-choice questions (MCQ). The two tasks' performances are highly correlated as shown in Fig 11.
preferences over conversations. Addressing this gap is essential for advancing LLMSs toward truly personalized and scalable conversational agents.
Effective personalization in conversation settings also requires robust long-context abilities, as it re- quires aggregating and adhering to user preferences over extended interaction histories. personal assistant should proactively infer, memorize, and adhere to these preferences over long horizon, en- suring responses are not only relevant but also aligned with the user's preferences. Additionally, express preferences in various forms-explicitly explicitly implicitly-making it challenging for the assistant to recognize them accurately. Users may also present multiple or conflicting preferences within single conversation. The assistant must navigate these nuances for delivering truly per- sonalized experience.
In response these challenges, introduce PREFEVAL, benchmark to evaluate, understand, and improve LLMs' capacity for preference following in conversation settings. Our benchmark con- sists of 3,000 manually curated prefereece-query pairs across diverse daily life topies, incorporating preference forms which are explicitly stated or implicitly revealed, and it includes both generation and classification tasks. With PREFEVAL, we assess 10 open-source and proprietary LLMs with varying context lengths of up to 100k tokens using various methods, analyzing their adaptability to conflicting and multiple user preferences, and demonstrate how finetuning on this dataset enhances performance. Our contributions can be summarized as follows:
We introduce a novel, comprehensive benchmark for evaluating LLMs' preference following ca- pabilities in conversational contexts, encompassing 3,000 manually curated question-preference pairs across 20 topics and preference forms.
We conduct an extensive evaluation of state-of-the-art LLMs, including Claude, Mistral, GPT4 and the LLaMA series, utilizing various context lengths and assessment methods such as prompt- ing, iterative feedback (Bai et al., 2022), and aatrieval-augment
generation (Lewis et al., 2020).
Our benchmark results show that, without explicit prompting, the preference following precision (evaluated using various measures we developed in this work) falls below 10% in zero-shot settings for 10-tur conversations of 3k tokens. Even with more advanced methods, performance still deteriorates with longer contexts.
We uncover critical limitations in current LLMs through extensive error analysis, including their inability recognize and proactively apply user preferences in long-context conversation settings.
2

---

Published as a conference paper at ICLR 2025
We find that, counterintuitively, multiple stated preferences within conversation lead to improved adherence, even in the presence conflicting preferences. We conjecture it is due to reinforced attention user preferences. Moreover, fine-tuning on PREFEVAL further enhances preference following and generalizes well to longer contexts.
2
THE PREFEVAL DATASET
2.1
PROBLEM FORMULATION
The goal of our benchmark is to assess how well LLMs can personalize their responses to user preferences in conversational settings, with the overall setup shown in Figure 1. Let C denote conversation comprising m turns {(u1, bi), (u2, b2)...., "0)...,(uun) (um, bm)), where u, and b, represent the user's message and LLM's response at turn respectively. Each conversation C is divided into sessions S = {81, ......, ) where each session represents a coherent dialogue segment focused on a specific topic and consists of contiguous turns {[u, ((u,,b)).... b.)..... (u++I, b(+1)) C.
At the core of our evaluation are preference-quers pairs (p, q). Here, refers to a user preference disclosed some point in the conversation, while g represents a query related to this preference, posed at the end of the conversation for the LLM to respond. The user preference can be ex- plicitly expressed as a single utterance or implicitly revealed through multi-turn dialogue. The query is the user's message at turn m., i.e., q tmm and the LLM's response is bm. The query q is constructed such that a generic, non-personnlized response would likely violate the previously stated preference p. We evaluate whether 6m adheres to the preference p. To simulate real-world conversational complexity, we include unrelated contextual turns between and q. These interven- ing turns act potential distractions, emulating the natural flow of dialogue, where multiple topics may be discussed across a single conversation. In such cases, maintaining awareness of the user's earlier preference p becomes crucial for appropriately responding the later query q. We evaluate the LLM's ability to navigate this complexity and maintain personalization, measuring how well it responds to q line with the user's preference p, even amidst unrelated sessions.
To perform well on our benchmark, LLMs should demonstrate four key capabilities: (1) Preference Inference-the capacity accurately infer user preferences through dialogue, whether explicitly stated or implicitly revealed; (2) Long-Context Retrieval-the ability to track and recall user pref- erences across long conversation; (3) Preference Following- the ability to generate responses that are both contextually relevant and aligned with the user's s preferences when knowing the preference; and (4) Personalization Proactiveness the initiative and ability to know when and how to utilize the first three capabilities to deliver personalized responses, rather than focusing solely general question answering. We will show our benchmarking results that, LLMs that miss a subset of these capabilities perform poorly.
2.2 PREFEVAL STATISTICS
PREFEVAL. consists of 1, 1,000 unique preference--uery pairs, each with three preference forms ($2.3), resulting in 3000 preference-query pairs. These pairs were manually curated with the assistance of GPT-4, Claude Sonnet, and Claude 3.5 Sonnet (see Appendix A. for detailed data construction methodology). The preferences cover day-to-day topics such travel, shopping, entertainment, and more, as shown in the topic distribution in Figure 2. We intersperse unrelated contextual conver- sation turns between the disclosure of extending up to 100k tokens ($2.4). For each pair (p, q). we consider two tasks: a generation task and classification task. In the generation task, the LLM is required to generate long-form sponse to the user's query. In the classification task, the L.LM is presented with four options related the query, with one option aligned with the user's preference, and the LLM is tasked select the correct option.
preference and the final query, with context lengths
2.3 PREFERENCE FORMS
User preferences can be expressed in various forms. In our benchmark, we consider three distinct methods of preference construction and expression: (1) Explicit Preference. The user directly expresses their preference to the LLM in a single sentence within a single conversational turn (Ex- amples in Table 14). (2) Implicit Choice-Based Dialogue. The user's preference inferred over the course of two-turn dialogue. In this setup, the initiates preference-related query, and the
3

---

Published as a conference paper at ICLR 2025
The PrefEval Benchmark
Entertainment
Travel
Lifestyle
Shopping
Education
Sports 5. 5.2%
Activities 5.8%
Hotel 5.4%
Dietary 5.7%
Fitness 5.2%
Motors 4 4.566
Resouroes
6.2%
5.6%
Music &
Books 5.6%
Games 5.1%
Restauren 5.0%
Transport 4.5%
Beauty 5.3%
Heelth 4.9%
Fashion 5.496
Figure 2: Distribution of domains and topics within PREFEVAL, which are commonly encountered during conversations with chatbots where users seek recommendations, suggestions, and advice.
assistant presents multiple options, some of which violate the user's preference while others align with it. The user can either agree with or reject one or more options, implicitly revealing their pref- erence through these choices (Examples in Table 15). (3) Implicit Persona-Driver Dialogue. To simulate more nuanced, implicit preferences, this form of elicitation unfolds over 4-8 turns. The di- alogue primarily revolves around persona-driven topic, with randomly assigned persona guiding the conversation. The user's preference is subtly revealed in a single sentence during the dialogue, while is not the main focus of the conversation, making the inference and reasoning process more challenging (Examples in Table 16).
2.4 MULTI-SESSION CONVERSATIONAL CONTEXT
To simulate realistic conversational dynamics, incorporate multi- multi-session turns from the LMSYS- Chat-IM dataset (Zheng et al., 2023), consisting of one million real interactions between users and 25 state-of-the-art language models across various topics. We randomly select multi-session context up to a length of 100k tokens and intersperse these conversation turns between the disclosure of a user preference and the final query. This presents challenge for the LLM to accurately infer, retain, and retrieve user preferences while navigating unrelated dialogues, assessing the LLM's ability to handle long-context preference following.
2.5 TASK TYPES AND EVALUATION PROTOCOLS
We offer two task types for each preference-query pair, each with its corresponding evaluation pro- tocol. By including both generation and classification tasks, we aim to thoroughly assess LLMs' ability to understand and adhere to user preferences in diverse contexts.
Generation Task and LLM-based Evaluators. In the generation task, the LLM generates a sponse in reply to the user's query. To evaluate preference following in the generation task, we apply the "LLM-as-a-judee juddge" framework using Claude 3 Sonnet. Specifically, evaluators to check the response against four binary metrics. Each evaluator is provided with a detailed prompt (see Appendix Sec A.12) containing the definitions and examples for each metric. These checks are then aggregated into four distinct error types, following the rules outlined in Ta- ble 1. Preference-following accuracy is defined the absence of any error type in the generated response, The four error types are: (1) Preference- Unawre Violation: The LLM provides generic recommendations that contradict the user's stated preference due unawareness of user preference. (2) Preference Hallucination Violation: The response fabricates or misattributes preferences, di- verging from the user's true preference and violates the true preference. (3) Inconsistent Violation: The response acknowledges the correct preference but generates contradicting response. (4) Un- helpful Response: The response lacks relevant recommendations or fails to address the query due to poor recall of the user's preference. To validate LLM-based evaluation method, checked 200 randomly sampled evaluations, with an observed 5% error rate. This demonstrates strong agreement between human judgment and LLM-based assessments with Claude 3 Sonnet.
employ four independent
manually
Classification Task and MCQ Accuracy. the classification task, the user presents a final query along with four potential options and asks the LLM to select the one that aligns with their preference. Only one option follows the user's 's stated preference, while the remaining options conflict with it. The LLM's preference-for following accuracy is determined by whether it selects the correct option. This task facilitates faster automatic evaluation, eliminating the need for costly human or LLM- based assessments by focusing single-choice response.
4

---

Published as a conference paper at ICLR 2025
Table 1: Error type aggregation rules: Evaluators perform binary checks for each metric listed in the column headers. The results are then aggregated according to the table rules to classify the error into one of the four defined types.
Type
Violate
Acknowledge Preference?
Hallielnatee Preference?
Helpful
Preference?
Response??
Preference- Caaeee-UUunn Unaware Violation
Yes
No
N/A
Yes
Preference Hallucination Violation
Yes
Yes
Yes
Yes
Inconsitenncy Violation
Yes
Yes
Yes
Unbelpful Response
No
Yes/No
N/A
No
3 EXPERIMENTS
Our experiments encompassed comprchensive evaluation and analysis of both open-source and proprietary state-of-the-art LLMs across multiple dimensions, including preference forms, task types, various methods to improve preference following, examination of how LLMs adapt to multiple and dynamically changing preferences, and fine-tuning open-source models to enhance preference following. We aim to investigate the following questions: (1) What is the performance of current SOTA I.MS multi-session long-context preference following with zero-shot, prompting and RAG baselines? ($3.2. $3.3, $3.4). (2) How do LLMs performance change with different forms of user preference expressions? ($3.3). (3) What causes LLMs to fail in preference following, and what are the prevalent error types? ($3.5). (4) How capable are LLMs in simultancously accom- modating multiple user preferences and adapting to dynamically changing user preferences? ($3.6). (5) How does finetuning on PREFEVAL improve the preference adherence ability of LLMs? ($3.7)
3.1 MODELS AND METHOOS
We extensively evaluate a variety of state-of-the-art LLMss, including Claude Sonnet, Claude Haiku, Mistral 7b Instruct, Mistral 8x7b Instruct, LLaMA 3 8b Instruct, and LLaMA 3 70b Instruct. We also assess more recent models Claude 3.5 Sonnet, GPT-ol -pre-vi-,e and Gemini- 1.5-pro in spe- cific settings. We investigate methods to explicitly help LLMs focus the preference-followingg task, with the aim of understanding they can proactively adapt to user preferences: (1) Zero-shot: The default case, where the LLM directly answers the question without any additional prompting. (2) Reminder: Before answering the question, the LLM is provided with a reminder sentence to consider the user's previously stated preference in answering. (3) Self-Critic: The LLM generates an initial zero-shol response to the question, critiques whether it has followed the user's prefer- ence, and then generates revised response taking the critique into account, similar to intrinsic self-correction (Huang et al.). (4) Few-Shot Chain-of- Chain-of-Thought (CoT): The LLM is given few- shot examples of CoT reasoning of how to follow the user's preference right before answering the Generation (RAG): A sentence embedding model is used retrieve the most similar conversation exchanges to the question, which are then provided the LLM in the prompt. Please refer to $A.4 for detailed prompt examples for each method. For the experiments, if not explicitly stated otherwise, we place the user's preference at the beginning the conversation and query at the end.
question. (5) Detrieval-Augemennt
3.2 EXPLICIT PREFERENCE FOLLOWING
SOTA LLMs exhibit limited proactivity in adhering to user preferences. As shown in the gen- eration task performance Figure 3, all LLMs exhibit substantial declines in preference-following accuracy as the dialogue length increases between the user's 's stated preferences and the final queries in zero-shot settings where no specific prompting provided. Accuracy drops steeply from ap- proximately 80% to below 30% as the number of conversation turns increases to merely 5. As the turns extend from 30 to 300, accuracy falls close to zero across all models. This suggests that LLMs often provide recommendations that conflict with the user's previously stated preferences, even when expressed only a few turns earlier. Such behavior is detrimental to user satisfaction and engagement. Additionally, more advanced LLMs continue to struggle with preference following in zero-shot settings, even within the first 10 turns. As shown in Table 2, GPT-ol has 50% accuracy while Claude 3.5 Sonnet and Gemini 1.5 Pro have near-zero preference following.
5

---

Published as a conference paper at ICLR 2025
Preference-folloviae accuracy across SOTA LLMs on the travel restaurant generation task. Accuracy is evaluated using two context lengths between the user's prefer-
Table 2:
tokens
Zero-shot
Reminder
Zero-shot
Reminder
Claude-1.5 Sonnet
0.07
0.45
0.02
0.02
Gemini-1.S
0.07
0.91
0.09
D,5
GPT-4o
0.07
0.98
0.23
ence and query. We show results for Zero-shot and Reminder (best prompting method). The
GPT-4.1
0.50
0.98
0.70
ol-preview
0,50
0.58
0.14
0,98
o-series models may involve additional test-
of-mini
0.23
0.98
0.02
0.54
0.71
1.0
0.05
0.43
time compute due to their "thinking" phase.
Prefarence Following Accuracy
Zero-Shot Performance with Explicit Preferences
Figure 3: Zero-shot performance of LLMs with explicit preferences, aver- aged across 20 topics. The x-axis rep- resents the dialogue length between the user's stated preference and the fi- nal query. measured by both the num- ber of tokens in the prompt and the number of conversation turns. All LLMs exhibit a rapid decline in accu- racy as the number of turns increases.
Claude 3 Sonnet
Claude Haiku
0.6
Llama3 88 Instruct
Llama3 70B Instruct
0.4
Mistral 7b
Mistral 8x7b
0.2
54k
84k
103k
110
150
250
300
How do different methods improve preference following performance? LLMs lack the proac- tiveness to follow user preference in the zero-shor setting. We evaluated four methods for enhancing preference following across models ($3.1) l that force the LLMs to utilize their abilities in Preference Inference, Long-Context Retrieval and Preference Following. As shown in Figure 4, all methods outperformed the zero-shot baseline. Among prompting techniques, the Reminder method, which simply reminds the model to consider the user's previously stated preferences (see prompt in $A.4),, surprisingly outperformed more complex methods such as Self-Critic and COT. Interestingly, per- formance trends varied across models: in Claude models, Self-Critic initially outperformed CoT but fell behind as the number of conversation turns increased, whereas the reverse was observed for Llama models. RAG consistently performed the best across most models, indicating that adherence to user preferences may hinge on retrieval capabilities. However, in models like Claude Sonnet and Mistral 8x7b, Reminder performed comparably or even surpassed RAG, suggesting that these models may have strong intrinsic Long- Context Retrieval abilitics.
Zero-Shot
Reminder
Self-Critic
CoT
RAG (Top 5)
Claude Sonnet
Claude Haiku
Mistral 8x7b
Mistral 7b
Liama3 70B Instruct
Liama3 BB Instruct
Lo.
1.0
1.
Accuracy
0.8
0.8
0.6
0.5
0.6
0.2
0.2
0.-
0.2
0.2
3.0
103k
0.5
35x
71k
0.0
0.
16c 23k
0.0
70
Figure 4: Performance comparison of 5 methods across 6 LLMs with explicit preferences on the generation task. Both Reminder and RAG consistently achieve the highest accuracy across models. Notably, Reminder outperforms more complex techniques such as Self-Critic and COT.
3.3 IMPLICIT PREFERENCE FOLLOWING
Implicit Preferences Add Complexity to Preference Inference. The ability to infer preferences especially critical when users implicitly reveal their preferences through a dialogue. PrefEVal. includes three forms of preferences, two of which are implicitly conveyed through conversation ($2.3). We evaluate how these different forms impact model performance using the best prompt-
Note that GPT-ol-previews may not be fair comparison with other models as it may require more test-time compute with "thinking" phase.

---

Published as a conference paper at ICLR 2025
Explicit Preference Zero-Shot
Implicit Choice-Bas
Implicit Persona-Driver
Reminder
Zero-Shot
Reminder
Zero-Shot Pro-Shot
Reminder
Input Prompt Token Length
77
62
99
50
95
32
d3
JK
99
1k
10
12
61
16
5
26
10
9
36
15
3K
99
3k
8
8
36
7
6
10
93
57
10k
23k
8
59
23k
30
30k
54K
84k
84k
6
103k
193k
103K
29
Clouds Sentnof
Ciudle Holes Histol To
Liamed 708 iecomet Ulanet uB hurrect Cleuds Sentoot Clusde Holes Mistrll So Nostrll TO
Mittrll To
Liamd 708 to secteee taamas oB tarrect
cuadaa Soynee Crasder Haloo
(edtrrl Bu70 aesoral TO Liaas 708 instrrcc
Llamaa Clustle 3 Sonnees
88 restraa
Craudie Halao
Mattrn 5.770 Miyrral To Liarai TO8 inetrrct Ulamaa 8B instrott
Clouds Sormet Caniie Holko Hottrll 70 Meytcel To
Liare's 780 borttcc
Unm''s to setiric
Clouds Surment
Cansle, Halko
Hotrel 70 pestzel 70 Llane's purtect Llemas to instrict
Figure 5: Comparison of 3 preference forms for 6 LLMS on the generation task, across varying lengths between the stated preference and query. Note that the user preference stated in the first turn. Results show that implicit preferences are more challenging to infer than explicit preferences.
As shown in Figure 5, implicit preferences add additional complexity to sreference-following tasks, even with shorter input lengths. Models exhibit varying capabilities in handling these complexities: for instance, Claude and Llama struggle more with implicit persona- driven preferences compared to implicit choice-based preferences, while Mistral models show the opposite trend. These differences indicate that each model has unique strengths and weaknesses in inferring and processing distinct preference forms.
ing
3.4 CLASSIFICATION TASK
Classification Task Brings Fast Evaluation with High Correlation to Generation Task Results. We introduce a classification task for each preference-query pair to complement the generation task ($2.5). This reduces reliance on costly human evaluations and LLM-based evaluators. We task LLMs to select the aligned option based on user preferences and get classification accuracy. Results in Figure 6 show higher overall accuracy compared to the generation task, reflecting the classifica- tion task's 's simpler nature. The Self-Critic method, with Claude models and Mistral-8x underper- forms the zero-shot baseline. Upon examining their critiques and revisions, we observe that when models fail to retrieve user preferences, the revisions tend to hallucinate and select random options distinct from the initial response. We conjecture that iterative self-feedback mechanisms may be less effective for classification tasks (which require structured output) compared to generation tasks. While both RAG and Reminder consistently achieve high accuracy, RAG demonstrates more advan- tage over Reminder in the classification task compared to the generation task. To further understand the relationship between the generation and classification tasks, we perform a correlation analysis preference alignment
between LLM-based preference following accuracy and classification- results, as illustrated in Figure 11. Across six models, five methods, and 12 dialogue lengths be- tween the preference and query, each data point in the scatter plot is averaged over 20 topics. We observe a correlation coefficient of 0.73, indicating a strong positive correlation between the two evaluation methodologies. This correlation suggests that despite task differences, models adhering to preferences in one setting perform similarly in the other. Classification tasks could thus serve as an efficient proxy for evaluating preference following in complex generation scenarios across models and methods.
3.5 ERROR TYPE ANALYSIS
What causes LLMS to fail and what capabilities are missing in preference following? LLM- based evaluations allow us to efficiently analyze error types defindd in $2.5 and we analyze them with the key capabilities defined in $2.1. Figure 7 shows the distribution of error types across two LLMs and five methods at turn 10. In zero-shot settings, LLMs generally lack awareness of user preferences, leading to high CiteernnceUnnwwarr Violations. With advanced methods, this error type percentage drops, indicating lack of Personalization Proactiveness ability in the zero-shot setting. However, with these methods, Inconsistency Violations appears, indicating that even when prefer- ences correctly retrieved, L.LMs struggle generate aligned responses, lacking the Preference
7

---

Published as a conference paper at ICLR 2025
Classification Task Accuracy (Explicit Preference)
Zer Zero-Shot
Reminder
Self-Critic
CoT
RAG (Top
0.2k
95
92
B7
90
85
96
DG
95
98
94
91
B7
73
75
90
79
96
26
54
90
95
03
Input Prompt Token Length
1k
80
66
70
53
98
95
94
85
95
80
85
70
60
05
86
73
93
88
87
79
86
07
90
95
BF
96
87
3k
74
39
57
48
96
92
94
75
76
58
54
58
57
02
90
83
79
73
69
50
97
97
91
86
95
80
10kk
85
59
43
81
2.kk
80
09
8.0
75
31k
50
79
89
10k
85
74
12
66
85
54kk
64
28
103k
49
68
Clust Bornet
Clasde Hebll
Mirnel Bert
Ulema) you insruct
seater
Litoaa ea insireet
Chaude deervet
clando Heals
Nixerall Bx70
Uiima? 708 Instret
Lammt eneette
Crawte Sterool
Cleste Hanlll
regerel Bxib
Lommd 708 Inatect
Crasse Usend al avensed Siered
Clnsare Hakd
eespraal be
Laandd evereet Liarnd na eretued
sistrll 7D
Chacte clavae Hisi
Stowel
Anterall ierra
Liernd iivccue Liama aB Instrct
Mizeol 70
Mistill
Figure 6: Performance of the classification task across models and methods on the explicit prefer- ences dataset across various input lengths between the user's 'er's preference and the final query. Results are averaged over 20 topics.
Following capability. Interestingly, while prompting methods are introduced to enhance preference following, they inadvertently incentivize LLMs to hallucinate preferences. RAG reduces the halluci- nation error percentage for Mistral 8x7b. Additionally, Claude 3 Sonnet exhibits a high percentage of Unhelpful errors with prompting methods, often refusing to answer queries due to a perceived lack of context regarding user preferences, which is undesirable for effective conversational person- alization. We include further discussions on long-context error type analysis with more LLMs and how they scale with turns in Appendix A.
Error Type Distribution (At Turns 10)
Figure 7: Distribution of 4 disjoint error types across 2 LLMs and 5 methods in the genera- tion task with explicit preferences. where the gap between the preference and the query 10 turns. Prompting methods introduce more hallucination and unhelpful errors, as LLMs either fabricate user preferences or refuse to respond due to failures in recalling the pref- erences, which is undesirable for personalized conversational chatbots.
Unhelpful Response
Preference Hallucination Violation Preference-Uma Una Violation
Incomistency Violation
Claude 3 Sonnet
Zero-Shot Reminder
CoT
SeMtCritiic
RAG
Zero-Shot
Reminder
CoT
Self-Critic
Mistral Bx7b
3.6 DYNAMIC PREFERENCE FOLLOWING
In real-world scenarios, users may express multiple preferences within single conversation, and these preferences can evolve or even conflict as the dialogue progresses across different sessions. We explore how the presence of multiple preferences and confficting preferences affect the model's ability to maintain adherence.
Impact of Multiple Preferences Stated in Conversation. We investigated whether introducing non-confticting preferences from different topics throughout a conversation would affect the model's ability to follow an initial preference. To evaluate this, we evenly inserted additional preferences at various points in the conversation and measured the model's adherence by asking a query related to the first preference. As shown in Figure 8, the results indicate that adherence accuracy for the initial preference increases as more preferences are introduced, even when these preferences are unrelated to the first. We conjecture that when more preferences are presented, the model may be implicitly encouraged to treat the cumulative set of preferences as a broader constraint its outputs. Conse- quently, introducing multiple preferences may help the model pay more attention to each sterestated preference throughout the conversation, leading improved personalization. Effect of Conflict- ing Preferences Stated in Conversation. We also analyzed how LLMs adapt to conflicting user preferences within conversation. We generated conflicting preferences for the original preferences
8

---

Published as a conference paper at ICLR 2025
Conflicting vs Non-confliicing Preferences
Accuracy Preference (Stated Turn 0)
Multiple Preferences Analysis:
Pref Following Accuracy
preference
Pref Pref Following Accuracy
0.4
0.45
0.3
0.40
0.2
0.0
Claude Haiku
Claude Sonnet
10
15
Conversation
Figure 9: Effect of adding conflicting versus non-confficting preferences on adherence. The red bar indicates the performance when only the original preference is present. Results are aver- aged over five topics using fixed 100-turn con- versation.
Figure 8: Introducing multiple preferences throughout conversation improves adherence to the initial preference. Results are using Claude Sonnet with Reminder prompting.
across five topics using Claude 3.5 Sonnet. These conflicting preferences were inserted alongside the original preferences predetermined positions in the conversation, while the rest of the con- versation was kept constant. For comparison, we also tested the impact inserting non-conflicting preferences. In both settings, the original preferences were stated later than the conflicting one, and we will test the model's preference following to the original preference. As shown in Figure 9, when conflicting preferences are introduced, the model actually demonstrates improved adherence to the original preference compared to when non-conflicting preferences are added. Comparing the adherence performance the baseline scenario where only the original preference is present (indi- cated by the red bar), both the conflicting and non-confiicing preference scenarios achieve better adherence than the sangle-preference scenario. reinforeing the observation from the previous section that introducing multiple preferences encourages the model to better retain and follow preferences. These findings suggest that conflicting preferences do not necessarily pose a challenge for the L.M but rather reinforce its capacity to track and adapt to evolving user preferences.
3.7 FINETUNING PREFEVAL TO IMPROVE PREFERENCE FOILLOWING
Enhancing Preference Following through Supervised Fine- Fine-Tuning (SFT) PREFEVAL: We fine-tuned the Mistral-7B model using SFT on 80% of the topics in PREFEVAL and evaluated on the remaining unseen 20% topics for the generation task. For SFT data, used Mistral-7B's responses generated using the Reminder method, without any contextual turns inserted between the stated preference and the query. During training, to simulate conversational preference following, we inserted 0, 5, or 10 contextual turns between the preference and query, resulting in training data of 2, 7, or turns (where the preference, query, and response constitute 2 turns). After fine-tuning, the model demonstrated significant improvement in the zero-shot setting, surpassing the previous best-performing method (RAG), as shown Figure 10. One notable benefit improved length generalization the capability to follow preferences for longer contexts compared to training. When the model is trained with 10-turn contextual turns, generalizes 70-turn contexts much more effectively than when trained with fewer contexts. This suggests that SFT enhances both the ability to handle long-context retrieval and the capacity to infer and follow user preferences over extended conversations.
To understand the mechanisms behind this improvement, we analyzed the attention patterns be- fore and after fine-tuning. Our analysis reveals that the SFT model consistently exhibits increased attention to preference-related information during generation, with improvements in preference re- gion attention up 4.97% across test examples. See Appendix A.13 for detailed attention score visualizations and analysis.
4 RELATED WORK
LLM Personalization and Benchmarks. Early personalization efforts focused dialogue sys- tems mimicking user styles (Zhang et al., 2018; Mazaré et al., 2018; Wu et al., 2021; Zhong et al., 2022) and tasks like news headline (Ao et al., 2021) and review generation (Li & Tuzhilin, 2019).

---

Published as a conference paper at ICLR 2025
Figure 10: Performance on the 20% unseen test top- ics: After fine-tuning, the Mistral-7B model exhibits superior preference follow- ing accuracy compared to other baseline models. also shows enhanced length generalization when trained with longer context inter- leavings, demonstrating its ability handle extended conversational turns more effectively.
SFT results with
Preference Following Accuracy
1.0
W/ SFT (Zero-Shot)
0.8
(0 turn insertion)
W/ SFT (Zero-Shot)
(5 turn insertion)
0.6
W/ SFT (Zero-Shot)
(10 turn insertion)
w/o SFT (Zero-Shot)
0.4
w/o SFT (Reminder)
w/o SFT (CoT)
0.2
w/o SFT (SelfCCniic)
w/o SFT (RAG (Top 51)
Token
1K3k
10k
16k
23k
Turn
510
30
50
70
Recent LLM personalization benchmark works include LAMP (Salemi et al., 2023), which empha- sizes explicit user profile conditioning through retrieval-augmented techniques, RPBench-Auto' character-based role-playing tasks (Boson AI, 2024), TIMECHARA's temporal consistency in char- acter representation (Ahn et al., 2024), and RoleLLM's fine-grained role-playing framework (Wang et al., 2024b). While these often focus on stylistic preferences or single-turn tasks (Lee et al., 2024; Li et al., 2024a; Jang et al., 2023; Zhao et al., 2023), our work addresses lifestyle preferences and extend to long-context, multi-turn conversations. Long Context LLM and Benchmarks. With extended context windows (Reid et al., 2024), long-context LLMs have emerged (Agarwal et al., 2024; Bertsch et al., 2024). Existing benchmarks primarily evaluate information retrieval capabil- ities through tasks requiring models to locate specific facts answers (Zhang al., 2024; Wang et al., 2024a; An et al., 2023; Li et al., 2023b), including tasks like fact reasoning, and coding (An et al., 2023; Bai al., 2023; Kočiský et al., 2018; Dasigi al., 2021; Huang et al., 2021; Li et al., 2024b; Kuratov et al., 2024). While these "needle-in-a-hayee hhaesiiin--aasttcc"" tasks test model's 's ability identify and extract relevant information, our benchmark introduces distinct challenge of preference following, where models need to infer from implicit preferences and dynam- ically apply this understanding across conversation contexts rather than simply retrieving explicit preferences. Instruction Following. Fine-tuning on numan-annotated instruction-response pairs has enhanced L.LMs' instruction- following capabilities, as seen in work like InstructGPT (Ouyang instruetionfoolloiing
retrieval,
et al., 2022). These models perform broad range of tasks, including summarization, translation, and problem-solving (Zhong ct al., 2024; Zhou et al., 2023). While existing benchmarks focus on executing discrete instructions, our work extends this paradigm by emphasizing the inference and adherence to user preferences across multiple conversation turns. Importantly, we also assess Per- sonalization Proactiveness, which is the initiative and ability to know when and where to apply user preferences, moving beyond discrete task execution. Due to space constraints, we provide a more comprehensive discussion of Related Work in Section A.2.
5 CONCLUSION
In this work, we present PREFEVAL, a comprehensive benchmark addressing a critical gap in eval- uating large language models' ability to follow user preferences in multi-session conversational settings. Our benchmark considers comprehensive aspects, including explicit and implicit prefer- ences, both generation and classification tasks, and employs LLM--bsed and automatic evaluation methods. Through rigorous testing of 10 state-of-the-art LLMs across 20 diverse topics and various conversation lengths up to 100k, we demonstrate that preference following remains significant challenge. Our findings reveal that even advanced models struggle to maintain adherence with user preferences in conversational setting, with accuracy dropping below 10% in default settings for con- versations exceeding 10 turns. These models struggle to proactively recall and incorporate user preferences stated earlier in conversations without explicit prompting. Implicit preferences create further difficulties for LLMs to infer user preferences. While prompting techniques, such as re- minders, show promise in mitigating this performance drop, substantial room for improvement still remains. PREFEVAL, not only highlights the current limitations of L.MMs in personalized interaction but also provides a valuable resource for researchers and developers to evaluate and enhance the personalization capabilities of conversational AI systems.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
M. Hong's work partially supported by the NSF grant CIF-2414372.
7 REPRODUCIBILITY STATEMENT
Our work presents a comprehensive benchmark that includes a manually curated dataset to evalu- current open-source and proprietary models (see Section 3.1), with detailed descriptions of the model versions provided in Table 3. We plan to release our benchmark in the future, enabling others to reproduce our results. Additionally, we will make available the contextual turns from the Lmsys- IM-dataset used to construct inter-turn distractors. To ensure reproducibility, we also provide the prompts used for LLM-based evaluators (Claude 3 Sonnet) in Section A. which are critical for obtaining performance results, well as the detailed method descriptions and their prompts used in our experiments (Section A.4). These resources will facilitate further research and allow for replication of our work.
8 ETHICS STATEMENT
In this paper, introduce PREFEVAL, benchmark designed to evaluate large language models' ability to infer, memorize, and adhere to user preferences in long-context multi-session conversa- tional settings. Our research prioritizes responsible and ethical practices, particularly concerning data privacy, data quality in terms of ethics, bias mitigation, and research integrity. Our dataset con- sists of manually curated preference-query pairs spanning 20 topics, generated with assistance from AI language models. In the construction process, we have invested significant effort in manually rating and filtering these pairs to ensure quality and relevance, removing any potentially unethical preference instances. We also examined inter-conversation data from the LMSYS dataset, which contains anonymized interactions between conversations from our experiments. Throughout this process, we maintained strict privacy stan- dards, ensuring no sensitive or personal information was collected or used. In the benchmarking process, we continuously examined the LLM's output to ensure preference pair led to unethical responses, and we have detailed the API versions of the L.MSs we benchmarked for reproducibil- ity. In terms of future deployment, the enhancement of an LLM's ability to remember and follow user preferences might raise privacy considerations. We advocate for responsible deployment prac- tices that protect user data. We acknowledge that LLMs may inherit biases from their training data, potentially leading to unfair or discriminatory outputs. To address this concern, we aimed to op- timize the diversity of the topics in dataset to minimize potential biases. All results presented accurately represent our findings, supported by detailed documentation of our methodologies, such the prompts we used. We conducted all experiments using either publicly available models or through documented commercial API access. To promote reproducibility and advance research in this field, we will make our benchmark dataset publicly available.
and language models, and removed problematic

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv: 08774, 2023.
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, al. Many-shot in-context learning. arXiv preprint arXiv:2404.
2024.
Jaewoo Ahn, Tachyun Lee, Junyoung Lim, Jin-Hwa Kim, Sangdoo Yun, Hwaran Lee, and Gunhee Kim. Timechara: Evaluating point-in-time character hallucination of role pole-playing large language models. In Findings of ACI,, 2024.
Chenxin An, Shansan Gong. Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307. 1088, 2023.
Xiang Ao, Xiting Wang, Ling Luo, Ying Qiao, Qing He, and Xing Xie. PENS: A dataset and generic framework for personalized news headline generation. Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 11th International Joint Conference on Nat- ural Language Processing (Volume Long Papers). pp. 82-92, Online, August 2021. As- sociation for Computational Linguistics. doi: 10.18653/v1/222..acl/ / s/aclanthol ogy- org/ 2021 20w.2.11ll1....... acl-llngllnon....
URL https:
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:22 2022.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang. Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework align large language model with recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, pp. 1007-1014, 2023.
Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew Gormley, and Graham Neu- big. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200, 2024.
Boson AI. Rp-bench. https: /wtp:////oooo.oo/ ai/rpbench/ 2024. Accessed: Novemenber 2024.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matr Gardner. A dataset of informatiion-sekking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011, 2021.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. Empirical Methods Natural Language Processing (EMNLP), 2021.
Himanshu Gupta, Saurabh Arjun Sawant, Swaroop Mishra, Mutsumi Nakamura, Arindam Mitra, Santosh Mashetty, and Chitta Baral. Instruction tuned models are quick learners. arXiv preprint arXiv:2306.05533 05330. 2023.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference Learning Representations.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. arXiv preprint XXiv:2/04.02//3 2021.
12

---

Published as a conference paper at ICLR 2025
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Per- sonalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310. 1564, 2023.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Mclis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of Ilms with long context prasoning-in---baystee arXiv preprint arXiv:2446. 2024.
Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to thousands of prefer- ences via system message generalization. arXiv preprint 1rXiv:2405.17977, 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et eetrieval-augmentee genera- tion for www.lldge-nnteniiiee nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Cheng Li, Mingyang Zhang, Qiaozhu Mei, Yaqing Wang, Spurthi Amba Hombaiah, Yi Liang, and Michael Bendersky. Teach llms to personalize-an approach inspired by writing education, arXiv preprint w:Xiv:2308.07968, 2023a.
Jiaqi Li, Mengmeng Wang. Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? arXiv preprint (oxXx:23/11049000 2023b.
Junlong Li, Fan Zhou, Shichao Sun, Yikai Zhang, Hai Zhao, and Pengfei Liu. Dissecting human and Ilm preferences. arXiv preprint arXiv:2402. 2024a.
Pan Li and Alexander Tuzhilin. Towards controllable and personalized review generation. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UCNLP), pp- 2019.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context iims struggle with long in-context learning, 2024b.
Jiongnan Liu, Yutao Zhu, Shuting Wang, Xiaochi Wei, Erxue Min, Yu Lu, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. Llms+ persona-plug= personalized llms. arXiv preprint arXiv:2409. 1901, 2024a.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024b. doi: 10.1162/tacl. a 00638. URL
https: /aaaacaaanttt aal.anthologgy.o..o.. org/ 2024 tacl-1.9.
Pierre-Emmaniu Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. Training millions of personalized dialogue agents. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2775-2779, Brussels, Belgium, October-November 2018. Association for Com- putational Linguistics. doi: 10.18653/v1/D18 1228.. URL petttss:/aclaatthtt .ggg.ogg////oooooooooo.ogo D18-1298.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 188)99 2019.
13

---

Published as a conference paper at ICLR 2025
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv preprinl urXiv:2304. 1406, 2023.
Omar Shaikh, Michelle Lam, Joey Hejna, Yijia Shao, Michael Bernstein, and Diyi Yang. Show, don't tell: Aligning language models with demonstrated feedback. arXiv preprint 2rXi::2406.008888, 2024.
Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. Democra- tizing large language models via personalized parameter-efficiee fine-tuning. arXiv preprint arXiv:2402.04400 2024.
Sebastian Vincent, Alice Dowek, Rowanne Sumner, Charlotte Blundell, Emily Preston, Chris Bayliss, Chris Oakley, and Carolina Scarton. Reference-less analysis of context specificity in translation with personalised language models. arXiv preprint arXiv:2303. 16618, 2023.
Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Qian Wang, and Yue Zhang. Novelqa: A benchmark for long-range novel question answering. arXiv preprint arXiv:2403.00 12766, 2024a.
Noah Wang, Z.y. Peng, Haoran Que Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang. Jie Fu, and Junran Peng. RoleLLM: Benchmarking, eliciting, and enhancing role- playing abilities of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties: ACL 2024, pp. 14743-14777, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/ f.n.02.ffnnigss---. 8c1.878. URL 878.
202/ Findings-ach
Yuwei Wu, Xuezhe Ma, and Diyi Yang. Personalized response generation via generative split mem- ory network. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 1956-1970, Online. June 2021. Asso- ciation for Computational Linguistics. doi: 10. 20118855//11/00 122/3//112211111 mmacl-main.1a URL https: //aclanthologg /tt///////// 2021 maaacclmmii.....
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: have a dog, do you have pets too? In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguisties (Volume 1: Long Papers), pp. 2204-2213, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18633/v1PP/ 1205. URL https: //aclanthologg org Pagg/P1811200
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402. 13718, 18, 2024.
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. arXiv preprint arXiv:2310.1 11523, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-Ine mm: A large-scale real-world Ilm conversation dataset, 2023.
Hanxun Zhong, Zhicheng Dou. Yutao Zhu, Hongjin Qian, and Ji-Rong Wen. Less is more: Learning to refine dialogue history for personalized dialogue generation. In Marine Carpuat, Cathe-CCteernee de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022
14

---

Published as a conference paper at ICLR 2025
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. tion for Computational Linguistics. doi: 11t6655/w/222222mmmlmmmaaamm faclanthology.... www.:/2...r.22.....oo//22 222.22222222a222 naacl - ma in.inn... 426.
Seattle, United States, July 2022. Associa- URL https:
Ming Zhong, Aston Zhang, Xuewei Wang, Rui Hou, Wenhan Xiong, Chenguang Zhu, Zhengxing Chen, Liang Tan, Chloe Bi. Mike Lewis, Sravya Popuri, Sharan Narang, Melanie Kambadur, Dhruv Mahajan, Sergey Edunov, Jiawei Han, and Laurens van der Maaten. Law of the weakest link: Cross capabilities of large language models. arXiv preprint arXiv:2409, 19951, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
Yuchen Zhuang, Haotian Sun, Yue Yu, Qifan Wang, Chao Zhang, and Bo Dai. Hydra: Model factorization framework for black-box Ilm personalization. arXiv preprint arXiv:2406.02888 2024.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 LIMITATION
Our benchmark aims to evaluate preference following rather than verifying the factual accuracy of the recommendations. While the LLM's recommendations may contain inaccurate information, fact-checking a separate dimension beyond the scope of this work. Additionally, although we have conducted extensive human filtering on our synthetic preference-quee query pairs, incorporating real user preferences in future work would help capture more nuanced aspects of user interactions.
For implicit preference elicitation, we designed the available options such that only one option adheres to or violates the target preference. While this approach yields high probability of inferring the user's preference from their choice, it does not guarantee 100% accurate inference from the multiple-choice selection. However, given the early turns' results of a relatively high preference- following accuracy of the strongest baseline is around 90%, we believe the errors should be within an acceptable range.
A.2 DETAILED RELATED WORKS
LLM Personalization and Benchmarks. Prior to the era of LLMs, personalization for language models mainly focused on personalized dialogue systems. These systems conditioned agents on user profiles, such as Reddit posting histories, to generate more engaging chit-chats that mimicked users' personalities or styles (Zhang et al., 2018; Mazaré et al., 2018; Wu et al., 2021; Zhong et al., 2022). Other personalization tasks included news headline generation (Ao et al., 2021) and re- view generation (Li & Tuzhilin, 2019). With the rise of LLMs, the scope of personalization tasks has broadened. Recent works aim to personalize the LLMS themselves to embody particular per- sonas and mimic speaking styles (Vincent et al., 2023). In recent work on personalization bench- marks, Salemi et al. (2023) introduced the LAMP benchmark, which tests LLMs' ability to produce personalized output in specific tasks such as movie tagging and email subject generation and em- phasizes explicit user profile conditioning through aurrival-augmentee techniques. The RPBench- Auto benchmark (Boson AI, 2024) focuses character- based and scene-based role-playing tasks, evaluating models' abilities to maintain consistent personas across 80 unique characters in free- form conversations and structured narrative scenarios. TIMECHARA benchmark (Ahn et al., 2024) specifically addresses point-in-time character representation, examining how well models maintain temporal consistency in narratives without revealing future events or contradicting established char- acter timelines. RoleLLM benchmark (Wang et al., 2024b) introduces a framework for fine-grained role-playing across 100 diverse characters, emphasizing speaking style imitation and role-specific knowledge capture through systematic instruction tuning. Beyond benchmarking efforts, researchers have explored various personalization methods, Bao al. (2023) worked on fine-tuning LLMs for recommending items using users' past interactions. Jang et al. (2023) aimed to align LLMs with multi-preferences that can be combined post-training through parameter merging. Zhao et al. (2023) considered sew-shot adaptation of LLMs to cater to human group preferences across demographies. However, these recent works mostly consider preferences about LLM response stylistic attributes such as conciseness tasks like email title generation. Our work differs from these benchmarks by focusing on adherence to more lifestyle-oriented, day-to-day user preferences and we extend to long-context, multi-turn conversations.
informativeness (Lee al., 2024; Li et al., 2024a) or focus on single-turn
Long Context LLM and Benchmarks. Recent LLMs such Gemini have extended the context window to millions of tokens (Reid et al., 2024), enabling researchers to expand from few-shot to many-shot settings (Agarwal et al., 2024; Bertsch al., 2024). To keep pace with increasing context length capabilities, new datasets and benchmarks have been proposed to evaluate long-context rea- soning abilities, primarily focusing on question-answering and summarization tasks (Zhang et al., 2024; Wang et al., 2024a; An et al., 2023; Li et al., 2023b). For instance, L-eval (An et al., 2023) and LongBench (Bai et al., 2023) are among the first to aggregate existing benchmarks such as Kočiský et al. (2018); Dasigi et al. (2021); Huang et al. (2021) into long-context benchmarks, span- ning tasks like question- answering. summarization, retrieval, and coding. Kuratov et al. (2024) designed benchmark to assess LLMs' fact reasoning abilities across extremely long documents. LongICLBende (Li et al., 2024b) evaluates long in in-context learning in extreme- classification tasks, where the model needs to comprehend the entire context to understand the label space. While
16

---

Published as a conference paper at ICLR 2025
these -nt-all-in- tasks test model's ability to identify and extract relevant information, our benchmark introduces a distinct challenge of preference following, where models need to infer from implicit preferences and dynamically apply this understanding across conversation contexts rather than simply retrieving explicit preferences. Our henchmark evaluates L.MSs long-context re- trieval capabilities in more realistic and practical setting by assessing sefference-following across multi-turn conversations, It demands a comprehensive understanding of the conversational flow, en- abling models to accurately infer user preferences they evolve over dialogue and know when and where to apply these preferences in responses.
Instruction Following, Recent LLMs are fine-tuned on human-annotatee annotated instruction- reepppcoe pairs to enhance their iestruction-followwee abilities. Works such as InstructGPT (Ouyang et al., 2022) have demonstrated this approach's 's effectiveness enabling models to understand and deliver on tasks specified by humans. These instructions encompass wide range of tasks and require com- position of capabilities (Zhong al., 2024; Zhou al., 2023), including answering questions con- cisely, summarizing long texts, translating between languages, explaining complex topics at various educational levels, and solving mathematical or logical problems step-by-step (Radford al., 2019; Gupta al., 2023). Our work extends this concept by considering another dimension of instruction following: the ability to infer and adhere to user preferences across multiple turns of conversation, rather than focusing solely on executing discrete tasks queries.
A.3 MODEL VERSION
With PREFEVAL, have evaluated the following large language models in our experiments with their versions in Table 3.
Table 3: The LLMs benchmarked and their respective versions used for evaluation.
Model Name
Version
Claude 3 Sonnet
matinppcccaaee---smmee-22222222-
Claude 3 Haiku
Sauue-3.5-Sonnee
Mistral 7b Instruct
mistallmiisrall/bbinsstuttt2222
Mistral 8x7b Instruct
mistralmiixraalk887-hisssttt--
LLaMA 8b Instruct
metaallamaa--B-iinsnucc...
LLaMA 70b Instruct
meta.llamaa--0hh-innnrnce---
GPT-4o
GPT-40-2024-08-06
ol -peeviiww ol 11-revview-2224-09-12
Gemini-1.5-Pro
as of Sep 30th, 2024.
A.4 METHODS DESCRIPTION
We extensively evaluate variety of staae-of-ttheaar large language models and investigate five meth- ods for each model:
Zero-shot:
The default case, where the LLM directly answers the user's query without any addi- tional prompting.
Reminder: Before answering the question, the LLM provided with reminder sentence con- sider the user's previously stated preference in its response. The reminder used is:
"In response, please ensure that you take into account our earlier discussion, and provide an answer that is consistent with my preference."
This reminder is added right after the question and before the LLM'' response.
17

---

Published as a conference paper at ICLR 2025
Self-Critic: The LLM first generates initial zero-shot response the question, then produces a critique assessing whether it has followed the user's preference, and finally generates a revised response incorporating the critique. In total, three calls to the LLM made. This self-critic process is akin to Intrinsic Self-Correction, as termed in Huang et al..
The critique and revision request prompts as follows:
Critique Request: Review your previous response to the user's question last conversation turn. Check if the response adheres to violates any user preferences stated earlier in the conversation that relate to this query. Provide critique on how well those preferences were followed in sentences.
Answer in this format:
Critic: (Your Critique Here]
Revision Request: Based on your critique, please rewrite your previous response to align more closely with the user's earlier stated preferences. Answer the question again:
Response: [Revised Response Here]
Few-Shot Chain-of-Thought (CoT): The LLM is given several few-shot examples (in exper- iments, we used 5-shot) of chain-of-thoughi that demonstrate how to follow the user's preference before answering the question. The chain of thought prompt is as follows:
When answering a user's question, a good assistant should carefully consider the user's stated prefer- ences and tailor the response accordingly.
Example 1: User's preference: "I have an intense phobia of bugs and insects. What outdoor activities would you recommend for the summer?" Good assistant response: "Considering your phobia of bugs and insects, I'll focus on recommending activities that minimize encounters with insects. Let's explore outdoor options with fewer bugs, like beaches, pools, urhan areas, well-maintained park trails, and activities during cooler times of day. I'll avoid suggesting things like camping, off-trail hiking, picnicking. or activities in dense forests. Here are some summer outdoor activities you might enjoy: spending a day at beach swimming playing sports, exploring ouldoor malls or pedestrian areas parks (avoiding dense wonds), attending evening outdoor concerts or movies, and enjoying water activities like kayaking paddle boating lakes rivers. These options should help you enjoy the outdoors while minimizing encounters with insects."
city, hiking on maintained trails
User's preference: "I'm vegan looking for new dinner recipes. Any ideas?" Good assistant response: "As a vegan, you'll need plant-based recipes without animal products. I'll focus on nutrient-rich, diverse ingredients to ensure balanced meals. Consider: quinoa and black bean burrito bowls, lentil and vegetable curry, roasted vegetable and hummus wrap, or zucchini noodles with avocado pesto. These options offer protein, fiber, and essential nutrients while adhering vegan principles."
Example 2:
Example 5: User's preference: "I'm sensitive to caffeine but need an energy boost in the mornings. Any suggestions?" Good assistant response: "Given your caffeine sensitivity, we'll explore natural, caffeine-free energy boosters. Focus on lifestyle changes and alternatives to caffeinated drinks. Try: starting your day with a brisk walk or light exercise, drinking glass water with lemon, eating protein- rich breakfast, or trying herbal teas like peppermint or ginger. Avoid coffee, energy drinks, and black tea. These alternatives can help increase alertness without the jitters crashes associated with caffeine."
Now, please answer the following question while considering preferences (not the user preferences in the examples above), which have stated either explicitly or implicitly in our previous conversation:
Retrieval-Augmennted Generation (RAG):
We employ a sentence embedding model, which is haancccteettttsssssiiieeeerrrrrr (Gao et al., 2021), to retrieve the most relevant conversa-
18

---

Published as a conference paper at ICLR 2025
tion exchanges based similarity to the current query. The top five most relevant exchanges then presented to the LLM as contextual information to guide its response.
The prompt used in this method is structured as follows, here show RAG with top-5 retrieved exchanges:
Before answering my question, please consider the following context from our previcus conversations. These are the most relevant exchanges that had previously, which may contain information about my preferences prior discussions related to my quezy:
#Start of Context#
exchange 1. [Most relevant exchange I|
exchange 2. [Most relevant exchange 2]
exchange 3. [Most relevant exchange
exchange 4. [Most relevant exchange
exchange 5. [Most relevant exchange
#End of Context#
Please use this context to inform your answer and adhere to any preferences I've expressed that are relevant to the current query. Note that not all contexts are useful for answering my question and there may be no context that useful. Now, please address my question:
A.5 CLASSIFICATION TASK CORRELATION PLOT
Correlation: Generation vs. Classification Task
1.0
Accuracy (Classification Task)
Regression line (r=0.73)
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.4
0.6
0.8
1.0
Accuracy (Generation Task)
Figure 11: Correlation analysis between LLM-based preference following accuracy in generation tasks and classification accuracy in classification MCQ tasks across all models and methods. Across models, methods and 12 turns, each point on the seatter plot is averaged over 20 topics. A correlation coefficient of 0.73 suggests strong alignment between the two evaluation approaches.
19

---

Published as a conference paper at ICLR 2025
A.6 PRACTICAL GUIDE TO USING PREFEVAL.
While our benchmark enables comprehensive evaluation across multiple dimensions (including var- ious baselines, conversation turns, preference forms and topics, and tasks), benchmarking com- plete setups is computationally intensive. For practical use, we provide guidance based on available resources. If an evaluator model like the Claude Sonnet is available, then the generation task can be used; otherwise, one can choose a local LLM as the evaluator or opt for our classification task, which does not require L.MM-based evaluators but still has strong correlation with genera- tion task performance (see Sec 3.4). For initial testing, we recommend starting with subset of topics and conversation lengths using explicit preference forms. As an example, our repository (https: /gitttttitg github.... com/ amazon-scienne seceeeee/eeeeeeeeeeeeeeeeeeeeeec /erefevall includes leaderboard comparing vari- ous LLMs on the "travel restaurant" topic at both 10 and 300 turns, assessing both short-turn and long-ture turn cefference-flllowiine capabilities. With additional computational resources, one fur- ther use generation task evaluators for detailed error- type analysis, and test implicit preference forms evaluate advanced sefference-followiee capabilities.
A.7 PROPRIETARY LLMS PERFORMANCE COMPARISON
The performance comparison shown in Table 4 shows that current SoTA proprietary LLMs struggle to proactively follow user preferences in short-turn, zero-shot settings, where no explicit prompting is provided. Only when using the Reminder method, which explicitly reinforces the need to adhere to preferences, do these models show improvement; however, accuracy still deteriorates with longer context lengths. Note that the o-series models' results may not be directly comparable to other models as they may require additional test-time computation with "thinking" phase.
Table 4: Comparison of preference-followee accuracy across SoTA LLMs evaluated at two context lengths with two methods: Zero-shot and Reminder (best prompting method), evaluated at two context lengths, on the travel restaurant topic and on generation task.
10 ~33k tokens
Turns 103k
Zero-shot
Reminder
Zero-sbot
Reminder
Claude Sonnet
0.02
Gemini-1.5-Po
0.09
GPT-4o
GPT-4,1
0.50
0.11
ol-preview"
0.50
0.14
04 o4-mini"
0.23
0.98
0.02
0.54
03'
0.71
0.05
0.43
20

---

Published as a conference paper at ICLR 2025
A.8 RAG METHOD ABLATION AND ANALYSIS
To examine how RAG Reerieval-AAugmente Generation) improves performance, we compare the RAG sentence transformer's ground truth retrieval accuracy in explicit preference settings. We con- sider it accurate if the explicit preference stated in turn is extracted by the sentence transformer and sent to the LLM reference. As shown Figure 12, we find that when k=5, the performance is among the best, similar to when k=10. Although when k=10 the ground truth retrieval accuracy improves, the preference following performance does not reflect this improvement. This may sug- gest that for RAG, providing more turns of exchanges as reference might serve as another form of distraction, potentially harming performance. Based on these findings, we select k=5 to report this method's results in the main paper.
RAG: Top-K Retrieval Accuracy
RAG Top-k Ablation: Claude Sonnet
Performance Topics (Explicit Preference)
1.0
Top-1
1.0
Zero-Shot
Top-5
RAG (Top
Top K Retrieval Accuracy
0.9
Tbp-10
Preference Adherence Accuracy
0.8
RAG (Top
Top-15
0.8
0.8
0.7
0.4
0.6
0.5
150
200
50
1000
150
200
250
300
Total Number of Conversational Tums
between
Figure 12: Comparison of RAG retrieval accuracy versus RAG method's performance in preference following across different Top-K K values.
A.9 CROSS-TOPIC PERFORMANCES
We present an extensive benchmark study conducted across 20 diverse topics, on 6 Large Language Models at three fixed conversation lengths: 10, 70, and 300 turns. These correspond to approximate context lengths of 3k, 23k, and 100k tokens, respectively, spanning from the user's initial stated preference to their final query. The user's stated preference is positioned at the beginning of each conversation. We show results across three preference forms and evaluates five different methods. The detailed results are presented in Tables 5, 6, and 7 for explicit preferences; Tables and 10 for implicit choice-based preferences; and Tables 11, 12, and 13 for implicit persona- persona-based preferences. This comprehensive set results allows for a thorough examination of LLM performance across various topics, conversation lengths, and preference revelation forms.
21

---

Published as a conference paper at ICLR 2025
Table Preference adherence accuracy across language models and two baselines when the number of conversation turns between the stated preference and the query is 10 (token length ~ 3k), across 20 topics in the explicit preference setting.
Travel Restaurant
Travel Hotel
Travel Activities
Travel Transportation
Entertain Music&Book
Entertain Sports
Entertain Shows
Entertain Games
Lifestlee Dictary
Lifestyle
Health
Clande Sonnet Zom-Shot
5.36
16.67
5.17
10.87
10.71
7.69
9.68
17.65
3.51
18.37
Claude Haiku Zero Shot
18.52
26.79
5.27
22.45
Llama3 8B Instrutt Zero-Shot
0.00
1.85
1.72
17.39
8.93
1.92
4.84
9.30
1.25
14.29
Llama3 70B Instruct Zero Shot
10.71
9.25
3.45
17.39
16.07
0.00
11.29
11.76
5.26
26.53
Mistral 7b Zero-Shot
3.57
31.48
1,72
19.57
16.07
7.69
6.45
17.65
8.77
26.53
Mistral 8x7b + Zero Shot
8.93
11.11
5.17
13.04
16.07
3.85
14.52
17.65
3.51
28.57
Claude Sonnet Reminder
96.43
100.00
100,000
100,00
100.00
100.00
98.39
100.00
98.25
100.00
Claude 3 Haiku Reminder
67.86
92.59
79.31
97.83
96.13
91.23
88.71
100.00
59.65
93.88
Ulama3 8B Instruct Rerninder
$7.14
88.89
N7.
97.
92.86
90.38
85.
98. 04
36,84
85.71
Llama3 Instrutt Reminder
37.50
24.077
29.31
41.07
54.39
Mistrl 7b Reminder
75.00
96.30
K2.76
95.65
91.07
94.23
88.71
98 04
$7.89
93.88
Mistral 8x7b Reminder
83.93
98.15
89.66
97.83
94.64
88.46
90.32
98. 04.04
85.96
95.92
Claude
(Top
64.15
$8.46
N7.93
95.00
100.00
91.11
Claude 3 Haiku RAG (Top
52.83
71.15
68.97
91.30
89.29
86.54
91.67
90:20
54.39
93.33
Llam33 SB Instruct + RAG (Top 5)
58.19
86.54
77.59
89.13
96.43
86.54
90.00
96.08
49.12
91.11
Llama3 70B Instruct RAG (Top
58.49
76.92
81,03
89.13
82.14
84.62
81.67
84.31
68.42
82.22
Mistrl 76 RAG (Top 5) Mistral
62.26
96.15
79.31
95.65
92.86
84.62
86.67
96. 08
70.
93.33
(Top
67.92
96.15
89.29
90.20
77.19
Clanle 5 SSnnet Self-Crilic
78.57
92.59
100.00
93.48
98.21
96.15
93.55
90.20
87.72
93.88
Clande Halka Self-Crinic
73.21
92.59
91.38
93.48
92,86
90.38
85.48
92. 16
63.16
91.84
Llama3
Self-Chitic
Llama3 70B Instruet Self-Crink
87.50
92.59
84.48
95.65
85.71
76,92
83.87
88. 24
82.46
89.80
Misnal Self-Critic
67.86
98.15
98.28
91.30
85.71
86.54
90.32
96.08
59.65
95.92
Mistral 8x7b Self-Critic
58.93
87.04
74.14
84.78
78.57
66.13
90.20
70.18
85.71
Claude 3 Sonnet + CoT
41.07
72.22
87.93
78.26
85.71
80.77
82.26
92.16
31.58
89,80
Claude Haiku
58.93
$5.19
92.86
90:20
50.88
69.39
Llam3 SB Instruct CoT
17.86
22.22
36.21
67.39
42.86
$7.69
21.19
21.57
36.84
75.51
Luma3 70B Instruct + CUT
$8.93
25.93
51.72
41.30
50.00
48.08
37.10
45 15.10
56. 14
77.55
Mistrl
32.14
83.33
22.81
Mistal 8k7b CaT
64.29
88.89
87.93
93.48
94.64
69.23
90.32
98.04
71.93
89.80
Lifestlee Fimess
I Lifodyle
Beauty
Shop Fishioa
Shop Home
Shop Mators
Shop Technodogy
Education Learaing Mtyle
Eduration Resources
Profredinal Wiek Styte
Pet Ownership
Average
Claude Sonnet Zero Shot
5:77
11.32
5.56
14.29
20.00
14.29
19:35
37.04
13.89
20.93
13:41
Cludde Halku
11:32
11.31
16.07
5.21
38.71
33.33
25.58
Llama3 SB Instruct + Zero Shot
5.27
7.55
5.56
14.29
17.78
11.43
12.90
14.81
8.33
32.56
9.67
Liama3 708 Instruct Zero-Shot
11:32
7.41
11.71
11.11
16.13
22:22
11.11
Mistol 7h Zero-Shat
11.54
22.64
14.51
28.57
17.78
17.14
25.81
18:52
11.11
39.51
17.35
Mistral 8x7b Zere-Shot
12.96
14.29
17.78
22
19.44
Clunde Sonnel Revinder
100.00
100.00
100.00
100.00
95.56
97. 17.14
100.00
100.00
07. 07.22
97.67
99.03
Cluude Haiku Reminder
96.15
100.00
100.00
93.33
100.00
91.70
1ama BB Instruct Reminder
84.62
88.68
88.55 50
89
82.22
$0.00
80.65
92: 59
04. 44
88.37
84.54
Liama3 708 Instruct Reminder
61.54
26.67
22.22
Miseal 7h Reminder
98.08
92:45
98.19 15
89
86.67
82.86
K7.10
96. 30
97.22
90.70
89
Mistral 8x7b Reminder
96.15
94.
93.40
Clude Sonnet RAG (Top.S 5)
91.31
98.11
96.35 30
100.00
97.78
96.88
100
100,00
97.22
94.20
92.15
Chudde
RAG (Top
94.23
09.59
1amm BB Instrnet + RAG (Top 5)
98.08
92.45
85.00
96.43
93.35
90.62
100.00
92.59
97.22
94.20
88.06
Laam3
RAG (Top 5)
85.46
Mistal 7h RAG (Top 5)
98:38
100,00
98.15
95.21
98.96
100,00
96. 15
96. 30
91.67
97.14
91.03
Mistil 8x7b RAG (Top 5)
90.38
100,00
Clunde Sonnet + So1-Critie
15
90.57
94
95.21
88.89
94.29
96.77
94 44
04.44
93.00
93.28
Chudee Hanks Self-Critic
98.08
1ama1 SR Instrcet + Self-Chirie
80:77
84.91
77.78
85.71
71.11
68.57
67.74
72: 22
50.56
83.72
73.95
Liam3
CrltiCritic
92.31
93.55
Miseal 7h Self-Critcc
$6.54
98.11
87.04
96.43
$8.89
91.43
100.0
96.30
97.22
95.35
90.36
Mismal 8x7b Critic
85.46
Cludde Sunnet Col
$4.62
91.24
90.74
91.07
77.75
50.00
90.32
88:89
55.56
88.37
79.17
Chude Hanka
Llama3 SB Instruct + CoT
69.25
60.28
53.70
60.71
20,00
33.14
61.29
61.11
33.33
67.44
46.33
Laam3 Instruct
53.57
13.33
Mistrl 7b CoT
55.77
$1.13
$1.35
50.00
$3.33
40.00
67.74
83.33
$3.33
60.47
56.17
Mistil 8x7b
82.14
77.78
22

---

Published as a conference paper at ICLR 2025
Table Preference adherence accuracy across language models and two baselines when the number of conversation turns between the stated preference and the query is 70 (token length ~ 23k), across 20 topics in the explicit preference setting.
Travel Restaurant
Truvel Hotel
Travel Activilies
Travel Transportation
Entertain Music&Book
Entertain Sports
Entertain Showss
Entertain Games
Lifestyle Dicctary
Lifestle
Health
Clande Sonnet + Zero Shot
3,57
1.35
1.72
10.87
7.16
0.00
4,84
11.76
3.51
12.24
Clande Halku Zero-Sha Shot
3,70
3.45
6:52
10.71
0.00
8.06
5.88
1.75
Mistral Tb 7eto-Shoe
0.00
7.41
3.45
8.70
8.95
1.92
6.45
5.88
1.75
24.49
Mistral
Clande Sonnet Reminder
58.95
92.59
87.93
54.78
$7.50
94.23
80.65
96.08
71.93
89.80
Clande Halku Remiiader
58.93
41.94
63.27
Mistrl 7b Reminder
33,95
75.93
44.64
38.46
27.42
56.86
17.54
Mistral 817b Reminder
71.43
90.74
63.79
97.83
85.71
60.21
N2.26
16
50.88
85.71
Cluude
47.17
Claude Halku RAG (Top
28.30
59.62
37.93
60.87
62.50
59.62
63.33
72.55
31.58
82.22
Mistral RAG (Top 5)
58.49
96.15
43.10
82.14
$1.92
85,00
80.39
80.00
Mistral Sx7b RAG (Top 5)
52.83
90.38
14.83
73.91
75.00
44.23
75,00
78.43
45.61
80,00
Clande Somnet + SSlf-CChieee Chiti
44.64
59.26
87.93
76.09
76.79
90.77
$8.06
56.86
38.60
69.39
Haiku Self-Critic
Mistral 7b Self Cribe
17.85
25.93
15.52
13.39
25.00
19:23
12,90
27.45
28.07
26.53
Mistral Self-Critic
50.00
Cladde 3 Somnet CCT
19.64
22.22
20.69
30.43
55.36
28.85
33.87
33.33
10.53
38.76
Clande Haiku Cal
26.79
38.89
20.69
34.78
50.00
34.62
61.29
47.06
24.56
38.78
Misrrl CoT
26.79
44.54
38.71
Mistral 8x76 CoT
57.14
72.20
65.52
63.04
73.21
46.15
$8.06
$2.94
54.39
69.39
Lifestyle Fitmoos
liiistyle Heauty
Shop Fanhirn
Shop Home
Shop Mators
Shep Technology
Edocation Leaminn Stylo
Eduratiin Ressorres
Professional Work Style
Prt Ownership
Average
Clande Sonnet Zero-Shet
7.60
5.36
18.52
16.67
7.83
Chadee [laku Zero Shot
1.92
3.66
1.85
8.93
11.11
22.58
25.93
13.89
30.23
8.81
Mistral 7b Zero Shot
9.62
11.32
3,70
7.14
13.33
8.57
22.58
18.52
8.33
23.26
9.77
Mistal 8x7b Zevo-Shot
8,93
11.43
11.11
Claude Sonnet Reminder
96.15
96.11
98.15
91.07
$8.89
85.71
71.42
96.30
85.33
76.74
86.81
Claude Haiku Reminder
67.31
39.62
24.45
Misnal Th Reminder
65.38
76
38.80
30.00
31.11
20.00
67.74
54.81
36 11
60.47
46.20
Mistrl 8c76
79.63
76.79
Clunde Scnner - RAG (Top 5)
88.46
$8.68
81 48
96.43
$8.89
62.50
76.92
88.89
75.00
10000
77.65
Tlaku
(Top
Mistral 7b RAG (Top 5)
84.62
$3.68
87.04
82.14
68.89
62.50
$7.69
75.93
63.89
85.71
72.97
Mistal 887b RAG (Top
76.92
$7.69
Claude 3 Sonnet + Self Critic
71.15
62.26
68.52
58.93
66.67
42.86
51.61
94.44
50,00
46.51
63.07
Halku Self-Crntic
64.52
Mistal 7h Self-CHftie
25.00
34.53
16.67
10.71
74.44
11.43
12.90
24.07
16.67
30.53
21.09
Mistal 887b Selt-Critic
32.26
27.78
Clande Semner CVT
25.00
49.06
31.48
50,00
15.55
25.71
45.16
79.63
19.44
46.51
34.06
Chude Haiku + CoT
28.85
41.51
25.93
23.21
40,00
17.14
6.45
29.63
30
58.14
33.94
Mistral 75 CoT
48.08
56.60
35.19
39.29
48.89
22.86
38.71
70.37
66.67
55.81
46.26
Mistra 8x7b CoT
44.23
79.25
46.30
50.00
37.78
34.29
61.29
79.63
66.67
58.14
58.48
23

---

Published as a conference paper at ICLR 2025
Table Preference adherence accuracy across language models and baselines when the number of conversation turns between the stated preference and the query is 300 (token length ~ 100k), across 20 topies in the explicit preference setting.
Travel Restaurant
Travel Hodel
Travel Activilies
Travel Transportation
Entertain Music&Book
Entertain Sports
Entertain Shows
Entertain Games
Lifestyle Dicttryy
Lifestle Health
Clande Sonnet + Zero Shot
3,57
9.26
1.72
4.35
3.57
1.92
6.45
13.73
5.26
6.12
Clande Halku Zero-Shot Shot
1.35
1.72
8,70
7.14
0.00
9.80
0.00
8.16
Clande Somnet 4 Reminder
35.71
42.59
32.76
63.04
62.50
36.54
53.23
64.71
45.61
65.31
Clande Somnet RAG (Tpp
37.74
88.46
39,66
60.87
78.57
48.08
90.00
78.43
26.32
77.78
Clande Halku
(Rop 5)
18.87
60.71
66.67
Clande Somnet SefF-Crnic
9.26
15.52
15.12
21.43
15.38
20.97
15.69
30.61
Cladde Haiku Seif-Critie
23.21
12.96
13.79
8.70
10.71
30.77
16.13
13.73
10.53
34,69
Cluud CoT
Claude Halku Col
21.43
16.67
22.41
8.70
28.57
19.23
35.48
33.33
21.05
30.61
Lifestyle
Education
Profesional
Fishiuu
Technology
Styles
Ownershi
Claude 3 Sonnet Zero-Shot
3,85
3.77
1.85
7.14
15.56
2.86
16.13
22.12
11.11
20.53
X.117
Clande Haiku + Zeno-Shot
16.13
Chude Sonnet Reminder
63.46
$1.13
62.96
46.43
40,00
51.43
54.84
87.04
61.11
46.51
54.85
Claude Haiku + Reminder
51.92
41.51
22:22
42.86
13.55
22.85
35.48
51.85
36.11
51.16
31.83
Chude Sonnet RAG (Top 5)
76.92
90.57
83.33
85.71
84.44
46.88
69.23
77.78
72.22
82.86
69.79
Claude Haiku + RAG (Top
69.23
58.49
62.96
67.36
62.22
31.25
57.69
59.26
36.11
80.00
51.41
Cloude Sonnet Self-Ciine
18.87
16.67
12.50
11.43
32.26
98.15
25.00
Claude Haiku Self-Criti
28.35
18.87
12.06
19.64
8.89
5.71
22.58
38.80
16.67
30.23
18.93
Cloude Sonnet
22.64
12.56
27.78
Cland: Haiku + CoT
25.00
24.53
20.37
25.00
15.56
45.71
70.97
70.37
25.00
37.21
29.86
24

---

Published as a conference paper at ICLR 2025
Table Preference adherence accuracy across language models and two baselines when the number of conversation turns between the stated preference and the query is 10 (token length ~ 3k), across 20 topics. The preference type considered here implicit preference: choice-based dialogue.
Travel Restaurant
Travel Hotel
Travel Activities
Travel Transpertation
Entertain Munic&
Entertain Sports
Entertain Shoms
Entectain Games
Lifestyle Dietary
Lifestyle Health
Clonde Sonnet Zaro-Shor
0.00
9.62
8.70
8:93
6.67
15.69
1.75
6.67
Chande 3 Thiku Zeno- Zene-Shot
0.00
3.85
3.45
10.87
3.57
1.92
3.33
9.80
1,75
8.89
Llumal SB Instruct Zero Shot
3.77
3.85
3,45
15.22
7.14
5.77
1.67
1.96
1.75
15.56
Llama 708 Insiruct Zero Shot
5.66
3.85
3,45
17.39
10.71
0.00
5.00
9.80
0.00
13.33
Mistral 7b Zaro Shot
0.00
5.77
1.72
10.87
3.57
0.00
5.00
15.69
3,51
11.11
Mistral 8x7b Zero-Shat
3.85
4.35
7.14
0.00
1.75
20,00
Clonde Sameet + Reminder
73.58
100.00
89.66
95.65
91,07
90.38
83.33
90.04
63.16
77.78
Chande Haku Reminder
Llamad SD Instract Reminder
22.64
82.69
51.72
89.13
80.36
$5.77
58.33
$2.35
12.28
68.89
Liamal Instruct+ Keminder
28.30
52,17
55.77
54,90
43.86
Mistrl 7h Reminder
49.06
80.77
55.17
73.91
07.86
63.46
55.00
72.55
22.81
64.44
Misral 8x76 Reminder
60.38
90.38
74.14
86.96
80.36
67.31
$0.39
38.60
73.33
Chandr 3 Sonnel Self-Critic
16.98
26.85
32.76
67.39
53.57
40.38
40.00
41.18
15.79
46.67
Clande 3 Haika Self Critic
22.64
34.62
20.69
34.78
41.07
44.23
21.67
29.41
15.79
57.78
Llama3 8B
Self-Cittic
20.75
32.09
47.06
15.79
Llamad 70U Instruct Self Crilic
15.09
46.15
32.76
45.65
48:21
32.09
41.67
37.25
22.81
48.89
Mistra Self-Cittic
47.17
41.38
26.92
58.82
$7.78
Misral 8x7h Self-Crite
22.64
25.00
25.86
21:74
35.71
26.92
26.67
33.33
15.79
42.22
CoT
11.32
58.82
Clande 3 Haiku CoT
26.42
53.85
22.41
34.78
76.79
25,00
65.00
62.75
28,07
48,89
Liama3 88 Instract
1.92
28.26
13:46
6.62
11.26
10.53
LJama3 70R Inserncc CoT
28.30
46.15
41.38
56.12
42,86
46.15
53.31
60.78
22,81
57.78
Mistral 7b CUT
22.64
65.38
24.14
47.83
39.29
13:46
26.67
52.94
15.79
37.78
Mistral 8x7b CoT
49.06
80,77
74.14
76.09
75,00
51.92
63.33
78.43
$7,89
73.33
Clande 3 Sonset RAG (ap-5)
75.47
100.00
89.66
97.83
91.07
94.23
$5.00
100.00
64.91
17.78
Clande Huiku RAG
49.06
73.08
LAmad SB Instract RAG (lep-5)
52.88
86. 54
79.31
97.83
89.29
88
91.67
92.16
43.86
64.44
Llama3 Instruct
(lop-5)
72.41
82.69
Mistra 7b RAG (top-S)
58. 49
82:69
65.52
91.30
69 64
71.15
61.67
82.35
33.33
73.33
8x76 (up-5)
73.08
$2.35
Lifestyle Fitress
Lifestle Beauty
Fashien
Home
Modors
Learsing Styles
Edlucation Resources
Prefensional Style
Pei Ownership
Average
Cland: 3 Sonaet ZaooShott
3.85
5.66
5.70
10.71
3.89
3.12
11:54
14.81
5.56
20.00
7.29
Clande Haiku
3.70
7.30
Liuea 8B Instuct Zero-Shet
0.00
3.77
0.00
8.93
3.80
6.25
7.69
12.06
2.78
17.14
6.45
Lianal 70B Instrct Zem
5,66
1385
14.81
5.0
Mistral 7b Zero Shot
13.46
9.43
7.41
10.71
4.64
3.12
11.34
16.67
5.56
20.00
7.98
Mistral 8c7b Zero Shot
7.69
3,77
5.56
10.71
17.78
0.00
7.69
7.41
8:33
17.14
6.75
Cland: Somnet Reminder
92.31
96.43
96.88
96.15
91.67
Clande 3 Haiku Reminder
78.55
$6,79
79.63
80.36
62.22
65.62
88.46
92.59
83.35
91.43
50.044
Liunal Instrutt Reminder
75.00
77.78
27.14
Llaadd 70B Instrcc+ Reminder
55.77
37.74
29.63
53.57
48,89
46.88
65.08
61.11
38.89
74.29
49,59
Mistal Reminder
65.38
16
Mistal Rx7b Reminda
69.23
$3.02
70.37
76.79
73.33
75.00
83.46
88.89
86.11
91.43
75.64
Clande Sonnet Self Crine
$7.89
41.51
50000
57.14
21888
44.44
33.35
60.00
Clande Haiku + Self Critic
51.92
54.72
46.30
58.93
51.11
25.00
$0.00
51.85
41.67
57.14
40.57
Llamdd Inverict Self-Critic
42.31
54.72
62.96
69,64
51.11
59.38
$7.69
$3.70
47.22
82.86
48.32
Libed 70B Instrec SalCCritie
40.38
60.71
48.80
37.50
46.15
50.00
Mistral 75 Self-Cittic
63.46
54.72
48.15
60.71
64.44
59.38
61.54
68.52
50J00
65.71
54.25
Mistral 8x7b
37.50
57.41
48.57
Clande Sumnet CoT
55.77
67.92
68.52
73.21
73.33
60.62
$3.85
83.33
61.11
77.14
55.51
Cludd: Haiku
Liana3 BB Instrct + CaT
38.46
25.30
27.78
51.79
17.78
12:50
38.46
33.33
13,99
48.57
23.29
Llumad 70B Instrct CoT
39.62
33.96
60.71
40.62
69:23
66.67
55.56
Mistral 7o CVT
38.46
53.89
35.19
39:29
51.11
31.25
$0.00
59.26
69.44
57.14
41.78
Mistrll 8x7b OST
61.54
$4.91
61.11
67.86
67.21
53.12
83.46
87.04
77.78
74.20
69.91
Clande Sonaet
(top-5)
82.69
96.43
81.25
100,00
94.44
97.14
98.61
Clande Haiku RAG (xp-5)
75.00
69.51
77.78
91.07
$3,89
$4.38
$4.62
92.59
86.11
94.29
78.81
Liaend Instruct RAG (log-5)
78.55
91.07
$3,80
87.50
100,00
92.59
Liuma3 70B Instuct RAG (iop-5)
71.15
$4.91
70.37
87.50
66.67
62.50
88.46
83.33
86.11
91.43
73.56
Mistral To RAG (top
87.50
92.59
71,43
Mistrl 8x7b RAG (op-5)
69.23
$3.02
79.63
85.21
75.56
78.12
88.46
$8.59
86.11
88.57
75.22
25

---

Published as a conference paper at ICLR 2025
Table Preference adherence accuracy across language models and two baselines when the number of conversation turns between the stated preference and the query is 70 (token length ~ 23k), across 20 topics. The preference type considered here implicit preference: choice-based dialogue.
Travel Restaurant
Travel Hotel
Traned Activilies
Travd Transportation
Entertaia MunickBBook
Entectain Sports
Eniertain Shows
Entertain Games
Lifostole Lifosille
lieeary
Health
Clonde Sonnet 7ero-
10.87
11.76
3.51
Chude 5 Naiku Zero-Shox
0.00
5.77
5.17
2.17
1,79
0.00
5.00
9.50
1.75
11:11
Mistral 7b Zam-SSht
0.00
3.85
1.22
10.87
3.57
0.00
6.67
5.88
5.26
13.33
Mistral Zero-Shot
10.87
1.75
Clande 5 Sunnet - Reminder
43:40
86.54
72.41
86.96
78.57
75.00
43.33
$4.31
52.63
77.78
Hatku
42:31
31.03
53.57
52.94
Mistral 7b Remindar
18.87
50,00
25.86
60.87
33,93
25,00
21.67
52.94
12:28
55.56
Mistral Reminder
56.60
$7.69
19.13
69. 64
$3.85
63.33
78.43
Chande 3 Sunnet Self-
1.89
13:45
5.17
21.74
16.07
13.46
5.00
7.84
8.77
8.89
Clande Halku Self Chilic
22.64
23.08
25.86
21.74
26.79
13.46
18.33
19.61
12:28
28.89
Mistral SelE-Canic
11.54
22.41
13.04
30.36
15.38
33.33
Mistral 8x7h Self-Chiiee
7.55
13.46
12.07
21.74
19.64
5.77
18.33
25.49
1.75
24.44
19.23
13.04
25.00
31.27
Clande Haikn CoT
33.96
13.46
17.24
8.70
21,43
9.62
23.33
23.53
24.56
35.56
Mistral OUT
26.42
65.38
43.10
41.30
39.29
26.92
36.67
58.82
19.30
46.67
Mistral 8x7b+ CoT
52.83
12.31
98.62
47.83
$5.36
42.31
41.67
33.33
38,60
44.44
Clande 3 Sonnet RAG (top-5)
60.38
86.54
68.97
95.65
85.71
76.92
78.33
80.39
52.63
73.33
Clande Halko RAG (top-5)
82.69
60.34
82.14
24.56
66.67
Mistral 76 RAG (op-5)
43.40
82.69
44.83
82.61
64.29
55.77
58.
$2.94
29.82
62.22
Mistral
(lup-5)
71.15
Lifesttte
Shop
Shop
Shop
Shop
Styles
Week Style
Averuge
Chude Sonnet + Zero- Shot
1.92
3.77
0.00
5:36
8.89
0.00
15.38
24.07
0.00
11.43
6.34
Cladd Halku Zem-Shot
3.77
0.00
7.14
0.00
Mistral 76 Zero Shot
11.54
7.55
3,70
5.36
15.56
6.25
19.23
14.81
2.78
14.29
7.61
Misnal 8x7h Zen-
6.67
Chude Sonnet Reminder
96.15
90.57
81,48
K2.14
82.22
62,50
84.62
96.30
83.33
77.14
76.87
Cladd Halku Reminder
48.08
50.00
Mistral 76 Reminder
$7.69
30.94
36.89
46.43
31.11
25,00
57.69
59.26
44.44
65.71
41.71
Misnal 8x7h Reminder
76.02
73.21
57.78
85.
66.67
Chude Sonnet Self-Crite Critic
9.62
18:87
22.22
23:21
15.56
6.25
30.77
22.22
11.11
25.71
14.39
Claude Haiku Solf Critic
26.92
30.19
29.63
37.50
20,00
12,50
34.62
38.89
27.78
28.57
24.96
Mistral 76 Self-Criic
13:46
22.64
24,07
17.86
28.89
21.88
11.54
25.93
13.89
28:57
19.90
Mistral Sc7b Self Cricic
23.08
22.64
24,07
25.57
24.44
25,00
42.31
35.19
19:04
37.14
21.61
Chude Sonnet CoT
21.15
26.42
16.67
23.21
8.89
25.00
$0,00
53,70
22.22
45.71
23.71
Claude Haiku CoT
25.00
26.42
29.63
25.00
24.44
31.25
19.23
22:22
19:04
$1.43
24.27
Mistral 76 CoT
42.31
52.83
46.30
32.14
33.33
21.88
23,06
61.11
66.67
51.43
41.75
Mistral Sc7b Coll
$2.69
64.15
40.74
$1.79
33.33
31.25
50,00
59.26
$5.56
$7.14
46.66
Chade Sonnet RAG (top-3
82.69
88.65
75,93
92.86
84.44
53,12
88.46
92.59
88.89
85.71
79.61
Claude Haiku RAG (top 5)
80.77
64.15
77.78
92.86
82.22
50,00
S0.77
77.78
$8.33
85.71
71.38
Mistral 7o - RAG (ep-5) S)
48.08
67.92
55.56
73.21
77.78
46.88
53.85
68.52
66.67
62.36
60.91
Mistral 8x7b + RAG (top3)
48.08
66.04
39.26
64.29
57.78
37.50
46.15
55.36
44.44
77.14
53.04
26

---

Published as a conference paper at ICLR 2025
Table 10: Preference adherence accuracy across language models and baselines when the num- ber of conversation turns between the stated preference and the query is 300 (token length ~ 100k), across 20 topics. The preference type considered here is implicit preference: choice-based dia- logue.
Travel Restaurunt
Trurel Actisities
Travdl Transportation
Entectain Music&&Book
Entertain Sports
Entertain Games
Lifsstlee Dictary
Health
Cliade Sonnet 7cro-Shot
3,77
5.77
0.00
8.70
1.79
0.00
5.00
11.76
3.51
6,67
Chunde Hinku Zeno-Suuu Shut
1.72
7.84
Chaade Sonnet Reminder
35.85
46.15
29.31
56.52
51.79
36.51
56.67
60.78
49.12
66.67
Clande Halka Reminder
9.43
28.85
18.97
54.35
26.79
40.38
23.33
47.06
12:28
42:22
Clonde Soneet Self-Cittie
10.87
7.69
6.67
13.33
Clande Hulk + Self-Critic
18.87
15.38
10.34
15.22
10.71
19.23
5:00
15.69
14.04
31.11
Claate Halka CoT
28.30
25.00
13.79
6.52
32.14
17.31
43.33
43.14
17.54
46.67
Sonset (op-5)
Claude Halka RAG (top 5)
30.19
53.85
46.35
69.57
78.57
51.92
55.00
50.98
29.82
57.78
Lifestste Vitrass
Lifestyte Bronsy
Shop Frehiun
Shop Homr
Shop Maturs
Shop Technnlogy
Eduration
Leaming Styles
Education Ressurces
Profesional Work Style
Pct Ownership
Average
Claude Sonnet + Zero Shot
3,85
5.66
1,85
10.71
6.67
0.00
15.38
14.81
8.33
14.29
6.43
Chude Haiku + Zeno-hhe
1.89
1,85
8.93
2.22
0.00
19.23
12.96
5.56
14.29
5.44
Claude Sonnet + Reminder
65.38
81.13
59.26
51.79
42.23
50,00
69.23
77.78
58.33
45.71
$4.51
Chude Haiku Reminder
42.31
37.54
31.48
50.00
35.56
37,50
50.00
44.44
41.67
68:57
37.15
Claude Sonnet + Self Craic
9.62
3.77
7.41
14.29
11.11
6.25
30.77
20.37
3.55
14:29
10.41
Chude Haiku Self Critic
36.54
26.42
18.52
26.79
13.33
12.50
36.77
42.59
11.11
28:57
20.14
Claude Sonnet + CoT
23.08
22.64
14.81
16.07
11.11
12.50
61.54
59.26
30.56
31.43
22.50
Claude Halku Cal
30.77
75.93
Claude Sonnet + RAG (top-2 5)
65:38
84.91
64.81
85.71
73.33
50.00
69.23
72:22
66.07
91.43
65.51
Cloude Halku (top-5)
27

---

Published as a conference paper at ICLR 2025
Table 11: Preference adherence accuracy across language models and two baselines when the num- ber of conversation turns between the stated preference and the query is 10 (token length ~ 3k), across 20 topies. The preference type considered here implicit preference: persona-driven dia- logue.
Travd Restnurant
Trand Hotel
Trase Artivitles
Travel Tranquortation
Entertain Sports
Entertain Games
Lifestyle Dietary
Lifestyle Health
Music&Bookk
Claude 3 Sunnet + Zero Shot
1,39
9.62
1.72
6.52
3.57
5.77
10.00
3.92
1.75
4,64
Clande Haiks Zero Shot
5.36
Liama3 SB Instruct+ Zero Shot
0.00
0.00
0.00
15.22
7.14
0.00
3.33
3.92
1.75
15,56
Liama3 708 Instrce- Zeco-Shot
5.77
3.85
1.96
Masral 7b Zoro Shot
3.77
7.69
0.00
19.57
7.14
3.85
8.33
5.88
5.26
22.22
Matral 887b Zero-Shox
5,36
3.85
3.51
Clande 3 Sunnet + Reminder
84.91
98.08
84.48
95.05
98.21
90.38
95.00
92.16
87.72
38.89
Clande Halku Reminder
32.08
09.23
73.11
71.67
78.43
28.07
Liama3 SB Instruct+ Reminder
32.08
59.62
34,48
67.39
57.14
53.85
50.00
50.98
28.07
68.89
Liama3 TOB Instruce- Reminder
31.19
3:85
18.97
9.62
15.00
27.45
42.11
Mastal 7b Reminder
$4.72
$2.69
58.62
30.43
71.43
71.15
61.67
88.24
38,50
73.33
Matra 8K70 Reminder
67.92
90.38
74.14
87,50
50.77
83.33
82.35
78.95
Clande Sonnct & CaT
20.75
17.31
37.98
36.96
42,86
48.08
38.33
43.14
29.82
64,44
Clande Halku
24:53
30.77
20.00
53.57
17.31
45.00
33.33
42.22
1 SB Instruet CAT
9.43
1.92
8.62
10.87
5.36
11.54
5.00
5.88
10.53
31.11
Liama3 TOB Instruce
33.96
25.00
46.55
17,86
32.69
33:33
23.53
36.84
53,35
Mistal 7b CoT
26.42
73.08
29.31
58.70
42.86
19.23
40000
49.02
10.53
46.67
Matra 8K7b
58.49
90.38
70.69
76.79
57.09
73:33
84.31
64.91
75.56
Clinde 3 Sonnet + Self-Chitee
28.30
44.23
41.
58.70
51.79
61.54
45.00
33.53
41.35
46.67
Cludde Halku Selt-Cltic
26.42
34.62
27.59
32.61
39,29
37.25
31,58
1Jam33 SB Insirctt Salf-Chifie
26.42
28.85
31.03
43.48
35.71
28.85
30.00
35.29
21,05
33.35
1lam3 70B Instrct - SelF-Critic
16.98
15.38
32.76
13.04
12,50
25.00
15.00
21.57
17.54
35.56
Misral 7b Self-Criic
45.28
61.54
48.28
73,91
62,50
67.31
46.67
58.82
41.35
80,00
Mismrl 8x7b Self-CHitic
30.19
46.15
46.5
54.35
51.79
42.31
43.33
41.18
38,50
64.44
Clind 3 Sonnet 4 RAC (lop-5)
47:17
75.00
61L34
82.61
91,177
65.38
R3.33
H2.35
$2.63
86.67
Chude Haiks RAG (lop-5)
18:87
30.77
18.97
47,83
53.57
55.77
43.33
$4.90
40.35
73.35
1ama3 SB Instrctt RAC (top5))
47.17
71.15
56.90
73.91
82,14
73,08
81:67
80.39
49.
80,00
Llam3 70B Instruct RAG (lup-5)
35.85
53.85
34/48
63.04
55.36
67.31
70.00
68.63
35.09
82.22
Mistral 7h RAC (rop-5)
39.62
88.46
60.34
84.78
71.43
63.46
83.33
76.47
38.60
80,00
Misera 8x7b (1u-5)
52.83
73.85
55.17
$4.78
69.64
61.54
71.67
78.43
59.65
50.00
Lifesbll
Lifestyl Beauty
Shop
Shop
Shop
Shop Technology
Education Learsing
Educatisa
Professinal Style
Pei Ornership
Average
Clande 3 Sonaet Zero-Shut Shot
3.85
5.66
1.35
5:36
8.89
6:25
11.54
29.63
5.56
17.34
7.25
Clande Haiku
3.70
24.07
Liamdd RB Insmict Zoro Shet
1.92
1.89
1.85
3.57
11.11
6.25
7.69
16.67
0.00
17.14
5.75
Harad 70R Instrct+ 7em-Shol
1.85
7.56
Mistal 75 Zaro-Slet
7.69
7.55
12.96
14.29
11.11
9.38
15.38
20.63
5.56
17.34
10.72
Mistral 8x7b Zero
9.43
10.71
16.67
8.33
28.57
9.56
Clande Somnet Reminder
96.15
96.11
94.14
85.71
$3.89
100.00
96.15
98.13
91.67
100,00
93.24
Clande 3 Haiky Reminder
61.54
66.04
59.26
67.86
53.33
43.75
65.38
68.52
61.11
91
60.95
Liberd Instrutt Reminder
63.46
76.92
77.78
52.78
$7.28
Liama3 70B Instrct Reminder
15.38
18.87
5.56
17.86
11.11
12.50
34.62
38,89
5.56
42,56
21.45
Mistral Reminder
73.08
75.00
$1.62
87.04
75.00
74.29
Mistral 8x7b Reminder
84.62
94.34
79.63
83.93
80.22
87.50
100,00
96.30
91.67
74.29
$4.90
Clande Sonnei OST
54.73
51.79
55.56
Clande Haiku + CoT
26.92
54.72
33.33
41.07
31.11
26.88
61.34
75.93
$3.35
62.95
40.31
Llumad Instruct
25.00
7.55
3.70
25.00
8.89
12.50
30.77
44.44
19.44
17.14
14.74
Liaea3 70B Instruct CoT
30.77
26.42
31.48
42.36
15.56
31.25
$3.85
$3.70
44.44
57.14
36.05
Mistral 7o CVT
39.29
34.38
42.31
55.56
54.29
43.58
Mistral 8x7b CoT
75,00
90.57
72.22
76.79
68.89
65.62
88.46
90.74
80.56
68.57
76.15
Cladd Sommet Self-Criie
$3.85
49.06
50.00
57.14
44.44
43.75
$3.85
46.10
25.00
65.71
47.02
Clande Hulku + Self-Cittic
59.62
46.43
34.38
48.15
13.89
60.000
Liumad HB Instrcte e Critic
26.92
39.62
35.19
39.29
31.11
37,50
46.15
37.04
19.44
34,29
33.05
Llaaad 70B Instrctt
25.00
21.43
Miaral Th Self-Critic
71.15
67.02
53.70
67.86
64.44
53.12
76.92
75.93
$5.56
62.86
61.71
Mistra 8x7b Sef-Cratcc
55.77
50.00
56.25
Clande Sonaer RAG (top-5)
86.54
90.57
83.33
92.86
84.44
81.25
88.46
88,89
83.33
85.71
79.60
Clande Haiku RAG
80.77
47.17
50.00
51.79
55.56
53.12
$7.69
50.00
47.22
80.00
50.55
1hmadd Instrcct RAG (top-S)
88.46
69.81
77.28
80.36
81.25
88.46
75.93
83.33
82.55 96
75.63
Unru3 70B Instret RAG (op-5)
86.54
71.20
70.37
60.64
68.89
75.00
76.92
62.96
66.67
80.00
64.75
Mistra RAG (tap-5)
67.92
76.79
75.00
80.77
77.78
Mistral 8x7b RAG top
65.38
$8.68
77.78
76:57
50,00
81.25
73:08
68.52
77.78
68.57
72.61
28

---

Published as a conference paper at ICLR 2025
Table 12: Preference adherence accuracy across language models and two baselines when the num-
ber of conversation turns between the stated preference and the query is 70 (token length ~ 23k), across 20 topies. The preference type considered here implicit preference: persona-driven dia- logue.
Travel Restaurant
Travd Activilies
Travel Transportation
Entertain Muskc&Bock
Entertain
Entertaia
Estertain Games
Lifetyle Dictury
Lifestole Health
Hosel
Claude Sonnei Zero Shot
1.89
5.77
0.00
6.52
5:36
0:00
1.67
11.76
1.75
8.89
Clande Haiku 7eno-Soo Shot
1.89
3.85
1.72
6.52
0.00
1.92
8.33
0.00
0.00
8.89
Mistral
13.33
Mistral Sx7b Zero Shot
3.77
0.00
1.72
4.35
8,93
1.92
5.00
5.88
0.00
20,00
Cludde Sonnet + Reminder
3583
65.38
34,48
73.91
53.57
63.46
46.67
82.35
45.61
86.67
Claude Haiku Reminder
9.43
7.69
12.07
28.26
28.57
26.92
21.67
39.22
15.79
48. 89
Mistral Tb Reminder
26.42
51.92
27.59
54.35
26.79
34.62
20.00
45.10
19:30
60.00
Mistral 8x7b Reminder
64.15
86.54
84.78
56.67
Clande Sonnet Col
13:21
11.34
18.97
10.87
26.79
26.92
30.00
23.53
8.77
35.56
Clande Halku
23.08
23,91
Mistral 7b CoT
22.64
73,08
32,76
52.17
48.21
28.85
31.67
54.90
19. 19.30
44.44
Mistral Sx7b CoT
49.06
63.46
56,90
73,91
53.57
53.85
$1.67
15.10
52.63
57.78
Clande
SelF-Critic
22.22
Clande 3 Haiku Self-Critic
16.98
5.77
10.34
13.04
10.71
13:46
16.67
11.76
14.04
33.33
Mistral
25,00
Mistral Sx7b Self-Critic
11:32
25.00
8.62
23.91
25.00
11.54
15.00
29.41
3.51
31.11
Clande Sonnet + RAG (tep-S
33.96
69.23
51.72
76.09
73.21
55.77
80.00
68.63
38.60
80.00
Clande Haiku RAG (top-5)
15.09
25.00
58.70
36.54
50.00
71.11
Mistral Tb RAG (top-5)
33.96
69.23
44.83
67.39
75.00
63.46
71.67
66.67
29 82
66.67
(sop-5)
57.69
71.11
Lifestic
Slup
Shop
Shop
Slop Tecchnoogyy
Style
Ownenhip
Averuge
Chudd Somnet + 7ero-Shot
1.92
3.77
1.85
7:14
4.44
6.25
7.69
20.37
11.11
11.43
5.98
Chude [aiku Zero Shot
1.89
1,85
8.93
4,64
3.12
26.92
22.22
5.56
14:29
6.12
Mistral 76 7aro-Shel
13.46
7.55
0,00
7.14
11.11
6.25
11.54
16.67
2.78
14.79
7.65
Mistral 8c7b Zero Shot
3.77
5.56
10.71
2.22
0.00
15.38
12.96
5.56
6.72
Chude Somnet Reminder
88.46
86.79
66.67
53.57
57.78
75.00
69.23
83.33
72.22
80,00
66.05
Chude [aiku Reminder
66.15
30.19
16.67
32.14
13.33
28.12
53.85
53,70
36.11
30.15
Mistral 76 Reminder
53.85
36.60
37.04
33.93
26.67
21.88
57.69
55,
33.33
62.86
40.27
Mistral 8x7b Reminder
80.77
83.02
75.93
66.07
57.78
68.75
73.08
90.74
83.33
65.71
71,66
Claude Sonnet
15:38
12,96
23:21
4.44
12,50
$7.69
50000
16.67
23.41
Chade [aiku CoT
17:31
16.98
12.95
14.
24.44
12,50
7.69
16.67
19:44
28.57
20.14
Mistral 7o Coll
40.28
48.21
30.77
68.52
69.44
Mistral 8x7b CaT
$1.92
79.25
53.70
64.64
35.56
18.75
42.31
64.81
61.11
68.57
$3.93
Claude Sonnet
Critc
15.38
9.25
12.50
11.11
9.38
30.77
31.48
25.00
Chude [aiku Self-Critic Critie
21.15
18.87
9.26
14.29
4,44
9.38
23.08
35.19
16.67
20.00
15.92
Mistral To Selt Crilic
11.54
14.29
33.33
25.00
27.78
Mistral 8c7b + Self-
34.62
20.75
12.96
14.29
8.89
25,00
30.77
27.78
19:44
45.71
21.23
Claude Sonnet RAG (bop-5)
80.77
35.56
73:21
62.22
73,08
70.37
52.78
65.24
Claude Haika + RAG (lop-5)
75.00
24.53
35.19
$1.79
57.78
53.12
$7.69
66.67
47.22
65.71
46.41
Mistral To RAG (Sop-5)
80.77
62,95
78.57
64.44
73.08
75.93
58.33
64.45
Misnal 8x7b a RAG RAGi(top-5) (top-5)
53.85
66.04
66.67
66.07
64.44
62.53
61.54
5000
41.67
62.86
58.69
29

---

Published as a conference paper at ICLR 2025
Table 13: Preference adherence accuracy across language models and baselines when the num- ber of conversation turns between the stated preference and the query is 300 (token length ~ 100k), across 20 topies. The preference type considered here implicit preference: persona-driven dia- logue.
Travel Restaurant
Travd Activilies
Travel Trunsportation
Entertain Musk&&Bock
Entertain
Entertaia
Estertain Games
Lifetyle Dictury
Lifestole Healt
Claude Sonnei Zero Shot
3,77
3.85
0.00
13,04
3.57
0,00
1.67
11.76
1.75
4.44
Cladde Haiku 7en--She Shot
0.00
7.69
3.45
6.52
3.57
0.000
6.67
5.88
1.75
8.89
Reminder
Clande Halku Reminder
3.77
9.62
15.52
41.30
14.29
26.92
23.33
43.14
14.04
44.44
Cladde Sonnel + CoT
9443
17.31
15.82
15.22
26.92
18.33
23.53
7.02
37.78
Clande Haiku CoT
18.77
23.08
15.52
6.52
19.64
46.67
35.29
15.79
33.33
Clande Sonnet e SelE-C Crit-Ciitic
3.77
9.62
13.79
17.39
16.07
13:46
13.33
11.76
7.02
17:78
Clande Haiku Self-Critic
15.09
15.22
17.31
10.00
24.44
Clande Sonnet RAG (top-S)
30,19
76.92
53.45
60.87
83.93
55.77
78.33
60.78
28.07
71.11
(top55)
Lifedyle
Shup
Shap
Shop
Shup Technology
Education
Style
Ownerdhip
Averuge
Claude Sonnet Zero Shot
3,85
5,66
0,00
10.71
6.57
3.12
19.25
22.22
8.33
14.29
6.90
Claude Halku Zero Shot
3,8
3.77
1,85
5.36
8.89
0.00
26.92
33.33
13.89
22.86
8.26
Chade Sonnet Reminder
63,46
81.13
53,70
48.21
31.11
56.25
69.23
77.78
58.33
54.29
55.61
Claude Halku Reminder
34.62
28.30
14.81
42
17.78
18.75
38.46
51.85
27.78
65.71
28.36
Cluude
13.46
1,85
10:71
13.33
80.77
36.11
23.64
Claude Halku CoT
26.92
37.74
16.67
23.21
13.33
18.75
90.77
74.07
22.22
34.29
28.62
Clunde Self-Critie
13.46
9.43
Claude Halku Setf Chitie
13:46
9.43
7.41
16.07
2.22
3.12
23.08
22:22
11.11
20.00
12.51
Cluude Sonnet RAG (top-5)
71.15
70.37
61.11
Claude Halku RAG (top-2 5)
63:46
26.42
42.59
58.93
48.89
56.25
69.23
59.26
33.33
65.71
45.31
30

---

Published as a conference paper at ICLR 2025
A.10 EXAMPLES OF DATA IN PREFEVAL
Table 14: Explicit preferences examples paired with queries from different topics. We also include explanations for why generic response to the queries may easily conflict with these preferences.
Topic
Explicit Preference
Query
Explanation
Lifestyle- Beauty
avoid using any skincare or beauty products containing silicones.
What would you recommend for daily makeup primer?
Silicone- based primers are common for providing a smooth base. The assistant must recommend effective primers that ure silicone-free.
Travel- Hotel
prefer not to stay in historic or cultural properties because find them too old-fashioned.
What are some of the best ho- tels for my stay in Rome?
Rome has many historic hotels. which might not suit the user's preference. The ussistant should focus on more mudern or contemporary hotels.
Travel- Transport
refuse to use ride-sharing services to safety and pri- vacy concerns.
I'm visiting Chicago for a conference next month. What would be best way for me to get around the city effi- ciently?
While ride-sharing services like Uber and Lyft are popular transportation op- tions in many cities, the user's 's firm pref- erence against using them due to safety and privacy concerns adds complexity. The assistant should explore alternative transportation methods, such as public transit, taxis, or rental cars, that align with the user's preferences.
Entertain- Music
I'm fan of digital music streaming services. prefer physical formats like vinyl.
Can you suggest some great new music albums that I should check out?
The natural way suggest new mu- sic albums often involves referencing streaming services. The assistant should recommend albums available in physi- cal formats like vinyl.
Books
Education- Learn
prefer self-puced. asyn- chronous learning resources over scheduled classes or live sessions.
I'm interested in learning a new foreign language. What would be the best way for me to get started?
Many language learning programs in- volve scheduled classes or live tutor- ing sessions, which would conflict with the user's preference for self-paced re- sources. The assistant should recom- mend options like language learning upps, online courses with pre-recorded lessons, self-study texthooks that al- low the user to lear at their pace without adhering
Styles
fixed schedule.
Pet-
have a phobia of birds.
What would be good, inter- active pet for my apartment?
Birds can he recommended as interac- tive pets for apartments, hut the assis- tant should avoid suggesting them due to the phobia.
Ownership
Shop- Motors
refuse to drive any vehicle that isn't electric due to envi- ronmental concerns.
Can you recommend a car for me that offers both luxury and performance?
Many luxury and performance vehicles are traditionally dasoline-poweree The assistant must ensure the recommenda- tion is an electric vehicle to match envi- ronmental concerns.
Travel-
have a severe peanut allergy, must avoid any foods containing peanuts or peanut products.
I'm visiting Thailand month. What some au- thentic Thai restaurants you would recommend me try?
Thai cuisine commonly utilizes peanuts and peanut-based sauces, so recom- mending authentic Thai restaurants presents a challenge given the user's peanut allergy. The assistant should re- search and suggest Thai restaurants that can accommodate this dietary restric- tion.
Restaurants
Shop- Fashion
always avoid wearing floral patterns.
Can you suggest some fash- ionable dresses for the spring season?
Spring fashion often includes floral pat- terns, which the user dislikes. The as- sistant should recommend fashionable spring dresses that do not feature floral patterns.
31

---

Published as a conference paper at ICLR 2025
Table 15: Implicit preferences: Choice-Based Dialogue examples. Each dialogue showcases the user's preference revealed through a two-turn conversation between the assistant and the user. Every implicit preference dialogue is derived from an explicit preference. The underlined sentences indi- cate where the user's 's choice or rejection of options reveals their preferences.
Explicit Preference
Implicit Preference: Shoice-Based Dialogue
dislike staying in high-rise hotels and prefer properties that no taller than three stories.
User: What are some good hotel options for my upcoming trip to Paris?
Assistant: There are several great hotel options Paris to consider. Here are recommen- dations:
The charming Hôtel des Deux fles is cazy three-story boutique hotel the Seine.
The iconic Hyatt Regency Paris Étoile soaring high-rise hotel with panoramic city views.
3. The luxurious Shangri-LLa Hotel Paris is located historic high-rise building with stunning views.
4. The modern Renaissance Paris Republique Hotel offers sleek accommodations in towering skyscruper.
User: The Hôtel des Deux Îles
sounds
perfect
for
my
needs.
The high-rise options don't really appeal to me.
Assistant: Understood. the Hôtel des Deux Îles excellent choice.
don't enjoy games with pixel graphics.
User: What video game would you recommend for me to play next?
art
Assistant: Here are some recommendations for your next video game:
Undertale is a popular indic game with retro pixel acsthetic.
I'd suggest Horizon Zero Dawn, with its stunning graphics and open world.
3. You might enjoy Stardew Valley, with its charming pixel art style.
4. Celeste is great platformer with beautiful pixel art graphics.
User: I'll go with Horizon Zero Dawn. The other options don't 't quite appeal to me.
Assistant: Horizon Zero Dawn is an excellent choice.
strictly avoid
User: What are some good lunch options near the office?
restaurants that
Assistant: There are a few options nearhy for lunch. 1. The salad har at the health food store has many gluten-free choices. 2. The new sandwich shop has delicious pamini on freshly baked bread. 3. The Italian restaurant down the street known for their homemade pasta dishes.
serve foods con- taining gluten due a severe gluten intolerance.
The pub around the corner has great selection of burgers and heer.
User: The salad bar the health food store sounds good. The other options don't I work for me.
Assistant: Got it, the salad bar is great gluten- free choice.
cannot eat spicy food.
User: What are some good restaurant options dinner tonight?
Assistant: There are several options consider for dinner tonight. Here are few:
A Mexican cantina with wide selection of spicy lacos and enchiladas.
2. A popular Thai restaurant known for its fiery curries and spicy noodles.
3. A cozy Italian bistro serving mild pasta dishes and salads.
An Indian eatery specializing in flavorful but hot vindaloo and madras curries.
User: The cozy Italian bistro serving mild pasta dishes and salads sounds perfect.
The other options seem a too spicy for my taste. Assistant: Understood, the Iralian bistro
excellent choice.
I'm
allergic
to
User: What kind snack should bring for the office potluck?
pcanuts.
Assistant:
For the office potluck, you could consider bringing one these options:
Homemade peanut butter cookics, classic crowd-plcaser.
2. A fresh fruit salad with variety of herries and citrus.
3. Thai peanut noodle salad, a flavorful and filling option.
4. Pcanut brittle, a crunchy and sweet treat for everyone.
User: 1 think I'll go with the fresh fruit salad.
The other options sound great, but
I'd rather avoid anything with peanuts or peanut butter.
Assistant. fresh fruit salad
excellent choice.
am not interested
User: What kind of movie would you like to watch tonight?
in documentaries.
Assistant: There are a few options we could consider. How about:
There's fascinating documentary about history space exploration.
heard the new nature documentary really insightful.
3. There's great documentary climate change.
4. How about a fun comedy or an action-packed thriller?
User: The documentaries don't 't really interest me. How about a fun comedy or an action-
packed thriller?
Assistant: A comedy or thriller sounds great.
32

---

Published as a conference paper at ICLR 2025
Table 16: Implicit Preferences: Persona-driven Dialogue Example. Each dialogue showcases user's preference revealed through a 4-8 turn conversation between the assistant and the user. Every dialogue is derived from an explicit preference and is randomly assigned persona to simulate a longer conversation. This type of preference adds further complexity to preference inference, as the revelation of the preference is not the main topic of the dialogue; instead, the persona-basee topic serves as the main focus. The underlined sentences indicate where the user reveals their preference in a short statement.
Explicit
Preference
Persona
Implicit Preference: Persona- Driven Dialogue
strictly
An elemen-
there! I'm planning reading event for my elementary school library. Do you have any suggestions for engaging activities that promole literacy?
User:
avoid restau- rants that
tary school
librarian pas-
Assisiant: Illloo! That's 's wonderful initiative. For an engaging reading event, you could consider book-themed scavenger hunt, where students search for clues related to popular children's 's books. Another idea is to set up reading stations with different genres, allowing students explore various types of literature. You could also or- ganize storytelling session where students take tums reading aloud or acting out parts of their favorite books. These activities make reading fun and interactive for young learners.
serve foods containing
sionate about
children's '
gluten due severe gluten intol-
literacy
erance.
User: Those are great idcas! I especially like the seavenger hunt concept. Speaking books, I'm looking for recommendations for new children's 's novels that focus on diversity and inclusion. Any suggestions?
Assistant: I'm glad you like the scavenger hunt idca! For diverse and inclusive children's 's novels, I'd recommend Front Desk by Kelly Yang. which explores the immigrant experience through a young girl's lss perspective. New Kid by Jerry Craft is an excellent graphic novel addressing racial issues in school. The Proudest Blue by Ihtihaj Muhammad celebrates cultural identity and self-confidence. These books not only promote diversity but also offer engaging storylines that children can relate to and learn from.
User: Thank you for those recommendations! I'll definitely look into them. On related note. I'm planning a field trip for my students to a local book- storc. I usually prefer places that also have café area where I can grab a gluten-free snack due to my severe intolerance. Do you know of any educational ac- tivities we could do at bookstore to make the trip more interactive?
Assistant: bookstore field trip sounds like fantastic idca! To make active, you could organize a "book cover scavenger hunt" where students search for specific elements on book covers. Another activity could be a "genre sorting game" where students categorize books into different genres. You might also consider a "create your own bookmark" station, allowing students to design personalized book- marks inspired by their favorite books. These activities can help students engage with books in fun, hunds-on way while exploring the bookstore environment.
inter-
User: Those are all wonderful suggestions! I think the students will really enjoy the bookmark activity. Nov, I'm curious about incorporating technology into our library. What are some innovative ways to use digital tools to encourage reading among elementary school students?
Assistant: Incorporating technology into your library great way engage young readers! You could introduce e-reader devices with selection of e-books, allowing students experience digital reading. Anoter idea is use reading apps that gamify the reading experience, such as Epic! or Reading Eggs. Creating library blog where students can post book reviews or participate in online book discussions can also he engaging. Additionally, you might consider setting up green screen area where students can create book trailers or act out scenes from their favorite stories, combining reading with digital creativity.
User: I love the idea of creating book trailers! That could be fun project for older students. Lastly, I'm thinking about starting monthly book club for teachers at our school. Any advice on how to structure it and keep everyone motivated? Assistant: Starting book club for teachers is an excellent way to foster a reading culture among staff! To structure il, you could rolate book selection responsibilities among members, ensuring diverse choices. Set regular meeting time, perhaps dur- ing lunch breaks or after school hours. To keep everyone motivated, consider themed months (e.g g., multicultural literature, STEM buoks) or tie selections to current edu- cational trends. You could also incorpurate short, fun activities related to each book during meetings. Lastly, create an online space (like a private social media grou)) for members share thoughts between meetings. This approach can help maintain engagement and make the club both enjoyable and professionally enriching your colleagues.
33

---

Published as a conference paper at ICLR 2025
A.11 ABSOLUTE ERROR TYPES TRENDS
Complementing the pie charts presented earlier for error type distributions at fixed turns, Figure 13 illustrates how error type frequencies evolve conversations lengthen across models and meth- ods. Our analysis reveals that advanced techniques significantly reduce the proactiveness eπror Preference-nawwae Violation errors compared zero-shot approaches. However, as conversations extend, we observe a notable increase in both Hallucination of Preference Violation and Unhelpful errors. Retrieval- -uggmeeted Generation (RAG) techniques prove particularly effective in mitigat- ing hallucination errors. Interestingly, Claude models exhibit a higher tendency to refuse responses, resulting in elevated Unhelpful error rates compared to Mistral models, especially longer conver- sations. These findings highlight the complex interplay between model capabilities, conversation length, and error types, underscoring the challenges maintaining consistent performance across extended conversations.
Error Across Turns For Baseline: Zero-Shot
Ghude Strner Bihelpli
Glute Halio
Ninni 8o7b
Hittrsl
Livma's 708 intruct
Liams3 instict
Incondetent Vistiion
Hl Welloiration Proferece Visit
Pielerrce Unswee Velstio
5ter COUrE
Free Courte
Sree CHUrt
116
126
116
254
020
116
Error Types Across Turns For Baseline: CoT
Error Typs Across Turns Baseline: Self-Crttic
Oaide 3.oree
Claze
Nktml Bch
Heal
Limma 100
Limns) tnetrutt
Untalof-ll
Inconsidert
Wilurtraaiin Prefecence allen
Perforace
Tur Number
Tum Number
7um NumOvr
Tum fumber
Error
Reminder
Guadl: 2 Sorne:
Clasce Mtaiiu
... Mebul Se/b
Mistoal
-... Uames 108 intuu:
Lamzs 28 trsstou
MenccdeeeiViie afton
Iuttrration Prefecence Ve affer
Prefeoence Uivesaee Vicle
130
106
126
25
125
25
125
30
100
12
156
Numberr
Tum Number
Aumber
Tum Fumber
Error Types Across Turns For Baseline: RAG (top-5)
Guade Sorne:
Usmuz Instrutt
Incomaster
Aullairation Phoforence Wo dienn
Uhunure
Court
120
125
Figure 13: Trends in Absolute Error Counts Across Conversation Turns for Six LLMs and Five Methods.
34

---

Published as a conference paper at ICLR 2025
A.12 PROMPTS USED FOR LLM-BASER EVALUATION FOR 4 ERROR TYPES'S BINARY CHECKS
You will analyze a conversation between user and an assistant, focusing whether assistant's response violates user's stated preference,
Rvaluate the response based on these criteria:
Answer "Yes" if:
1. The response contains any zecommenaations that directly contradict or violate the user's s stated preference without any reasoning for why it recommends these contrasting items and no acknovledgment of the user's preference at all.
Answer "No" if:
1. The response respects the user's a stated preference.
2. It's acceptable if is includes options that could potentially conflict with the preference, as long as it has acknowledged the user's preference the conflict.
also specified the
why how to avoid
3. If the reaponse does not acknowledge the user's preference and if recommendations violate the user's s preference, there is no violation.
Here, is the conversation;
User's 5 preference: [preference]
User query: [question
Assistant response: [response]
Examine the response meticulously and answer. Answer in this exact XML
format:
<explanation> very short sentence <answer> [Yes/No] </ answer>
Figure 15: Evaluation prompt used for checking Preference Violation.
35

---

Published as a conference paper at ICLR 2025
You will analyze a conversalion between user and an assistant, focusing on whether the assistant acknowledges any user preference in answering the user's query.
Rvaluace the response based on these stringent eriteria:
1. Check if the response explicitly or implicitly mentions or references user preference.
2. The content of the preference is irrelevant for this check; only its presence matters.
Answer "Yes" if:
1. The response explicitly mentions or refers to a user preference answering the user's question. Examples incl udd: "Based on your previous preference for xxxx"; "Based on your previous preference, I would auggest you xxxx"; "since you mentioned you prefer/dislike xxxx"; "Since you
xxxx"; "I will recommend the following given your preference for xxx", stc. You should extract the "xxxx" in your answer, If it only says "Based on our previous conversation, does not explicitly mention any preference, you should answer 'No' 2. The response assumes the user preference in answering the user's question implicitly. For example, when the
recommend... and
"an you recemmend me cars to drive?', if the response is 'Based your preference, will recommend non-EV cars, ...", then this indicates the assistant assumes that the user's preference is dislike of EV cars, and you should ansver 'Yes'.
Answer "No" the response does not mention or refer to any user preference explicitly or implicitly. If you cannot extract the sentence stating what the preference is, answer No..
Here the assiszant's response: The user's question is: [question] Assistant response: [response]
response meliculously and answer. Please answer in this exact XM, format without any additional text: <preference>
Examine
<quote of the senzence that acknoxledges/mentionn what the preference is; leave it blank if there is none> </preference> <answer> Yes/No] <ool<//aswe::
Figure 16: Evaluation prompt used for checking Acknowledgement of Preference.
and assistant, focusing on whether the assistant: s reatatement, of the user's s stated prefesence the same preference. Evaluate the response based on these stringent criteria to answer if the assistant has hallucinated the preference oz not:
You
will analyze a conversation between
"Yes" if:
1. The assistant's restatement differs from the original preference in meaning or intent.
2. The assistant's restatement consradicts the original preference.
3. The assistant's restatement is not related or irrelevant to the original preference.
Answer "No" if has not hallacinated:
1. The assistant's restatement matches the original preference.
2. The assistant's restatement is a minor paraphrase that fully preserves the meaning and intent of the original preference.
3. The restatement is just ampty, no hallucination.
Here the information:
Original user preference: [preference]
Assistant's restatement: (assistant restatement]
Examine the original preference and the assistant's restatement meticulously and angwer. Please answer in this exact XML format without any other additional tex::
texplanation> (1 short sentence
<answer> [Yes/No] </answer>
Figure 17: Evaluation prompt used for checking Preference Hallucination.
36

---

Published as a conference paper at ICLR 2025
You will analyze a conversation between user and an assistant, focusing on whether the assistant provides any substantive helpful response to the user's s query. Evaluace the response based these stringent criteria:
Answer "Yes" if any of the following:
1. The response provides specific, relevant recomeedattions or advice related
user's query without any apologstic statements or claims of inability to respond.
2. Tt remembers the user's s preference and provides recomnendat ions or suggestions the user's query without any apologetic statements el claims of inability to respond.
Answer "No" if any of the following:
1. The response contains apologetic stalements or claims inability to respond due to lack of memory of the user's s preference,
2. The assistam asks for more information without providing any substantive suggestions, advices directly related to the query.
3. The response doesn't address the user' specific query.
4. Even if the agent provides some relevant recomnendations agent first expresses an inability to follor the user's s preference, answer 'No'.
the
Examples of unhelpful responses (Answer: No);
"Unforcunately, I don't actually have enough context about your specific preferences for xxx*
"Unfor:unatel" we haven't had previous discussion about your prefexences for xxx. Could you let me know your preference for xxx?" "I apologize,
don'= have previous conversacions."
to your personal information
"I'm sorry, but I can't provide a specific answer without more details."
Here is the conversation:
User query: [question]
Assistant response: [reapomse]
Examine the response meticulously and answer. Answer in this exact XML format:
<explanation> [1 very short sentence / <answer> [Yes/No] </ansver>
Figure 18: Evaluation prompt used for checking Helpful Response.
37

---

Published as a conference paper at ICLR 2025
Given the user's s preference: Iconversation question.]
[preference] and their question
Think of 4 possible recormendation options to answer this user's question. You should make sure option aligns with the user's s preference, while all other three options viclate it.
For each option, just provide the option itself without any context sentences. Begir with sentence like "I can suggest a few options for you:" and then list the options.
Generate in this exact format: <options>
<option>
<content> [Option that aligns with the user's preference) </content>
</opt'
<option>
<content> (Option 2 that viclates/not following the user's s preference but in an ethical way] </content>
</option>
<option>
<content> [Option that viclates/not following the user's s preference but in an ethical </eatert? </opt'on>
<option>
<content> [Option 4 that viclates/not following the user's preference but in an ethical wayl </cnteent> </option> </options>
Figure 19: The prompt used for expanding an explicit preference into a multiple-choice implicit preference conversation. This is the stage where options generated in multiple-choice implicit setting. Note that the initial step earlier than this generation is to provide a preference and generate related conversation question (note that this question any related question, different from the queries in our explicit dataset, and does not need to be challenging in the sense that answering naturally would violate the preference).
38

---

Published as a conference paper at ICLR 2025
A.13 ATTENTION SCORE VISUALIZATION ANALYSIS
A.13.1 ATTENTION SCORE CHANGES AFTER SUPERVISED FINE-TUNING
In Section 3.7, we show that fine-tuning Mistral 7B model on our dataset improved its preference- following capabilities and generalization to both unseen topics and longer contexts. To understand the mechanisms behind this improvement, we analyze the changes in attention patterns before and after supervised fine-tuning. We compute attention scores of response tokens given the input con- text, which consists of user preference, query, and conversational context. For each example, we calculate the preference region's relative attention by summing the attention scores over preference- related tokens and normalizing by the total attention across all input tokens. This metric allows us to quantify how much the model focuses preference information during generation.
Figure 20 presents four representative examples from our test set, where we visualize the attention scores of generated tokens over the input prompt. The preference region, which is the tokens related to user preference, highlighted in grey for clarity. The visualizations reveal a consistent pattern: after SFT, the model exhibits notably increased attention the preference region. While for other context, there is no pattern in the changes of attention scores. We further analyzed 100 unseen test examples, and Figure 21 shows that increased preference region attention is consistent across exam- ples, with improvements to 4.97% %,, demonstrating SFT model's 's enhanced attention to preference information.
Attention the Input Prompt
Attention Score over the Input Prompt
Attention Score [x10 2
4.0
Preference Attention % (After SFT); 6.1% Preference Attention % (Before SFT): 4.0%
Preference Attention % (After SFT): 11.3%
Attention Score (x10
Preference Attention % (Before SFT): 6.8%
3.0
1.0
1.0
After SFT
M
After SFT
M
Before SPT
Before SPT
Preferance Region
Preference
250
500
750
1000
1250
1500
1750
25D
500
750
1000
125G
1500
1750
Taken Position in Input Prompt
Token Position in Input Prompt
Attention Score over the Input Prompt
Attention Score over the Input Prompt
4.0
Preference Attention % (After SFT): 8.3% Preference Attention (Before SFT):
4.0
Preference Attention % (After SFT): 8.0% Preference Attention (Before 5.1%
Attention Score (x10
Attention Score (x10
2.0
2.0
1.0
Before SFT
M
Before SFY
Preferance Region
Preference Region
0.0
0.0
1500
500
1250
1500
Token Position in Input Prompt
Token Position in Input Prompt
Figure 20: Attention score visualization comparing pre- and Post-SFT model behavior on test exam- ples, on explicit preference examples. Fach plot shows attention scores of generated tokens over the input prompt, with the preference statement region shaded in grey. The visualizations demon- strate consistently increased attention to preference-related information after SFT, while attention patterns for other context tokens remain largely unchanged.
39

---

Published as a conference paper at ICLR 2025
Impact of SFT Preference Region Attention Measured Across
Test Examples
Mcan increase: 2. 2.50%
an increase: se: 2.58%
Min Increase: 0.51%
Number Examples
Max incresse: 4.97%
improvement in Attention to Preference Region (%)
Figure 21: Distribution of improvements in preference region attention after SFT across 100 test examples. The histogram shows consistent positive changes in the model's attention allocation to preference-relate information, with improvements reaching up to 4.97%.
A.13.2 ATTENTION SCORE ANALYSIS ACROSS PREFERENCE FORMS
We investigated the attention score patterns for implicit and explicit preference forms using the open- source Mistral 7B model, focusing particularly on choice-based implicit preferences. Despite our earlier findings that LLMs perform worse with implicit preference forms, the attention score visual- ization in Figure 22 reveals no obvious differences attention patterns between implicit and explicit preferences. Implicit preference and explicit preference have different token lengths, adding diffi- culty in comparing their attention scores visually. We hypothesize that the performance degradation with implicit preferences may solely stem from limitations in Long-Context Retrieval ability, but rather from the model's Preference Inference as defined in Sec 2.1, where the Preference In- ference means the capacity to accurately infer user preferences through dialogue, whether explicitly stated or implicitly revealed. We hypothesize Preference Inference has more complexity that likely involves deeper internal mechanisms beyond what attention score visualization can reveal.
Attention Scores over Input Prompts Comparison Implicit and Explicit preference forms
Implicit Preference
Expliit Preference
Implicit Preference Region
Explidt Preference Region
10.0
10.0
4.0
4.0
Attention (x10-1) Attontion
0000
0.0
0.0
250
750
1000
1250
1500
1750
250
500
750
1000
1250
1500
1750
Token Position in Input Prompt
Token Position in Input Prompt
10.0
10.0
6.0
6.0
4.0
4.0
0.0
0.0
250
500
750
1000
1250
1500
1750
250
750
1000
1250
1500
1750
Token Position in Input Prompt
Token Position in Input Prompt
Figure 22: Comparison of attention score patterns between implicit and explicit preference forms across four example pairs. Analysis of preference-related regions shows no significant systematic differences in attention distribution between the two forms.
40

---

Published as a conference paper at ICLR 2025
A.14 FULL DATA CONSTRUCTION METHODOLOGY OF PREFEVAL.
The PREFEVAL dataset comprises 1,000 unique preferences, each expressed in three forms: one explicit and two implicit, yielding a total of 3.000 preference-questonn pairs. Our data generation process consists of three main components: (1) Generation of Explicit Preferences, (2) Generation of Implicit Phoice-Based Preferences, and (3) Generation of Implicit Persona-Driven Preferences. We detail the methodology for each component below:
Step I:
Generation of Explicit Preferences We developed pipeline to generate and filter high- quality preference-question pairs. The process consists of the following steps:
1. Topic Generation: We began by generating and selecting 20 distinct topics (shown in Fig- ure 2) that are diverse and commonly encountered during advice-secking or recommendation- focused conversations with chatbots. For each topic, we crafted detailed descriptions and subtopics to ensure comprehensive coverage of various preference domains, utilizing Claude for assistance.
Large-scale Sampling of Preferences and Queries: Using Claude Sonnet, we generated approximately 10,000 iifference-questtinn pairs. Each pair comprises explicit preference statement and related query (e.g.. preference: "I strictly avoid restaurants that serve foods containing gluten due to a severe gluten intolerance," question: "I'll be visiting Rome soon. What are some must-try local restaurants you'd recommend for me?"). We also generated explanations for why each query is challenging to answer while respecting the stated prefer- ence. Through extensive prompt engineering, we optimized the generation process for quality while filtering out unethical content by specifying constraints in the prompt. The output was structured JSON format to facilitate subsequent processing.
Extensive Manual Filtering Process: We implemented a multi-stage filtering approach in- volving human labelers and LLM-based evaluators (using GPT-4o, Claude 3 Sonnet) to evalu- ate each preference-question pair based on the following criteria:
Validity Assessment: Labelers discarded samples exhibiting any of the following issues:
Questions that directly contradict the user's preference
Questions already aligned with the user's preference, requiring no additional consid- eration
Questions impossible to answer due to insufficient information (e.g., missing location or specifics)
We sampled responses from 5 different LLMs with- out providing the preference to assess the preference-unwwan violation rate. Pairs with higher violation rates were prioritized create more challenging dataset.
Automatic Violation Rate Analysis:
Automatic In-Context Difficulty Rating: We developed rating prompt using 50 human-labeled examples as in-context demonstrations, Each example included human ratings along two dimensions:
[High, Medium, Low] (a higher rating indicates the prefer- ence easier to violate without knowledge of it)
Violation Probability:
Reasoning Difficulty: [High, Low] (indicates whether, even when aware of the pref- erence, answering the query in a preference-follooingg way requires reasoning) Multiple iterations of prompt tuning and example selection ensured reliable ratings. We will release the data generation prompts in our repository.
The filtering process yielded approximately 3,000 high-quality pairs. We then manually selected approximately 50 preferences per topic, resulting in final dataset of 1, 1,000 high-quality explicit preference-questionn pairs.
Final Selection:
Step 2: Generation of Implicit Choice-Based Preferences Building upon the explicit prefer- ences and to develop more challenging preference types for preference following, created two- turn conversations incorporating multiple-choice questions, where the user's 's preference will be im- plicitly revealed through option selection. The generation process followed these steps using Claude 3 Sonnet: (1) For each explicit preference-query pair, we generated a simpler, related query that differs from the final test query. (2) We created four options for each query, ensuring that only one
41

---

Published as a conference paper at ICLR 2025
option aligns with the user's preference while the other three violate it. (3) We constructed two-turn conversations where the user selects the single option that aligns with their preference. (4) Each conversation concludes with brief assistant acknowledgment that avoids explicitly restating the user's preference.
Step 3: Generation of Implicit Persona-Driven Preferences To create more natural preference expressions within extended conversations, we aimed to extend the preference revelation over longer conversations. However, simply expanding explicit preference into a long conversation can be challenging and may inadvertently reduce task difficulty by reinforcing the preference across mul- tiple turns. Therefore, we decided to craft conversations where the topic mainly revolves around persona, and the preference is only briefly mentioned. We developed persona-augmenn preference conversations as follows:
1. We first generated and filtered 100 distinct and diverse personas using Claude 3.5 Sonnet, ensuring that the personas were topie-independent to prevent preference conflicts.
2. For each of the 1,000 explicit preferences, we randomly assigned one persona and we make sure the persona does not conflict with reveal the preference. Using Claude 3.5 Sonnet, we then generated 5-8 turn conversations that incorporated both the explicit preference and the assigned persona. The primary conversation focus centered on persona- related inquiries rather than explicit preference discussion.
This three-component methodology resulted a diverse dataset of preference following, ranging from explicit statements to naturally embedded implicit preferences within extended conversations. Our dataset will be released along with the propmts used data construction above.
A.15 How DOES FINETUNING PREFEVAL GENERALIZE TO IMPLICIT PREFERENCE SETTINGS?
In Section 3.7, we demonstrated that fine-tuning a Mistral 7B model on our dataset enhanced its sefference-followiee capabilities and generalization to both unseen topics and longer contexts. While the training dataset consisted solely of Mistral model's responses in explicit preference settings using the reminder baseline-with no intervening contextual turns-we now investigate whether the trained model generalizes effectively to implicit preference settings.
As shown in Table 17, preference fine-tuning improves performance on implicit preference follow- ing tasks. This generalization suggests that training on explicit preferences not only enhances the model's attention to user preferences but also strengthens its Preference Inference capability (as defined in Section 2.1).
Table 17: Preference following accuracy (%) on implicit settings before and after supervised fine- tuning (SFT). We evaluate the model's ability to follow preferences in two implicit scenarios with 5 turn contextual conversation. We show the results across over 100 preferences instances over topics in the zero-shot setting. We find preference finetuning brings more improvements for Implicit Pervona-Driven preferences.
Topic
Model
Preference Following Accuracy (%)
Implicit Persona-Driven
Implicit Choice-Based
Before SFT
1.79
3.57
Travel Restaurants
After SFT
55.36
14.29
Before SFT
14.81
1 11.11
Travel Hotels
After SFT
74.07
51.85
A.16 LOST IN THE MIDDLE: IMPACT ON PREFERENCE FOLLOWING
Recent work has shown that language models struggle to effectively use information placed in the middle of their context window, showing better performance when important information appears
42

---

Published as a conference paper at ICLR 2025
at the beginning end (Liu et al., 2024b). Following this finding, investigate whether this "lost in the middle" phenomenon extends to preference following behavior. As shown in Figure 23, we experiment with Claude 3 Sonnet and Claude 3 Haiku across four diverse topics by inserting the preference in different locations a fixed 100 turn conversation. We observe that preference fol- lowing significantly degrades when preferences are placed in the middle of the conversation (around turn 50) compared to when they are positioned at the beginning or end. This phenomenon preserves across models with different sizes. This aligns with Liu et al. (2024b) findings about LLMs' difficulty in accessing mid-context information.
Pref Following Accuracy
Lifestyle
Shop Motors
Travel Activities
Travel Transportation
1.0
Clause Sonnet
1.0
1.0
Cloude Sonnet
Claude Helku
0.5
Claide 3 Halks
0.6-
0.6
Sonnec Citue Halku
0.8
0.4-
0.4
0.2-
3 Halu
0.2
0.2
0.2
0.0
0.0
70
Figure 23: Preference following accuracy across different preference insertion locations in fixed 100-turn conversation for Claude Sonnet and Claude Haiku, tested on four topics. This indicates the "lost in the middle" phenomenon extends beyond factual retrieval tasks to preference following.
A.17 ADDITIONAL RESULTS ON DYNAMIC PREFERENCE FOLLOWING
In Section 3.6, we demonstrated that inserting multiple preferences and conflicting preference pairs in conversations improved preference following performance for Claude Sonnet and Claude Haiku. We conduct additional experiments across multiple models to further validate the obser- vations. As shown in Figure 24, the positive correlation between the number of preferences and preference following accuracy extends to Mistral 8x7b and Mistral 7b. We hypothesize this is be- cause inserting multiple preferences throughout the conversation reinforces the model's attention to user preferences, the LLM allocates more attention user preferences relative to other unrelated contextual information. Further, we extend conflicting preference experiment to Mistral models, as shown in Figure 25. Overall, three models (Claude 3 Sonnet, Claude Haiku, and Mistral 7b) demonstrate improved performance with conflicting preference pairs. However, Mistral 8x7b ex- hibits similar performance between conflicting and non-conflicting pairs, with a slight advantage for non-conflicting pairs. This suggests the effect is model model-dependent and our findings still holds that conflicting preferences do not necessarily harm performance. We attribute this phenomenon to ropio-reinforcemment effeett although the preferences conflict, they address the same topic do- main, potentially strengthening the LLM's memory of the preference context and leading to higher expresses "I prefer detailed responses
accuracy in preference following. For example, when expreseeeee when I ask for paper summarization" and later states "I prefer concise responses when ask for paper sunmarization" these contradictory preferences nonetheless reinforce the LLM's attention to response length as a significant preference dimension.
A.18 HUMAN EVALUATION OF THE LLM-BASED EVALUATOR
To validate the reliability of our LLM-based evaluation approach, we conducted a comprehensive human evaluation study comparing human judgments against Claude 3 Sonnet's assessments. We randomly sampled 100 evaluations for each preference form, encompassing diverse scenarios across all models. baselines, and conversation turns. Table 18 presents the agreement rates between human annotators and the L.M evaluator across 4 different evaluation checker as defined in section 1. The results demonstrate strong alignment between human and LLM judgments, with particularly high agreement rates in detecting helpful responses and hallucinations.
43

---

Published as a conference paper at ICLR 2025
Multipic ple Praferences Analysis:
Accuracy of the First Preference (Stated at Turn 0)
Pref Following Accuracy
Conflicting vs Non-conflicting Preferences
Mistral Bx7b (50 Tam)
Pref Following Accuracy
slngl preference
contlct
Tan)
0.6
pon-corflicppal
0.60
0.4
0.40
0.2
0.0
of Preferences Stated In the Conversation
Mistral 7b
Mistral 8x7b
Figure 24: Preference following accuracy gen- erally improves with more stated preferences across Mistral 8x7b, Mistral 7b, and Claude 3
Figure 25: Effect of adding conflicting versus non-conflicting preferences on adherence. The red bar indicates the performance when only the original preference is present. Results are aver-
models. Results shown with 50-turn inter con- versation inserted (Mistral) and 80-turn inter conversation inserted (Claude 3 Sonnet) due to context length limits.
aged over five topics using a fixed 100-turn con-
versation.
Table 18: Human-LLM agreement rates across different error checker as well the final preference following accuracy. We randomly sampled 100 evaluations from each preference form and calcu- lated the agreement rate between human annotators and the LLM evaluator in their judgments.
Error Checker
Explicit Preference
Implicit
Implicit
Choice-based
Persona-driven
Violate Preference?
0.92
0.86
0.95
Acknowledge Preference?
0.88
0.90
0.97
Hallucinate Preference?
0.98
0.96
0.92
Helpful Response?
0.96
0.93
0.90
Preference Following Accuracy
0.97
0.92
0.96
44


Reviews:
[{"review_id": "brnmhR86v2", "reviewer": "Reviewer_Uqsa", "summary": "This Datasets and Benchmarks paper presents PREFEVAL, a benchmark designed to evaluate large language models’ capabilities in following lifestyle preferences in long-context, multi-turn, personalized conversational settings. PREFEVAL includes 3,000 manually curated preference-query pairs spanning 20 diverse topics, assessing models’ ability to remember, infer, and follow explicit and implicit preferences over long conversational contexts (up to 100,000 tokens) and tests various settings such as zero-shot, reminding, CoT, RAG and Self-critic approaches. Using generation and classification tasks, the authors evaluate 10 LLMs, including prominent models like Claude, GPT-4, and LLaMA. The results indicate significant challenges for LLMs in following user preferences over extended dialogues, with a steep decline in preference-following accuracy as context length increases.", "questions": "- Authors mention breakdown of queries by explicit vs implicit preferences (choice based, persona driven) and share results in Figure 5. Shedding more light on some visualization of attention (and the lack of it) could have shed more light on understanding why the results degrade so much in different scenarios. \n- Authors use LMSYS- Chat-1M dataset for the experiments (where they intersperse persona/preference based queries) - some of the models they test may have seen this dataset before during training - what would be the impact of such exposure?  \n- Since this is a dataset contribution, some more clarity on understanding dynamic user preferences might have been useful. For example, mining instances where personalization instances already exist in LMSys dataset and understanding preference following there would have made the work very convincing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- 3,000 manually curated (with the help of LLM tools) preference-query pairs across 20 diverse topics is a good dataset contribution, covering various preference forms (explicit and implicit) and contexts, making it a valuable tool for future research on personalized conversational AI.\n- The paper presents valuable insights across major open and proprietary models along with an understanding of various strategies that work to help improve preference following. \n- The error categorization (preference-unaware violation, preference hallucination, inconsistency violation, and unhelpful response) provides actionable insights into the types of failures occurring during preference following. These insights allow targeted intervention methods to improve the performance of the models and reduce hallucinations and increase helpfulness.\n- Personalization is a very important topic for LLMs that's still underrepresented and scope for major advancements. Need for more benchmarks in this field will help with faster progress to create more personalized LLMs.", "weaknesses": "- The LLM-based evaluation framework introduces a subjective component and lacks qualitative and manual validations for in assessing preference adherence (despite a very small manual checks on the responses with 5% error rate) \n- inserting preferences within the LMSys-Chat-1M dataset may be unnatural. It is not clear what strategies were used for maintain semantic coherence of the dataset while introducing preferences and queries. \n- There is some overlap with long context recall and reasoning based benchmarking. There is no analysis of existing datasets and benchmarks in the area that may qualify for personalization based benchmarking. Also, there is no discussion around other efforts like 'LaMP: When Large Language Models Meet Personalization' or others (authors mention these in Appendix, however an exhaustive personalization benchmark could cover some aspects of these other benchmarks mentioned in Appendix).\n- This study emphasizes improvements with inserting 'Reminders' and other strategies like self-critique. This might skew model development more towards prompt engineering instead of focusing on understanding intrinsic improvements methods in memory and retrieval capabilities.\n-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1KLg8CuSDt", "reviewer": "Reviewer_L7qv", "summary": "The authors introduce PREFEVAL, a benchmark for evaluating if LLMs follow personal preferences in conversational settings. PREFEVAL assesses how well LLMs can infer, remember, and adhere to personalized preferences across multi-session conversations. It includes 3,000 preference-query pairs across 20 topics, with preferences presented explicitly or implicitly. Ten LLMs were evaluated using prompting, iterative feedback, and RAG. LLMs struggle to maintain preference adherence over longer dialogues, especially without _reminders_, with accuracy dropping below 10% for 10-turn interactions. Finally, advanced prompting and fine-tuning improved model performance, particularly in handling long contexts and multiple or conflicting preferences.", "questions": "A particularly irritating effect of LLM \"personalization\" / \"memory\" is that it assumes that some attribute (e.g. I like pizza) should now apply to all my queries, even if not relevant (e.g. high recall, low precision). Am I correct in assuming this is a \"preference hallucination\" violation? \n\n> To validate our LLM-based evaluation method, we manually checked 200 randomly sampled evaluations, with an observed 5% error rate.\nHow was this error rate computed? What was the distribution of errors in the 200 randomly sampled evals? \n\nHow were the implicit personas generated and validated? How much variance is there between the generated personas?\n\nWith the additional space in the camera ready, I would move some examples of (p, q, explanation) pairs to the main text.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think this paper covers a timely and interesting topic. The paper and dataset yields several strengths:\n\n1. The error analysis is quite interesting! The authors look at violations of preference applications that are particularly aggravating when interacting with instruction following LLMs\n\n2. The interventions are well-motivated, simple, and moderately effective (though there is still space for improvement).\n\n3. The authors validate their prompts with human annotators.\n\n4. The paper is well-written and generally easy to follow.", "weaknesses": "I have three concerns. If the first two are resolved, this would improve the paper!\n\nFirst, I'd like to see some analysis of the \"Lost in the Middle\" effect (see: https://arxiv.org/abs/2307.03172). I suspect that some of the failures can be attributed to how models use long context. If we include the reminder at the start instead of the end, or in both places---does that affect performance? I'd also encourage the authors to draw distinctions between needle-in-a-haystack benchmarks for long-context LLMs and this dataset. One reason might be that the implicit / persona settings in the dataset require one to _reason_ over preferences, not just retrieve a needle---but I would make this comparison more explicit in the main text.\n\nSecond, the long-context integration feels a bit heavy-handed. The authors seem to randomly sample conversations and randomly intersperse (p, q) pairs within this context. This seems somewhat artificial. Were there any measures taken to make sure that the bridge between the pre-existing context and the (p, q) pairs were natural?\n\nFinally (and I don't expect or require the authors to directly fix this), the (p, q) pairs are entirely synthetically generated. This is an unfortunate limitation: a personalization dataset generated with no \"real\" people. One could argue that generations from LLMs \"simulate\" some aspects of human behavior. Still, I'd recommend mentioning this in the limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gxbCo64yuZ", "reviewer": "Reviewer_PRWG", "summary": "This paper introduces PREFEVAL, a benchmark for evaluating large language models' (LLMs) ability to infer, remember, and follow user preferences in multi-turn conversations. The main contributions are: \n\n(1) Introduction of a comprehensive benchmark assessing LLMs' ability to follow user preferences in conversational settings, comprising 3,000 manually curated question-preference pairs across 20 topics and 3 preference forms. \n\n(2) Through extensive evaluation of 10 state-of-the-art LLMs, the paper reveals current challenges in actively following user preferences during conversations and presents important findings about personalized preference adherence.", "questions": "(1) **The detailed dataset construction process is not introduced**. Multiple methods should be employed to verify the dataset's high quality, which would make subsequent experiments more convincing. I recommend adding a dedicated section that outlines the data preparation process and quality control measures. This should include details on selection criteria, query development, and validation procedures used to ensure data quality.\n\n(2) **How were the reasoning steps for CoT obtained?** Since reasoning steps are crucial for CoT effectiveness, the authors did not discuss this detail. I suggest the authors provide examples of CoT prompts, or describe their process for generating these prompts.\n\n(3) **When multiple preferences are injected, why does the adherence to original preferences improve?** Intuitively, when multiple preferences are introduced, LLMs would struggle to accurately identify user preferences. This should make it harder for LLMs to recognize the original preferences. However, the paper's results show the opposite.(Section 3.6, Page 8). I recommend the authors explore potential explanations for this phenomenon, or conduct additional experiments to verify if this effect is consistent across different models.\n\n(4) **When conflicting preferences are introduced, model performance improves compared to single preferences, which is counter-intuitive**. Why does this phenomenon occur, and do other LLMs besides Claude 3.5 Sonnet exhibit similar behavior? (Section 3.6, Page 9). It is recommended that the authors expand their analysis to include more models beyond Claude 3.5 Sonnet. Additionally, they should provide potential explanations for these counter-intuitive results and discuss their implications for understanding LLM behavior in complex preference scenarios.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper's strengths are as follows: \n\n(1) **In terms of originality**, it proposes a comprehensive benchmark for evaluating LLMs' personalized preference adherence capabilities, which is a relatively novel research direction.\n\n(2) **Regarding quality and writing**, the authors thoroughly validated LLMs' capabilities through extensive evaluation of multiple models' preference adherence across various conversation topics. The paper structure is clear, with well-presented experimental design and results.\n\n(3) **In terms of significance**, the findings reveal key challenges in current LLMs' personalized preference adherence. In daily use and experiments, we also observe that LLMs' adherence capability diminishes with increased conversation length. Broadly speaking, ensuring LLMs consistently follow user characteristics or instructions is crucial for their real-world applications.", "weaknesses": "The paper's weaknesses are as follows: \n\n(1) **The data construction process is not detailed**. A major contribution of this paper is the manually constructed personalized preference dataset. The construction process directly affects data quality and thus impacts model results and experimental reliability. However, I did not find a section detailing the dataset construction process.\n\n(2) **Additional experiments could enhance the comprehensiveness of findings**. The authors verified LLMs' adherence to explicit preferences through fine-tuning. However, implicit preferences are often more challenging to identify and follow. Experiments in this area are recommended for a more comprehensive understanding of LLMs' preference recognition and adherence capabilities.\n\n(3) **Line numbers are missing**. The paper lacks line numbers, making it difficult to precisely reference specific content locations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AWN4fPdxnb", "reviewer": "Reviewer_d6T5", "summary": "This paper presents a challenging and important problem of LLMs: memorizing the user preference in a long-context, multi-turn dialogue (could be up to 100k tokens). To achieve this, the authors create a PrefEval dataset containing 3000 examples (i.e., preference-query pairs), distributed across 3 preference forms and 20 different topics related to our daily conversations with chatbots. \n\nSpecifically, in their dataset, the stated user preference could be in three different forms (blue box in Figure 1; see also Table 14 to 16 in Appendix):\n\n* one turn (also only one sentence in the user utterance, e.g., I don't like seafood)\n\n* two turns (e.g., a user first asks a question, then after an LLM provides four options, the user chooses one of them and rejects the others)\n\n* $>$ 2 turns (i.e., they insert the user preference within an unrelated 4~8 turn dialogue; for instance, a user \"accidentally let slip\" and mentions he/she doesn't like seafood in a library-related dialogue)\n\nTo make the conversation turn extremely long, they leverage the LMSYSChat-1M dataset by randomly selecting the dialogue up to 100k tokens, and then they insert this long context between the user preference (typically in the first turn) and the query question (at the end of the dialogue).\n\nThe authors benchmark the PrefEval dataset using a variety of LLMs (GPT, Claude, Mistral, Llama-3, and Gemini), as well as the latest GPT-o1 system. Concretely, they conduct different experimental settings (see also Appendix A.4): \n1. (pure) zero-shot\n2. zero-shot + inject a sequence (like “provide an answer that follows my preference”) after the test question (termed \"reminder\" in this paper)\n3. let LLM evaluate its own response and generate the answer again, similar to the self-correct task or simple CoT baseline (termed \"self-critic\" in this paper)\n4. give few-shot examples to the LLM before the LLM response (termed \"$CoT$\"* in this paper)\n5. use RAG method to retrieve relevant sentences in the previous dialogue then provide these sentences before the LLM response. \n\nAdditionally, they finetune the Mistral 7B LLM using their dataset with the reminder method and observe that SFT improves in alignment with user preferences. This simple SFT approach could be more robust in long-term dialogue if the training data has more unrelated turns inserted between the user preference and the query dialogue (which is reasonable).\n\n(*) though this could be a bit confusing, see weakness 1 below", "questions": "Despite the above major weaknesses, **these two weaknesses could be easily fixed** to improve this paper's clarity as follows (first migrate this paper to the 2025 template):\n\n### Suggestion for Weakness 1-(i) \n\nPerhaps (a) modify the \"$CoT$\"* to \"few-shot\", and (b) either include some (intrinsic) self-correction papers that the authors found related to your \"self-critic\" or simply state self-critic is also the CoT baseline (with CoT citations). \n\n### Suggestion for Weakness 1-(ii) \n\nI personally would prefer to report accuracy in MCQ experiments, whereas LLM-as-a-judge is for error-type analysis only. However, If replacing the generation task performance with the classification task requires substantial revisions, then it's best to explicitly state something like \"the accuracy in the generation task **is highly correlated** to the classification task\" either in Figure 1 (on page 2) or in Section 2.5 (on page 4). \nEither way should be better than first mentioning it on page 7 (in Section 3.4): \"This correlation suggests that despite task differences, ... Classification tasks could thus serve as an efficient proxy for evaluating preference following in complex generation scenarios across models and methods.\" Moreover, this may not be generalized to other complex generation tasks such as machine translation.\n\n### Suggestion for Weakness 2 \n\nDetail the dataset construction in the Appendix, especially the process of how the user preference, its related query, as well as the construction of implicit preference form (Tables 15 and 16). Is the PrefEval dataset constructed in a more standardized way that could be formalized for future research to follow this criterion? If not, then it is also appreciated to include the flowchart or the pipeline process of how the explicit and two implicit forms are generated, perhaps with a walkthrough example, in the Appendix.\n\n\nI could be convinced if the authors can address the following questions below. No need to conduct additional experiments, just clarify the rationale behind this paper (e.g., the figure shown, experimental setup).\n\n\n# Questions\n\n1. See Weakness 1. If I am very much mistaken, perhaps the authors could provide some papers that use this term in this scenario?\n\n2. See Weakness 2. \n\n3. In Figure 1\n* Do authors conduct experiments where the user preference (blue box) is in between the multi-session conversations (green box)? Specifically, as the multi-session conversations (green box) are from LMSYS-Chat-1M dataset in Section 2.4, I wonder if the top one is also filled in the experiments (just curious if I am missing something in this paper). \n* In explicit preference form, as the conversation turn is defined as a pair of user and chatbot utterances in Section 2.1, what is the corresponding \"chatbot\" utterance in user preference when a user states \"I dislike eating seafood due to an allergy\" in the first turn (see Figure 1)? Is it an empty string or some pre-defined sentence (because I cannot find it in Table 14)? If so, is it used across all explicit data?\n* In the implicit choice-based preference form, does the first user question the same as the query? Specifically, as shown in Table 15, suppose a user asks \"What are some good hotel options for my upcoming trip to Paris?\" in the first turn, then does the user ask the same question in the last turn (which is the query)?\n* When evaluating results, does the green box (multi-session conversations) also present to the evaluator? If so, then the figure is missing, which may cause some confusion as the evaluator would become impractical in real-world scenarios (as we normally don't have labeled data). If not, then could the authors explain why it's excluded in reporting accuracy (I think it is reasonable to exclude this only for error analysis, as shown in Appendix A.11)?\n\n4. In the Introduction, does \"prompting\" in the second contribution mean \"zero-shot, reminder, and $CoT$*\" in Section 3.1? Similarly, does \"iterative feedback\" mean \"self-critic\"? As for RAG, is the retrieved context \"utterance-based\" or \"sentence-based\" (e.g., if a user utterance has 3 sentences, only one of which is related to the query)?\n\n5. In Section 2.5, does LLM-as-a-judge similar to the work in [1]? Are the four error types used in the previous studies, or did the authors pre-define them? In Table 1, why the value of (Preference Hallucination Violation, Hallucinate Preference?) is \"No\"?\n\n6. In Section 3.1, as few-shot examples are given in the \"$CoT$\"* method, did authors previously also experiment with placing those examples on top (as many papers first provide LLMs with examples and then ask them to perform some tasks)?\n\n7. In Table 2, Is the number of data conducted in this restaurant 3000 x 5.6% (168) or 1000 x 5.6% (56) in 10 Turns and 300 Turns, respectively? I am also curious how this travel-restaurant topic is reported but not the others. Is it because of the cost, or does GPT-o1 perform better in this topic?\n\n8. In Section 3.3, why does Claude struggle more with implicit persona-driven dialogue compared to the choice-based one? In Figure 5, I found that Claude 3 Sonnet is generally on par with or better than in the reminder scenario (it is worse if the token length is around 16k~23k, but for longer context, there isn't much difference).\n\n9. In Section 3.4, what is the \"structure task\"?\n\n10. In Section 3.6\n\n* Could the authors provide an example in \"Impact of Multiple Preferences Stated in Conversation\"? Does it mean that if a user says \"I don't like seafood\" (restaurant) in the first turn, then he/she also states \"I don't like wearing jeans\" (fashion), \"I like to read\" (music & books) in the following turns?\n\n* Similarly, could the authors provide an example in \"Effect of Conflicting Preferences Stated in Conversation\"?\n\n* What is the number of data tested in Figures 8 and 9, respectively?\n\n11. In my opinion, even though the $(p, q)$ pair is indeed different in this sense, I would expect 3,000 to be the \"totally\" different dataset (if only glancing at the Abstract). Maybe it is better to rephrase something like 1,000 data, each with three \"levels of difficulty\" (user preference forms).\n\n12. Is the temperature set to 0 for all LLMs?\n\n13. Do the authors include the Ethics Statement?\n\n[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica, NIPS 2023\n\n# Typos and Format Suggestions\n\n1. In Figure 1 \"shnmp\" $\\rightarrow$ \"shrimp\"\n2. The em dash ($-$) in Section 3.7 should be consistent with the previous ones.\n3. In Figure 10, \"W/\" $\\rightarrow$ \"w/\"\n4. Could try to fit GPT-4o in Section 3.1 since there are only nine models mentioned in this section.\n5. Should use wrapfigure in Figures 3, 7, and 10 so that the caption is below the figure.\n6. From my understanding, when citing the papers in Appendix A.2, \\citet{} should not be used even if they are in the middle of the sentence (consistently use \\citep{}).\n7. In Table 14~16, if Table 14 shows 10 examples while Table 15 could only display 6 within a page, perhaps it would be more visually comparable to select the first 6 examples from Table 14 (in the same order).\n8. For color consistency in Figure 10, the author could draw the 5 w/o SFT methods first so it will match the color in Figure 4. The authors could also consider drawing two legends in this figure (for example, a legend displays the marker: star denotes w/ SFT and circle denotes w/o SFT).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The strength is mostly summarized in the Summary.\n\nOverall, this paper is well-motivated by the current capabilities of LLMs to follow the user preference in long-term conversation, which is also an emerging issue of contemporary (\"general-purpose\") LLMs.\nTo take a step further, they construct this dataset and gauge whether LLMs could implicitly retrieve the user preference and follow it in the long-context, long-term setting. \n\nWhile the originality of following user preference may not be really novel $-$ since memorizing the user preference has been studied in other NLP tasks such as task-oriented dialogue (TOD) $-$ there is still a lack of comprehensive experiment in this extremely long-term scenario, not to mention various LLMs are conducted in this paper. Multiple baselines are performed when LLMs are frozen, including the pure zero-shot, zero-shot with a \"follow my preference\" string, few-shot, self-correction/CoT (termed \"self-critic\"), and RAG. \nDifferent question (i.e., query) types are also tested (\"generation\" task and (MCQ) \"classification\" task).\nThis is perhaps one of the most comprehensive evaluations I've read in recent CoT-prompting papers.\n\nThe authors also plan to open-source the dataset and release the prompts used in the Appendix.", "weaknesses": "The major weaknesses are that (1) some terms may confuse the readers at first glance and (2) the dataset construction is not very clear in this paper (as it is in the datasets and benchmarks track) $-$ even though the benchmark experiments are exceedingly extensive. \n\n## Weakness 1\n\n(i) In this paper, the \"$CoT$\"* method is defined as having several (i.e., 5) examples before an LLM response (visualized in Appendix A.4, page 17). However, providing examples to the LLM is generally considered to be few-shot learning (or, previously in-context or demonstration learning). In particular, the term CoT should be more related to the \"self-critic\" method in this paper (which can also be viewed as a simple self-correct task [1]). \n\n(ii) The evaluation metric of the \"generation task\" is the accuracy (see page 2, Figure 1's caption) in this paper. If I understand correctly, the generation task in NLP typically refers to translation, summarization, story completion, etc. (hence, accuracy is generally *not* the primary evaluation metric, which could be BLEU/ROUGE/METEOR/BERTscore/etc.). While the authors state that \"the LLM generates a response of approximately 300 words in reply to the user's query.\" on page 4 and provide a real-life scenario (asking for a travel recommendation in New Orleans and see if an LLM knows the user doesn't like jazz) in the Introduction, the evaluation is still mainly the accuracy by leveraging LLM-as-a-judge, which could be done $-$ and is also simpler and cheaper than LLM-as-a-judge $-$ using the answer extraction in [2]. To be more specific, we can prompt the LLM by asking \"Does the response contain any [seafood] cuisine?\" in Figure 1, provided that a user stated that he/she doesn't like [seafood], then extract the Yes/No response afterward.). \n\nMoreover, perhaps using LLM-as-a-judge is costly and complex, the authors also conducted multiple-choice question (MCQ) experiments and included the relation between the classification task and generation task in Appendix A.5, which demonstrates that there is a high correlation in accuracy. In this vein, while LLM-as-a-judge also reports four error types introduced in this work (see Table 1 and Figure 7), there isn't a significant difference between the \"generation\" task and the \"classification\" task in this paper to my knowledge (though those four error types should be useful for error analysis in this sense). As a result, after the readers look at Figures 5 and 6, they may not immediately understand why the authors evaluate two \"different\" tasks (though I believe they *should be* the same, see explanation below).\n\nTake Figure 1 as an example. The user first asks \"What Japanese cuisine dishes should I try on my visit?\", then this query (from my understanding) is for the generation task (because LLMs tend to reply with lengthy sentences for general (i.e., non-specific) questions), which is evaluated by LLM-as-a-judge (Claude 3 Sonnet). On the other hand, if, for example, the query becomes \"What Japanese cuisine dishes should I try on my visit? (A) beef and mushroom noodles (B) vegetable stir fry (C) seafood with shrimp and crab (D) grilled chicken Caesar salad\", then perhaps this is for the classification task? (I cannot find the query related to MCQ so I give an illustrative example above.)\n\n## Weakness 2\n\n(i) Though the three forms of user preference (blue box) and the multi-session conversations (green box) are clearly stated in Figure 1, it is unknown what the **source** of the user preference is, as well as the construction between three different preference forms. After reading the paper, I only found \"These pairs were manually curated with the assistance of GPT-4, Claude 3 Sonnet, and Claude 3.5 Sonnet.\" in Section 2.2.  In Tables 14 to 16, the authors only explain \"Every implicit preference dialogue is derived from an explicit preference\" and \"Every dialogue is derived from an explicit preference and is randomly assigned a persona to simulate a longer conversation.\"\n\nSpecifically, how to first generate the explicit user preference and get the distribution in Figure 2? Are the preferences mined from existing dataset(s) or generated by LLMs? After this, how is the corresponding query generated? Next, what are the criteria for generating the implicit choice-based/persona-driven dialogue? I found in Table 15 that the templates are rather fixed. For example, the chatbot only provides four choices in the first turn, and the user as well as the chatbot response are highly similar across topics. Are there any rules for prompting LLMs or are they human-generated templates (with some text combinations)?\n\n(ii) Another issue might be the problem formulation in Section 2.1, as it is not formalized in math. Although I have no trouble understanding the PrefEval dataset and the proposed framework/experiments in general, other not-too-familiar readers may need to re-read several times to grasp the idea behind this paper. For example, the formulation of the conversation flow in Figure 1. Here, we only know user preference $p$ is in the blue box, and the query $q$ is in the red box. How to generalize it in a $m$-turn conversation (i.e., {$u_1, b_1, u_2, b_2, ..., u_m, b_m$})? Alternatively, as each box is separated, can the PrefEval dataset construction be formalized in a mathematical way?\n\n\n[1] Large Language Models Cannot Self-Correct Reasoning Yet, Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou, ICLR 2024\n\n[2] Large Language Models are Zero-Shot Reasoners, Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa, NIPS 2022\n\nRegarding the format, the line number is missing in its current PDF format, so the presentation is somewhat unclear, making it difficult to address the concerns below (not sure if it is because the version is the previous one).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)