Exploring Persona Sentiment Sensitivity in Personalized Dialogue Generation
Authors: YongHyeon Jun, Hwanhee Lee
Accepted at: acl25

Exploring Persona Sentiment Sensitivity in Personalized Dialogue Generation

Yonghyun Jun and Hwanhee Lee*
Department of Artificial Intelligence, Chung-Ang University
{zgold5670, hwanheelee}@cau.ac.kr

Abstract
Personalized dialogue systems have advanced considerably with the integration of user-specific personas into large language models (LLMs). However, while LLMs can effectively generate personalized responses, the influence of persona sentiment on dialogue quality remains underexplored. In this work, we conduct a large-scale analysis of dialogues generated using a range of polarized user profiles. Our experiments reveal that dialogues involving negatively polarized users tend to overemphasize persona attributes. In contrast, positively polarized profiles yield dialogues that selectively incorporate persona information, resulting in smoother interactions. Furthermore, we find that personas with weak or neutral sentiment generally produce lower-quality dialogues. Motivated by these findings, we propose a dialogue generation approach that explicitly accounts for persona polarity by combining a turn-based generation strategy with a profile ordering mechanism and sentiment-aware prompting. Our study provides new insights into the sensitivity of LLMs to persona sentiment and offers guidance for developing more robust and nuanced personalized dialogue systems.*

1 Introduction
Personalized dialogue systems have evolved through advances in dataset construction (Li et al., 2016; Zhang, 2018), training methodologies (Kingma, 2013; Bowman et al., 2015), and studies on integrating user-specific preferences to enhance conversational abilities (Zhang, 2019; Roller, 2020). With the advent of Large Language Models (LLMs), embedding personas via system prompts (Yang et al., 2023; Wang et al., 2023b) has become effective. This progress has enabled applications ranging from personalized chatbots (Shuster et al., 2022; Lee et al., 2023) to enhanced performance in downstream tasks through inter-LLM conversations (Park et al., 2023) and synthetic dataset generation (Jandaghi et al., 2023). These developments underscore the growing importance of personalization in dialogue systems. However, despite these advances, recent studies have shown that LLMs are highly sensitive to contextual senti-

Figure 1: Examples of dialogues generated by polarized users. We indicate Positive personas in blue text and Negative personas in red text. Matching bullet points denote where user persona attributes appear in the utterance, and we highlight them in yellow if they contradict. Incoherent segments of the dialogue are marked with a red background.

![Examples of dialogues generated by polarized users.](page_1012_340_388_388.png)
ment polarity (Liu et al., 2024; Wu et al., 2024b). In contrast, the impact of assigned persona sentiment on dialogue quality remains less explored. For example, Figure 1 illustrates dialogues generated by users with distinctly positive and negative personas. In dialogues driven by positive personas, the integration of persona attributes occurs in a manner that supports coherence and aligns naturally with the flow of conversation. Conversely, dialogues featuring negative personas often display contradictions and inconsistencies, as the persona attributes may be overemphasized or misaligned with the dialogue context. These findings highlight the necessity of systematically examining how variations in persona polarity influence LLM behavior.

In this paper, we construct and analyze personalized dialogues generated across a spectrum of polarized profiles. (§3) Our extensive analysis of personalized dialogues generated across a diverse spectrum of polarized profiles reveals several key insights: 1) Dialogues between negatively polarized users tend to overemphasize persona attributes, which leads to increased contradictions and reduced overall coherence. 2) Dialogues between positively polarized users selectively incorporate persona information, resulting in interactions that are both coherent and contextually appropriate. 3) Profiles with weak or neutral sentiment generally yield lower-quality dialogues, as evidenced by significant differences in various metrics.

These insights underscore the need for methods that explicitly account for persona polarity in the design of personalized dialogue systems. To address this challenge, we propose a dialogue generation framework that combines a turn-based strategy, a profile ordering mechanism, and a sentiment-aware prompting. Specifically, personas exhibiting neutral or weak sentiment are placed earlier in the conversation, whereas more positive personas are positioned later. A concise prompt further instructs the model to attend carefully to negative or neutral sentiments when generating responses. These approaches promote more consistent and coherent interactions (§4). Our findings shed light on LLM sensitivity to persona sentiment and offer valuable guidance for developing more nuanced and effective personalized dialogue systems.

2 Study Design and Experimental Setup

This section explains the overall basic setup for our study. We first introduce the dataset and describe how we construct polarized user profiles. Next, we present our research questions, detail the dialogue generation strategies, and describe the metrics used to assess the dialogues. We use four RTX 4090 GPUs to generate and evaluate all the dialogues.

2.1 Dataset

Early research on personalized dialogue systems typically used two-party conversations paired with user profiles. These profiles are commonly represented in three ways (Chen et al., 2024):

• Descriptive sentences: Each persona is described by a few sentences (often five) that together form a user profile (Zhang, 2018; Dinan et al., 2020).

• Sparse key-value attributes: Personas are specified using concise attributes (Qian et al., 2017; Gao et al., 2023).

• User history: Each speaker is identified by an ID that can be used to retrieve external persona information (Salemi et al., 2023).

Although several datasets exist, most trace back to PersonaChat (Zhang, 2018) and ConvAI2 (Dinan et al., 2020). In our experiments, we adopt the ConvAI2 dataset and define user personas primarily through descriptive sentences.

2.2 Polarized User Profile Construction

We extract polarized user personas from the ConvAI2 dataset using a sentiment classifier and then build coherent profiles by combining multiple personas while avoiding contradictions.

Polarity-Aware Persona Sampling  Each persona in the ConvAI2 spans various content domains and carries a distinct sentiment polarity. We use the distilbert-base-uncased-finetuned-sst-2-english model (HF Canonical Model Maintainers, 2022) to classify the sentiment of each sentence, retaining only those with a confidence score above 0.99 as polarized personas. Consequently, as shown in Figure 2, 2,691 are classified as positive, 1,006 as negative, and 2,429 as neutral (weak), indicating that polarized personas constitute the majority (around 60% of all personas). Notably, approximately 17% of the dataset’s personas are negatively polarized—a higher proportion than one might intuitively expect. These figures suggest that polarized personas are not merely reflect a contrived scenario fabricated for the experiment. For detailed statistics, see Appendix F.
Figure 2: Distribution of persona sentences by polarity in the ConvAI2 dataset. We only map predictions with a confidence score of at least 0.99 to their respective polarity, classifying all remaining cases as neutral (weak).

Synthesizing User Profiles To create a user profile, we combine \( K \) polarized personas, where \( K \) is the number of personas we include in each profile. To avoid internal contradictions, we follow the procedure in (Jandaghi et al., 2023): starting with an initial persona, we iteratively add additional personas only if the nli-deberta-v3-large (He et al., 2021) detects no conflicts with the existing set. This approach yields three profile types:

• Negative Profile: only negative personas.

• Positive Profile: only positive personas.

• Mixed Profile: a random mix of positive and negative personas.

We generate 10K unique profiles for each type, which serve as the foundation for our dialogue generation experiments.

2.3 Research Questions
Building on the established user-polarized profiles, we structure our study around several inquiries. Considering Large Language Models (LLMs) can generate dialogues that reflect a given persona without additional tuning (Tseng et al., 2024), our study focuses on the following research questions:

RQ1. Are LLMs Sensitive to Users’ Polarity?

RQ2. If so, How to Make LLMs Robust to Polarity?

To address these questions, we evaluate a range of dialogue generation methods that capture the nuanced influences of polarized personas, using multiple evaluation metrics to comprehensively assess performance.

2.4 Dialogue Generation Strategy
Traditional personalized dialogue systems use a turn-based approach to generate responses sequentially based on a given persona and dialogue context (Zhou et al., 2023; Tseng et al., 2024). However, recent advances in LLMs have enabled a dual-persona joint generation method that encodes both personas simultaneously to generate the entire dialogue in one pass (Liu et al., 2020; Xu et al., 2022; Jandaghi et al., 2023). For resource efficiency, we primarily use dual-persona joint generation for large-scale dialogue production and analysis, while also experimenting with turn-based generation to identify optimal strategies (§4. Both methods use greedy decoding (zero-temperature), with a maximum length of 4096 tokens for dual-persona joint generation and 128 tokens for turn-based generation. Prompt templates for both strategies are provided in Appendix E. After generation, we filter out outliers, such as refusal-to-answer responses or repeated utterances inside a dialogue.

2.5 Evaluation Metrics
Open-domain dialogue is inherently one-to-many, meaning a single utterance can have multiple valid responses. Moreover, ambiguous conversation topics require multiple evaluative perspectives (Wang et al., 2024; Samuel et al., 2024). In our work, we focus on two key dimensions: consistency—how well a dialogue reflects a user’s persona—and coherence—the logical, smooth flow of conversation.

Consistency We adopt the C score (Madotto et al., 2019), a common metric in personalized dialogue systems that calculates entailment scores for user persona-utterance pairs using an NLI model. Detailed adaptations of the C score for our task are provided in Appendix D.1. Since summing raw scores may lead to Simpson’s Paradox (Simpson, 1951) due to varying entailment and contradiction counts, we introduce the Contradiction Ratio (Contd.), which measures the proportion of contradictions among all cases. Additionally, we propose a novel metric called the perplexity gap (P gap). Using GPT2-large (Radford et al., 2019), we compute the difference between the perplexity of a dialogue \( D \) and the perplexity of \( D \) conditioned on two user profiles \( U_1 \) and \( U_2 \), quantifying how much the user profiles influence dialogue generation.

Coherence We assess dialogue coherence using the perplexity (Perp.) of the dialogue (Song et al., 2021; Zhou et al., 2023), estimated with GPT2-large (Radford et al., 2019). Additionally, we employ two automated coherence evaluation frameworks: Q-DCE (Ye et al., 2021), which uses a
BERT-based encoder to predict a coherence score, and PairEval (Park et al., 2024), which utilizes a fine-tuned LLaMA-2 model to compare generated dialogues with reference responses. Details for these metrics are provided in Appendix D.2.

GPT Score (G-Eval) Recent studies have shown that LLM-based evaluations can serve as an effective alternative to human annotation (Chiang and Lee, 2023). Inspired by CoT evaluation framework for summarization tasks (Liu et al., 2023), we adapt a similar approach with GPT-4o (Hurst et al., 2024) to evaluate dialogue quality, following (Jandaghi et al., 2023). Our prompt format, along with explicit guidelines for each evaluation dimension, allows the model to provide consistent judgments. Full prompt details are available in Appendix E.

Human Evaluation Beyond the automated metrics, we also conduct a human evaluation on a subset of samples. For each configuration under evaluation, we randomly sample 40 persona–dialogue pairs generated by Qwen. Three fluent English speakers then rate these samples using the same criteria as the LLM-based automatic evaluation, but on a condensed 1–3 scale. We set the final score for each configuration to the average of the three evaluators’ ratings.

3 Are LLMs Sensitive to Users’ Polarity?

3.1 Does Dialogue Quality Diverge According to Polarized User-Pairing?

Inspired by psychological research indicating that the quality of conversations varies with participants’ traits (Hassan et al., 2019), this section explores how polarized profiles and pairing types affect dialogue outcomes under various configurations. We conduct a large-scale, detailed study employing four backbone LLMs and eight metrics to verify the robustness of the results.

3.1.1 Experiment Setup
Pairing User Profiles We leverage the polarized user profiles described in §2.2, along with the original ConvAI2 (Dinan et al., 2020) user profiles, to construct a range of dialogue scenarios—including both homogeneous (e.g., two negative profiles) and heterogeneous (e.g., one negative profile and one positive profile configurations.) Specifically, we create five pairing types, each with 3K user pairs:

Original Pairing: Two original ConvAI2 profiles, serving as a baseline.

Negative Pairing: Two Negative Profiles are paired.

Mixed Pairing: Two Mixed Profiles are paired.

Positive Pairing: Two Positive Profiles are paired.

Opposite Pairing: One Negative and one Positive Profiles are paired. To reduce bias caused by prompt ordering (Lu et al., 2021), half the pairs begin with the negative, and half begin with the positive.

Experiment Pipeline For each pairing configuration, we feed all user pairs into four modern LLMs—LLaMa-3.1-8B-Instruct (Dubey et al., 2024), Qwen-2.5-7B-Instruct (Yang et al., 2024), Minstral-8B-Instruct (Jiang et al., 2024), and Gemma-2-9B-Instruct (Team et al., 2024)—which collectively generate 60K dialogues and filter out outliers, as mentioned in §2.4. We ensure that every model processes the same user pairs in the identical order. Dialogues where the model refuses to respond or produces repetitive utterances are discarded, resulting in approximately 58K valid dialogues. We provide a detailed distribution of dialogues by pairing scenario and model in Appendix F. Finally, we apply four consistency metrics and four coherence metrics to all dialogues, aggregating the results by pairing type and LLM. For utilizing GPT-4o as backbone LLM, we report the results in Appendix C.

3.1.2 Results & Analysis
LLM Generation Diverges with Polarity Configurations As shown in Table 3, dialogues produced under the Negative pairing configuration yield higher C scores and Contd. values than those under the Positive pairing. Specifically, although the gap between entailment and contradiction cases is greater for Negative pairings, the proportion of contradictions is also higher. Moreover, coherence is generally lower, implying that the model overemphasizes persona-related content. In such scenarios, frequent entailment goes hand in hand with frequent contradiction, reducing overall coherence, even consistency (i.e. P gap, GPT score)

Conversely, Positive pairing results in fewer entailment cases and fewer contradictions, leading to very low Contd. scores and overall high coherence. This suggests that the model tends to forgo incorporating certain persona elements if they undermine dialogue flow. Meanwhile, the Mixed configuration combines both tendencies, placing most metrics
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Pairing</th>
    <th colspan="4">Consistency</th>
    <th colspan="4">Coherence</th>
  </tr>
  <tr>
    <th>C score ↑</th>
    <th>Contd. ↓</th>
    <th>P Gap ↓</th>
    <th>G-eval ↓</th>
    <th>Perp. ↓</th>
    <th>Q-DCE ↑</th>
    <th>PairEval ↑</th>
    <th>G-eval ↑</th>
  </tr>
  <tr>
    <td rowspan="5">LLaMa-3.1-8B</td>
    <td><i>Original</i></td>
    <td>0.391</td>
    <td>14.33</td>
    <td>-0.43</td>
    <td>4.28</td>
    <td>5.31</td>
    <td>3.14</td>
    <td>2.79</td>
    <td>4.39</td>
  </tr>
  <tr>
    <td><b>Negative</b></td>
    <td><b>0.444</b></td>
    <td><b>14.71</b></td>
    <td><b>-0.27</b></td>
    <td>4.27</td>
    <td>5.33</td>
    <td><b>3.01</b></td>
    <td><b>2.74</b></td>
    <td>4.57</td>
  </tr>
  <tr>
    <td><b>Positive</b></td>
    <td>0.428</td>
    <td><b>9.83</b></td>
    <td><b>-0.46</b></td>
    <td><b>4.44</b></td>
    <td><b>3.40</b></td>
    <td><b>3.17</b></td>
    <td><b>2.84</b></td>
    <td><b>4.65</b></td>
  </tr>
  <tr>
    <td>Mixed</td>
    <td>0.396</td>
    <td>13.95</td>
    <td>-0.34</td>
    <td>4.32</td>
    <td>5.37</td>
    <td>3.09</td>
    <td>2.77</td>
    <td>4.51</td>
  </tr>
  <tr>
    <td><i>Opposite</i></td>
    <td>0.352</td>
    <td>13.62</td>
    <td>-0.32</td>
    <td>4.20</td>
    <td>5.30</td>
    <td>3.09</td>
    <td>2.77</td>
    <td>4.47</td>
  </tr>
  <tr>
    <td rowspan="5">Qwen-2.5-7B</td>
    <td><i>Original</i></td>
    <td>0.392</td>
    <td>15.33</td>
    <td>-0.73</td>
    <td>4.50</td>
    <td>7.05</td>
    <td>3.06</td>
    <td>2.69</td>
    <td>4.34</td>
  </tr>
  <tr>
    <td><b>Negative</b></td>
    <td><b>0.520</b></td>
    <td>13.48</td>
    <td>-0.80</td>
    <td>4.55</td>
    <td><b>7.36</b></td>
    <td>3.07</td>
    <td>2.67</td>
    <td>4.27</td>
  </tr>
  <tr>
    <td><b>Positive</b></td>
    <td>0.452</td>
    <td><b>8.84</b></td>
    <td><b>-0.96</b></td>
    <td><b>4.67</b></td>
    <td><b>7.04</b></td>
    <td><b>3.14</b></td>
    <td><b>2.75</b></td>
    <td>4.38</td>
  </tr>
  <tr>
    <td>Mixed</td>
    <td>0.404</td>
    <td>12.99</td>
    <td>-0.82</td>
    <td>4.45</td>
    <td>7.09</td>
    <td>3.03</td>
    <td><b>2.70</b></td>
    <td><b>4.43</b></td>
  </tr>
  <tr>
    <td><i>Opposite</i></td>
    <td>0.409</td>
    <td>12.58</td>
    <td>-0.77</td>
    <td>4.33</td>
    <td>7.13</td>
    <td>3.02</td>
    <td>2.67</td>
    <td>4.24</td>
  </tr>
  <tr>
    <td rowspan="5">Minstral-8B</td>
    <td><i>Original</i></td>
    <td>0.555</td>
    <td>10.61</td>
    <td>-0.95</td>
    <td>4.38</td>
    <td>5.98</td>
    <td>3.11</td>
    <td>2.66</td>
    <td>4.11</td>
  </tr>
  <tr>
    <td><b>Negative</b></td>
    <td><b>0.778</b></td>
    <td>9.93</td>
    <td>-0.97</td>
    <td>4.36</td>
    <td>7.27</td>
    <td>3.11</td>
    <td><b>2.61</b></td>
    <td>3.95</td>
  </tr>
  <tr>
    <td><b>Positive</b></td>
    <td>0.595</td>
    <td><b>5.78</b></td>
    <td><b>-1.15</b></td>
    <td><b>4.51</b></td>
    <td><b>5.80</b></td>
    <td><b>3.16</b></td>
    <td><b>2.67</b></td>
    <td><b>4.21</b></td>
  </tr>
  <tr>
    <td>Mixed</td>
    <td>0.651</td>
    <td>9.65</td>
    <td>-0.80</td>
    <td>4.43</td>
    <td>6.06</td>
    <td>3.10</td>
    <td>2.62</td>
    <td>4.01</td>
  </tr>
  <tr>
    <td><i>Opposite</i></td>
    <td>0.540</td>
    <td>10.48</td>
    <td>-0.81</td>
    <td>4.27</td>
    <td>5.88</td>
    <td>3.08</td>
    <td>2.62</td>
    <td>3.92</td>
  </tr>
  <tr>
    <td rowspan="5">Gemma-2-9B</td>
    <td><i>Original</i></td>
    <td>0.391</td>
    <td><b>16.10</b></td>
    <td>-0.69</td>
    <td>4.33</td>
    <td>6.47</td>
    <td>3.09</td>
    <td>2.52</td>
    <td>3.91</td>
  </tr>
  <tr>
    <td><b>Negative</b></td>
    <td>0.423</td>
    <td>13.57</td>
    <td>-0.80</td>
    <td>4.35</td>
    <td>6.06</td>
    <td>3.08</td>
    <td><b>2.39</b></td>
    <td><b>3.77</b></td>
  </tr>
  <tr>
    <td><b>Positive</b></td>
    <td><b>0.465</b></td>
    <td><b>7.58</b></td>
    <td><b>-0.90</b></td>
    <td><b>4.45</b></td>
    <td>5.83</td>
    <td><b>3.16</b></td>
    <td><b>2.56</b></td>
    <td><b>4.07</b></td>
  </tr>
  <tr>
    <td>Mixed</td>
    <td>0.383</td>
    <td>12.86</td>
    <td>-0.77</td>
    <td>4.39</td>
    <td><b>5.62</b></td>
    <td>3.08</td>
    <td>2.44</td>
    <td>3.85</td>
  </tr>
  <tr>
    <td><i>Opposite</i></td>
    <td><b>0.322</b></td>
    <td>13.41</td>
    <td>-0.64</td>
    <td><b>4.19</b></td>
    <td>6.31</td>
    <td>3.12</td>
    <td>2.42</td>
    <td>3.81</td>
  </tr>
</table>

Table 1: Combined consistency and coherence results. Metrics with (↑) indicate higher is better, while those with (↓) indicate lower is better. Best scores per model and pairing are bolded and worst scores underlined. Highest values across models are in blue, lowest in red, with a gradient highlighting favorable trends as Negative, Positive Pairing, and Mixed scores increase with polarity.

somewhere between those of the Negative and Positive configurations as illustrated in the color gradients of Table 3. In summary, dialogues generated between Negative users strongly reflect each user’s persona—often at the expense of coherence and even consistency—whereas those between Positive users selectively apply persona elements, thus preserving both consistency and coherence.

Positive Configurations Yield Better Dialogues
As shown in Table 1, dialogues generated under the Positive pairing configuration exhibit consistently high consistency and coherence across nearly all tested models. Overall, except for a few exceptions, it achieves the highest scores across all pairing settings. This result indicates that providing an LLM with a Positive profile can produce high-quality dialogues—surpassing even those derived from Original pairing configuration—and reinforces the idea that LLMs are sensitive to user polarity.

However, in the Opposite pairing configuration, where a Positive Profile is paired with a Negative Profile, the resulting dialogues degrade to levels comparable to (or even worse than) those seen with Negative pairing. In other words, the advantage of the Positive Profile vanishes, further confirming that LLMs respond sensitively to user polarity.

Impact of Backbone Model Selection
Our analysis reveals that the choice of backbone model significantly influences dialogue characteristics beyond the pairing method. As shown in Table 1, both Minstral-8B and Qwen-2.5-7B tend to produce dialogues with higher consistency. In particular, Minstral-8B outperforms other models on three of the four consistency metrics, while Qwen-2.5-7B achieves the highest score on the G-eval metric. In contrast, LLaMa-3.1-8B excels in coherence, leading across all four coherence metrics. Meanwhile, Minstral-8B and Gemma-2-9B show lower coherence scores, suggesting that even though these models maintain consistency, they have difficulty generating fully coherent interactions. This observation is further confirmed by the variations seen in the G-eval outcomes.

Human Evaluation
We verify whether the trends identified by automatic metrics persist under human judgment by evaluating dialogues generated with the Original pairing, Negative pairing, and Positive pairing configurations.

As shown in Table 2, human raters corroborate the automatic findings: dialogues generated
<table>
  <tr>
    <th>Pairing</th>
    <th>Consistency</th>
    <th>Coherence</th>
  </tr>
  <tr>
    <td><i>Original</i></td>
    <td>2.36</td>
    <td>2.01</td>
  </tr>
  <tr>
    <td><i>Negative</i></td>
    <td>2.40</td>
    <td>2.12</td>
  </tr>
  <tr>
    <td><i>Positive</i></td>
    <td><b>2.51</b></td>
    <td><b>2.30</b></td>
  </tr>
</table>

Table 2: Human evaluation results on consistency and coherence among the sentiment of user profiles.

from polarized users—especially <i>Positive pairings</i>—exhibit the highest quality. Notably, the decline in scores for the <i>Original pairing</i> is far more pronounced in the human evaluation than in the automatic assessment, further confirming the validity of our observations.

3.2 Does Dialogue Quality Diverge According to Users’ Polarity Level?

In the previous section §3.1, we analyzed the impact of pairings between <i>polarized users</i>. However, a more detailed quantitative interpretation of user personas’ sentimental polarity is necessary to clarify whether the observed effects arise from user polarity or the pairing strategy. In this section, we address this uncertainty by examining dialogue outcomes across systematically varied polarity levels. We construct user profiles that span multiple degrees of polarity and investigate how dialogue consistency and coherence evolve in each scenario.

3.2.1 Experimental Setup

Defining Polarity Level  We aim to determine whether dialogues between users with strongly polarized personas (e.g., clearly positive or negative) differ from those with weak or neutral emotional content. We define each persona’s polarity level based on the confidence score produced by the classification model used in §2.2: values closer to 0 indicate a highly negative polarity, those closer to 1 indicate a highly positive polarity, and scores near 0.5 represent a neutral persona. Because half of all personas lie at sentimental extremes, uniformly dividing confidence scores would yield imbalanced distributions across levels. Instead, we partition the score range into nine intervals that ensure a more balanced representation of persona instances. Further details on the partition criteria and the distribution across levels are provided in Appendix F.

Experiment Pipeline  We fix the number of personas \( K \) to 1 in each user profile to minimize external factors such as the effect of combining or ordering multiple personas. The profile’s polarity level follows directly from that single persona, and we pair profiles of the same level together. For each level, we create 500 distinct user pairs and generate dialogues using LLaMa-3.1-8B and Qwen-2.5-7B models. We then compute the average consistency and coherence scores—measured respectively by C score and PairEval, for each level.

![Line graphs showing consistency and coherence across nine polarity levels](page_1012_184_393_393.png)

Figure 3: Consistency and coherence across nine polarity levels. Dot x-coordinates show the median confidence score (0: most negative, 1: most positive).

3.2.2 Results & Analysis

Stronger Sentimental Extremity in Persona Yields Higher-Quality Dialogue  As illustrated in Figure 3, both consistency and coherence scores exhibit a U-shaped trend with respect to polarity level. In other words, dialogues derived from highly polarized personas, whether strongly positive or strongly negative, tend to outperform those derived from more neutral personas. This pattern is consistent across all tested models, suggesting that personalizing a model with a strongly polarized persona enhances dialogue quality.

Positive Polarity Results in Particularly High-Quality Dialogue  Furthermore, dialogues generated from positively polarized profiles demonstrate superior consistency and coherence compared to those from negatively polarized profiles. Notably, the consistency score for dialogues produced by the LLaMa-3.1-8B model at the highest polarity level
is up to seven times greater than that at the lowest level, while the Qwen-2.5-7B model shows an average improvement of approximately 0.3 points in coherence. These findings indicate that positive polarity yielding especially pronounced gains.

We further analyze the underlying causes of the performance gap discussed in this section in Appendix A.

4 How to Make LLMs Robust to Polarity?

Our previous findings indicate that persona polarity influences dialogue quality in LLMs. This observation underscores the need for methods that leverage polarity traits to build a more robust personalized dialogue system. To that end, we conduct experiments using real-world profiles in an original pairing configuration to determine how to make LLMs robust to polarity.

4.1 Adjusting Generation Strategy

In addition to the dual-persona joint generation method, we adopt a turn-based agent conversation approach (§2). In this strategy, each turn is generated using a single user’s profile, thereby avoiding the complexities associated with mixed polarity inputs (§3.1.2) and allows for better handling of persona-specific characteristics. Although this approach requires a separate model call for each turn, our experiments show that models with around 3B parameters can generate high-quality dialogues without requiring extensive computational resources. Consequently, we employ two LLMs, LLaMa-3.2-3B and Qwen-2.5-3B, in this configuration to reduce latency and computational load while maintaining performance.

When applying the turn-based approach, it is necessary to specify a dialogue endpoint. For a fair comparison, we align the number of turns with the distribution observed in the joint-generation method. For instance, if joint generation produces 10 dialogues—6 of which have 8-turns and 4 of which have 10—then we assign an 8-turn endpoint with a 60% probability and a 10-turn endpoint with a 40% probability before initiating turn-based dialogue generation.

Additionally, generating utterances on a turn-by-turn basis sometimes results in excessively long responses that cluttered the overall dialogue. To address this, we add an instruction in the prompt to produce short and concise utterances. The detailed prompt is provided in Appendix E.

4.2 User Profile Ordering & Sentiment-aware Prompting

Analysis in §3 revealed that profiles containing negative or neutral personas produce lower-quality dialogues compared to those with exclusively positive personas. Since modifying user personas is undesirable, we propose a simple but effective polarity-based ordering technique to enhance dialogue performance without altering the original content. Inspired by (Wu et al., 2024a) showing that LLMs perform better when desired persona content appears early in the prompt, we design three ordering strategies based on the confidence score from §2.2:
• Score Ascending: Arrange personas in order of increasing confidence (more negative first).
• Score Descending: Arrange personas in order of decreasing confidence (more positive first).
• Center-Out Score Ascending: Sort by |(confidence score − 0.5)|, adding a bias factor of \( a = 0.05 \) for positive personas to prioritize negatives.
We apply this ordering only to the turn-based approach, as the joint generation method combines both profiles into a single prompt.

Plus, we concatenate a lightweight prompt template: “Please ensure that each user’s persona, especially negative or neutral personas, is well integrated into the dialogue and that the overall dialogue remains coherent”. It selectively conditions the model on the sentiment of personas present in a user profile at the end of the generation prompt. Because large language models already handle positive personas reliably, the template explicitly instructs the model to be aware when responding to negative or neutral personas.

4.3 Results & Analysis

Turn-based Approach Improves Overall Quality.
For the LLaMa-3.2-3B model, the turn-based approach significantly enhances dialogue consistency compared to dual-persona joint generation. This improvement stems from separating users’ polarized personas into distinct inputs, thereby reducing internal complexity. In contrast, the Qwen-2.5-3B model demonstrates lower G-eval scores primarily because it blends the personas of both users into a single dialogue. When User 1 initiates the conversation, its persona tends to dominate, leading User 2 to adopt a similar framing. For example, if
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Strategy</th>
    <th colspan="3">Consistency</th>
    <th colspan="2">Coherence</th>
  </tr>
  <tr>
    <th>C score</th>
    <th>Contd.</th>
    <th>G-eval</th>
    <th>PairEval</th>
    <th>G-eval</th>
  </tr>
  <tr>
    <td rowspan="6">LLaMa-3.2</td>
    <td>Joint</td>
    <td>0.371</td>
    <td>15.01</td>
    <td>4.15</td>
    <td>2.70</td>
    <td>4.50</td>
  </tr>
  <tr>
    <td>Turn-based</td>
    <td>0.609</td>
    <td>7.92</td>
    <td>4.14</td>
    <td>2.79</td>
    <td>4.56</td>
  </tr>
  <tr>
    <td>+ asc.</td>
    <td>0.610</td>
    <td>7.97</td>
    <td>4.13</td>
    <td>2.78</td>
    <td>4.65</td>
  </tr>
  <tr>
    <td>+ dsc.</td>
    <td>0.597</td>
    <td>8.09</td>
    <td>4.13</td>
    <td>2.77</td>
    <td>4.63</td>
  </tr>
  <tr>
    <td>+ c-asc.</td>
    <td>0.617</td>
    <td>7.39</td>
    <td>4.21</td>
    <td>2.79</td>
    <td>4.67</td>
  </tr>
  <tr>
    <td>+ sap.</td>
    <td>0.688</td>
    <td>6.56</td>
    <td>4.18</td>
    <td>2.78</td>
    <td>4.59</td>
  </tr>
  <tr>
    <td>+ c-sap.</td>
    <td><b>0.717</b></td>
    <td><b>6.07</b></td>
    <td><b>4.25</b></td>
    <td><b>2.84</b></td>
    <td><b>4.68</b></td>
  </tr>
  <tr>
    <td rowspan="6">Qwen-2.5</td>
    <td>Joint</td>
    <td>0.470</td>
    <td>11.68</td>
    <td>4.32</td>
    <td>2.62</td>
    <td>4.36</td>
  </tr>
  <tr>
    <td>Turn-based</td>
    <td>0.557</td>
    <td>10.45</td>
    <td>4.02</td>
    <td>2.65</td>
    <td>4.60</td>
  </tr>
  <tr>
    <td>+ asc.</td>
    <td>0.557</td>
    <td>10.45</td>
    <td>3.99</td>
    <td>2.69</td>
    <td>4.69</td>
  </tr>
  <tr>
    <td>+ dsc.</td>
    <td>0.535</td>
    <td>10.99</td>
    <td>4.01</td>
    <td>2.67</td>
    <td>4.69</td>
  </tr>
  <tr>
    <td>+ c-asc.</td>
    <td>0.570</td>
    <td>10.07</td>
    <td>4.08</td>
    <td>2.69</td>
    <td>4.71</td>
  </tr>
  <tr>
    <td>+ sap.</td>
    <td><b>0.777</b></td>
    <td><b>8.27</b></td>
    <td><b>4.58</b></td>
    <td><b>2.61</b></td>
    <td><b>4.63</b></td>
  </tr>
  <tr>
    <td>+ c-sap.</td>
    <td>0.774</td>
    <td><b>7.49</b></td>
    <td><b>4.59</b></td>
    <td><b>2.69</b></td>
    <td><b>4.77</b></td>
  </tr>
</table>

Table 3: Dialogue consistency and coherence across generation strategies using LLaMa-3.2-3B and Qwen-2.5-7B with the original pairing. Abbreviations: asc. (score ascending), dsc. (score descending), c-asc. (center-out score ascending), sap. (sentiment-aware prompting), c-sap. (center-out score ascending + sentiment-aware prompting). Best results are in bold.

User 1’s persona is “I enjoy sewing”, the dialogue might begin with, “I heard you’re good at sewing”, prompting User 2 to continue discussing sewing. As a result, User 2’s responses inadvertently mix in User 1’s persona, while User 1 seldom reaffirms the persona initially designated for User 2. This behavior ultimately lowers the overall consistency score. Nonetheless, contradictions between personas and utterances actually decrease, boosting contradiction metrics (C score and Contd.). Also, we can resolve this issue by prompt engineering, which can be further explored.

Regarding coherence, the overall score improves for both models, with a notable enhancement in the Qwen-2.5-3B model. This outcome suggests that sequentially processing an updated dialogue context supports better integration of previously generated content.

Additional Profile Ordering is Helpful. As shown in Table 3, we show that the center-out ascending method yields modest yet consistent improvements across all five evaluation metrics compared to the other ordering strategies. In this approach, we arrange profiles so that those with moderate polarity appear first, while profiles with extreme negative traits are prioritized over clearly positive ones. This ordering leverages the model’s sensitivity to the arrangement of persona information (Wu et al., 2024a), enhancing personalization for ambiguous personas that are less effectively integrated (§3.2) while enabling well-incorporated positive personas (§3) to more robustly support the overall context.

Synergy between Profile Ordering and Sentiment-Aware Prompting Furthermore, as shown in Table 3, adding the sentiment-aware prompt alone produces some improvements for both models. More importantly, jointly applying the center-out score-ascending strategy and sentiment-aware prompting yields a strong synergy that markedly elevates dialogue quality. In particular, the coherence metric increases by around 0.1 points for both models, demonstrating substantial gains in conversational coherence. We attribute this improvement to positioning the personas that require heightened attention earlier in the prompt, thereby capitalizing on the uni-directional decoding of large language models in concert with the sentiment-aware instruction.

Human Evaluation We further manually compare turn-based generation with and without our profile ordering and sentiment-aware prompting techniques.

<table>
  <tr>
    <th>Strategy</th>
    <th>Consistency</th>
    <th>Coherence</th>
  </tr>
  <tr>
    <td>Turn-based</td>
    <td>1.80</td>
    <td>2.40</td>
  </tr>
  <tr>
    <td>+ c-sap.</td>
    <td>2.27<sup>(+0.47)</sup></td>
    <td>2.43<sup>(+0.03)</sup></td>
  </tr>
</table>

Table 4: Human evaluation results for consistency and coherence of dialogues generated by the turn-based approach, with and without profile ordering and sentiment-aware prompting.

As shown in Table 4, under the original pairing condition, applying our profile ordering and sentiment-aware prompting techniques to the turn-based generation improves consistency by 0.47 points and coherence by 0.03 points compared with the baseline. These observations further reinforce the robustness and efficacy of our proposed approach.

5 Related Works

5.1 Personalized Dialogue Generation

Personalized dialogue systems have increasingly focused on leveraging user-specific information for more contextually aligned interactions (Li et al., 2016; Zhang, 2018, 2019; Roller, 2020). Early approaches typically involved training generative models with VAE (Lee et al., 2022) to ensure dialogue coherence or NLI components (Song et al., 2021; Chen et al., 2023b; Zhou et al., 2023) to
capture persona representations. These persona representations often defined as descriptive sentences (Zhang, 2018; Dinan et al., 2020) or key-value attributes (Qian et al., 2017; Gao et al., 2023). With the advent of large language models and subsequent instruction tuning methods (Ouyang et al., 2022), persona information can now be more flexibly embedded directly into system prompts (Yang et al., 2023; Wang et al., 2023b).

Recent research has also advanced multi-agent (Park et al., 2023) or dual-persona (Xu et al., 2022; Jandaghi et al., 2023) strategies, enabling two distinct personas to converse within a single session. This approach enhances personalized-chatbot capabilities (Shuster et al., 2022; Lee et al., 2023) and supports large-scale synthetic data generation (Jandaghi et al., 2023) for further personalization.

5.2 Dialogue Evaluation

Evaluating open-domain dialogue is inherently multifaceted, reflecting diverse aspects such as coherence, fluency, and persona consistency (Wang et al., 2024; Samuel et al., 2024). Traditional metrics like ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002) often fail to capture higher-level qualities. Consequently, specialized metrics leveraging pretrained models—including C score (Madotto et al., 2019) (for consistency), QuantiDCE (Ye et al., 2021), and PairEval (Park et al., 2024) (for coherence)—have gained traction (Ghazarian et al., 2022; Li et al., 2024). We adopt these automated metrics for quantifiable evaluation.

In parallel, LLM-based evaluation strategies have rapidly emerged as a cost-effective alternative to human annotation (Chiang and Lee, 2023). Leveraging Chain-of-Thought prompting (Wei et al., 2022) further enhances evaluative transparency, allowing models to articulate their reasoning (Liu et al., 2023). In our work, we integrate both traditional and LLM-based metrics to comprehensively assess persona-driven dialogue.

5.3 Sentimental Sensitivity in LLMs

Large Language Models (LLMs) are known to be highly sensitive to a variety of factors, including prompt order, language, cultural context, and sentiment (Lu et al., 2021; Dang et al., 2024; Shen et al., 2024; Kwok et al., 2024). As a difference perspective from previous works, we focus on sentimental sensitivity in LLMs in our work. Although instruction tuning (Ouyang et al., 2022) and RLHF (Dang et al., 2024) can mitigate these effects, recent studies still show that contextual sentiment can strongly influence model outputs (Liu et al., 2024; Wu et al., 2024b). However, while much of the existing research focuses on sentiment that arises naturally in generated text, less work has considered sentimental polarity embedded naturally in explicit persona definitions. This underexplored avenue is central to our investigation, as it can profoundly affect both the coherence and consistency of persona-driven dialogues.

6 Conclusion

In this work, we demonstrate that LLMs are sensitive to the sentimental polarity of the persona during generating personalized dialogues. While positively polarized users yield smooth interactions, negatively polarized users overemphasize persona attributes and lead to lower coherence and even consistency. Furthermore, we find that personas with weak or neutral sentiment generally produce lower-quality dialogues. This findings suggest that the quality of the dialogue decreases if the user profile contains negative, or even neutral personas. Consequently, we introduce a dialogue generation approach that leverages persona polarity through a turn-based strategy with profile ordering, achieving more consistent and coherent outputs. Our findings underscore the importance of incorporating persona sentiment into personalized dialogue systems.

Limitation

Despite our extensive configurations designed to show that our observations are not confined to specific conditions, there remain areas for improvement. First, the source dataset used in our study is limited. To control for variations in LLM sensitivity that might arise from different domains or features of personas, we employed only a single dataset, focusing on sensitivity purely driven by sentimental polarity. Future work could expand upon our findings by employing alternative persona representations, including Sparse key-value attributes or User history in §2.1. Nonetheless, the ConvAI2 dataset we used contains a substantial volume of data, offering sufficient permutations of persona configurations to validate our research.

Second, the inherent bias of the backbone model. Automated evaluation metrics may not align with human perception if the underlying models themselves are biased. Similarly, if the polarity
classification model is biased, it could potentially misclassify a persona’s polarity based on certain keywords (e.g., those referencing race or culture). To mitigate such issues, we employ a broad set of diverse metrics to offset individual-model biases, and adopted confidence thresholds to classify polarity only when the model surpasses a pre-defined threshold (0.99). Furthermore, prior work indicates that BERT-based models exhibit fewer biases (Wang et al., 2023a), making complete inversions of our intuitive judgments relatively rare.

Finally, context length vaires along the configurations. Especially, negative polarity often involves negation words (e.g., “not,” “n’t”), potentially yielding longer sentences. However, Table 7 indicates that the overall difference in length is minimal. Additionally, dialogues generated by LLaMa-3.2-3B using a turn-based approach are notably longer, complicating direct comparisons. Nonetheless, it is noteworthy that these more extended utterances maintain high coherence, suggesting a promising direction for dialogue quality and consistency.

Ethics Statement

This research utilizes only fictional persona profiles from the ConvAI2 dataset, ensuring that the handling or storage of any real personal data is completely avoided. All dialogues generated by multiple large language models are entirely simulated; they represent virtual interactions created through algorithmic synthesis and not real human conversations. These simulated dialogues do not raise any privacy concerns, as they are devoid of actual personal data.

Acknowledgement

This research was supported by the Chung-Ang University Graduate Research Scholarship in 2025. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [RS-2021-II211341, Artificial Intelligence Graduate School Program (Chung-Ang University)].

References

Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.

Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.

Nuo Chen, Ning Wu, Shining Liang, Ming Gong, Linjun Shou, Dongmei Zhang, and Jia Li. 2023a. Is bigger and deeper always better? probing llama across scales and layers. CoRR.

Ruijun Chen, Jin Wang, Liang-Chih Yu, and Xuejie Zhang. 2023b. Learning to memorize entailment and discourse relations for persona-consistent dialogues. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 12653–12661.

Yi-Pei Chen, Noriki Nishida, Hideki Nakayama, and Yuji Matsumoto. 2024. Recent trends in personalized dialogue generation: A review of datasets, methodologies, and evaluations. arXiv preprint arXiv:2405.17974.

Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937.

John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. 2024. Rhf can speak many languages: Unlocking multilingual preference optimization for llms. arXiv preprint arXiv:2407.02552.

Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2020. The second conversational intelligence challenge (convai2). In The NeurIPS’18 Competition: From Machine Learning to Intelligent Conversations, pages 187–208. Springer.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Jingsheng Gao, Yixin Lian, Ziyi Zhou, Yuzhuo Fu, and Baoyuan Wang. 2023. Livechat: A large-scale personalized dialogue dataset automatically constructed from live streaming. arXiv preprint arXiv:2306.08401.

Sarik Ghazarian, Nuan Wen, Aram Galstyan, and Nanyun Peng. 2022. Deam: Dialogue coherence evaluation using amr-based semantic manipulations. arXiv preprint arXiv:2203.09711.
Narehan Hassan, Nur Athirah, and Rozilah Abdul Aziz. 2019. The influence of personality traits on communication competence. International Journal of Academic Research in Business and Social Sciences, 9.

Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543.

HF Canonical Model Maintainers. 2022. distilbert-base-uncased-finetuned-sst-2-english (revision bfdd146).

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Osstrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Pegah Jandaghi, XiangHai Sheng, Xinyi Bai, Jay Pujara, and Hakim Siddahmed. 2023. Faithful persona-based conversational dataset generation with large language models. arXiv preprint arXiv:2312.10007.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088.

Diederik P Kingma. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.

Louis Kwok, Michal Bravansky, and Lewis D Griffin. 2024. Evaluating cultural adaptability of a large language model via simulation of synthetic personas. arXiv preprint arXiv:2408.06929.

Jing Yang Lee, Kong Aik Lee, and Woon Seng Gan. 2022. Improving contextual coherence in variational personalized and empathetic dialogue agents. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7052–7056. IEEE.

Joosung Lee, Minsik Oh, and Donghun Lee. 2023. P5: Plug-and-play persona prompting for personalized response selection. arXiv preprint arXiv:2310.06390.

Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155.

Xue Li, Jia Su, Yang Yang, Zipeng Gao, Xinyu Duan, and Yi Guan. 2024. Dialogues are not just text: Modeling cognition for dialogue coherence evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18573–18581.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Qian Liu, Yihong Chen, Bei Chen, Jian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang. 2020. You impress me: Dialogue generation via mutual persona perception. arXiv preprint arXiv:2004.05388.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634.

Yang Liu, Xichou Zhu, Zhou Shen, Yi Liu, Min Li, Yujun Chen, Benzi John, Zhenzhen Ma, Tao Hu, Zhi Li, et al. 2024. Do large language models possess sensitive to sentiment? arXiv preprint arXiv:2409.02370.

Adian Liusie, Potsawee Manakul, and Mark JF Gales. 2023. Zero-shot nlg evaluation through pairwise comparisons with llms. arXiv preprint arXiv:2307.07889.

I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786.

Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale Fung. 2019. Personalizing dialogue agents via meta-learning. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 5454–5459.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

ChaeHun Park, Minseok Choi, Dohyun Lee, and Jaegul Choo. 2024. Paireval: Open-domain dialogue evaluation with pairwise comparison. arXiv preprint arXiv:2404.01015.

Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22.

Qiao Qian, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2017. Assigning personality/identity to a chatting machine for coherent conversation generation. arXiv preprint arXiv:1706.02861.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

S Roller. 2020. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637.

Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2023. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304.11406.

Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik Narasimhan, and Vishvak Murahari. 2024. Personagym: Evaluating persona agents and llms. arXiv preprint arXiv:2407.18416.

Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, and Rada Mihalcea. 2024. Understanding the capabilities and limitations of large language models for cultural commonsense. arXiv preprint arXiv:2405.04655.

Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, et al. 2022. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188.

Edward H Simpson. 1951. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological), 13(2):238–241.

Haoyu Song, Yan Wang, Kaiyan Zhang, Wei-Nan Zhang, and Ting Liu. 2021. Bob: Bert over bert for training persona-based dialogue models from limited personalized data. arXiv preprint arXiv:2106.06169.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussonet, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Yu-Ching Hsu, Jia-Yin Foo, Chao-Wei Huang, and Yun-Nung Chen. 2024. Two tales of persona in llms: A survey of role-playing and personalization. arXiv preprint arXiv:2406.01171.

Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuandong Tian. 2024. Learning personalized alignment for evaluating open-ended text generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13274–13292.

Yiwei Wang, Yujun Cai, Muhao Chen, Yuxuan Liang, and Bryan Hooi. 2023a. Primacy effect of chatgpt. arXiv preprint arXiv:2310.13206.

Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhun Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, et al. 2023b. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2018. Dialogue natural language inference. arXiv preprint arXiv:1811.00671.

Bin Wu, Zhengyan Shi, Hossein A Rahmani, Varsha Ramineni, and Emine Yilmaz. 2024a. Understanding the role of user profile in the personalization of large language models. arXiv preprint arXiv:2406.17803.

Zhen Wu, Ritam Dutt, and Carolyn Rose. 2024b. Evaluating large language models on social signal sensitivity: An appraisal theory approach. In Proceedings of the 1st Human-Centered Large Language Modeling Workshop, pages 67–80.

Chen Xu, Piji Li, Wei Wang, Haoran Yang, Siyun Wang, and Chuangbai Xiao. 2022. Cosplay: Concept set guided personalized dialogue generation across both party personas. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 201–211.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Fan Yang, Zheng Chen, Ziyuan Jiang, Eunah Cho, Xiaojiang Huang, and Yanbin Lu. 2023. Palr: Personalization aware llms for recommendation. arXiv preprint arXiv:2305.07622.

Zheng Ye, Liucun Lu, Lishan Huang, Liang Lin, and Xiaodan Liang. 2021. Towards quantifiable dialogue coherence evaluation. arXiv preprint arXiv:2106.00507.

Saizheng Zhang. 2018. Personalizing dialogue agents: I have a dog, do you have pets too. arXiv preprint arXiv:1801.07243.

Y Zhang. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536.

Junkai Zhou, Liang Pang, Huawei Shen, and Xueqi Cheng. 2023. Simoap: Improve coherence and consistency in persona-based dialogue generation via over-sampling and post-evaluation. arXiv preprint arXiv:2305.11130.
A Why Are Large Language Models Sensitive to Persona Sentiment?

We have observed the performance gap across different persona polarities in §3. As we utilize an LLM without any additional fine-tuning, we could infer that LLMs inherently exhibit sensitivity to persona sentiment. Consequently, we explain that this sensitivity is mainly due to the scarcity of negative expressions in the pretraining data, which may have been intentionally limited to prevent toxicity. And Figure 2 clearly illustrates a lower proportion of negative personas compared to other polarities. Such data imbalance is evident even in widely-used foundational datasets like ConVAI2, suggesting similar limitations in collecting sufficient negative persona data in other datasets or real-world contexts. Additionally, we also find that LLM’s sensitivity can stem from post-training techniques that steer models toward generating responses preferred by humans. For the experiment, we sample 1,000 dialogues per model under both the negative and positive configurations described in §3.1.1. For the neutral configuration, we follow a similar approach by grouping five intermediate-stage personas from Figure 3 into pairs, generating 1,000 dialogues for each model accordingly.

<table>
  <tr>
    <th>Pairing</th>
    <th>Persona</th>
    <th>Dialogue</th>
  </tr>
  <tr>
    <td>Negative</td>
    <td>1.71</td>
    <td>2.67 (+0.96)</td>
  </tr>
  <tr>
    <td>Positive</td>
    <td>4.25</td>
    <td>4.52 (+0.27)</td>
  </tr>
  <tr>
    <td>Neutral</td>
    <td>3.06</td>
    <td>4.09 (+1.03)</td>
  </tr>
</table>

Table 5: Average sentiment scores (1–5 Likert) for dialogues and their personas, rated by GPT-4o. Lower scores denote negativity, higher positivity; red values mark the dialogue’s sentiment gain over its persona.

As shown in Table 5, regardless of model type, we observe a consistent tendency to produce more positive dialogues than the assigned personas. Models consistently generate more positive dialogues, with the gap widening when dialogue quality decreases. This trend suggests that post-training practices bias models toward positive outputs, leading to confusion when persona sentiment diverges significantly, lowering dialogue quality.

B Which Factors Influence the Polarity-Based Trend?

In §3, we observe that dialogues between negative users tend to yield higher C score and Contd. values while remaining relatively incoherent, whereas dialogues from positive users generally exhibit lower Contd. values and improved coherence. This section investigates whether these trends persist under variations in key experimental factors or if new patterns emerge. Specifically, we examine the influence of: (1) the number of personas \( K \) in a user profile, (2) the polarity ratio within a user profile, and (3) the model size.

B.1 Varying Persona Count

Experiment Pipeline We vary the value of number of personas \( K \) in each user profile to 1, 2, and 10, then examine how these changes affect consistency and coherence in dialogues. Adopting the method from §2.2, we generate a total of 1K user profiles per polarity and create 1K user pairs according to the Negative, Positive, and Mixed Pairing strategies described in §3. When each profile contains only one persona, Mixed Pairing is not feasible; we therefore use Opposite Pairing in that case. We measure consistency with the C score and Contd. and coherence with the PairEval score.

Positive Pairing Exhibits a Direct Positive Relationship between K and Dialogue Quality As shown in Figure 4, dialogues generated under Positive pairing achieve the highest consistency and coherence scores except for C score in K=5 setting. Specifically, Figure 4a reveals a positive correlation between K and the C score for Positive Pairing, despite of growing Contd. values. It suggests that as the number of positive personas grows, LLMs can still effectively incorporate them while suppressing the increase in the number of contradictions. Interestingly, as illustrated in Figure 4b, coherence also improves with more personas in Positive Pairing, contrary to our initial assumption that additional personas might artificially distort the conversation context.

Negative and Mixed Pairing Show No Clear Improvement In contrast, Negative and Mixed Pairings exhibit only marginal gains in consistency and coherence as \( K \) increases. Notably, when \( K \) is either small or very large, contradictions accumulate faster than entailments, causing the C score to drop below that of positive pairings. In particular, Mixed Pairing displays a declining trend in both metrics, presumably because the model struggles to integrate multiple personas polarized at opposite extremes.
(a) C score and Contd. for Scaled K Values. The solid lines represent the trend in C score, while the dotted lines indicate the trend in Contd.

(b) Paireval Score for Scaled K values.

Figure 4: Consistency and Coherence Comparison Between Pairing Recipe. K is set to 1, 2, 5, and 10. We use Llama-3.1-8B for backbone LLM.

B.2 Impact of Polarity Ratio

Experiment Pipeline Next, we fix K to 5 and vary only the proportion of positive personas to assess its influence on dialogue generation. Building on the Mixed Pairing setup in §3, we categorize user profiles by their positive-persona ratios and measure the consistency of the resulting dialogues with the C score and Contd. Since each paired user may have a different proportion of positive personas, we calculate consistency of the dialogue separately for each user, then average the scores across each ratio category. We use two backbone LLMs, Llama-3.1-8B and Minstral-8B, which yield highly consistent dialogue in §3.1.

A Notable Anomaly in Mixed Ratios As shown in Figure 5, the C score decreases as the proportion of positive personas grows, in line with our initial expectations. However, Contd. does not steadily decline; it first rises and then starts to drop once the positive ratio exceeds 50%. This pattern consistently appears in both models, implying that when a profile contains negative personas mixed with a slightly smaller share of positive personas, the model becomes more overemphasized to incorporate the user persona. Nonetheless, because the distribution of data across different positive ratios is uneven and our sample size is relatively small, these findings serve only as preliminary observations rather than definitive conclusions.

Figure 5: C Score by Positive Persona Ratio. Using LLaMa-3.1-8B and Minstral-8B, higher positive ratios correlate with improved dialogue consistency.

tently appears in both models, implying that when a profile contains negative personas mixed with a slightly smaller share of positive personas, the model becomes more overemphasized to incorporate the user persona. Nonetheless, because the distribution of data across different positive ratios is uneven and our sample size is relatively small, these findings serve only as preliminary observations rather than definitive conclusions.

B.3 Effect of Model Size

Experiment Pipeline To determine whether polarity-based trends remain consistent across different model sizes, we employ various Qwen-2.5 models ranging from 0.5B to 3B, 7B, 14B, and 32B, alongside the default 7B model. For practical purposes, we sample 1K user pairs from the pool using the Negative Pairing, Positive Pairing, and Mixed Pairing strategies in §3, ensuring identical inputs for each model. The dialogue metrics follow those in AppendixB.1.

Polarity-Based Trends Fluctuate with Model Size In §3.1, we discussed why dialogues between negative users tend to yield higher C score and Contd. values yet remain relatively incoherent. By contrast, positive-user dialogues generally show lower Contd. values and higher coherence. As illustrated in Figure 6a, these tendencies shift with model size. When the model is 3B or smaller (i.e., 0.5B or 3B), negative-user dialogues exhibit markedly low C score and high Contd., indicating that smaller-scale models handle negative personas less effectively than positive personas. However, once the model size surpasses 7B (e.g., 7B, 14B, 32B), the original pattern reemerges, suggesting that while larger models gain the capacity to process negative personas more accurately, they continue to overemphasize them in the dialogue.
(a) C score and Contd. for Different Model Sizes. The solid lines represent the trend in C score, while the dotted lines indicate the trend in Contd.

(b) Paireval Score for Scaling the Model Size.

Figure 6: Consistency and Coherence Comparison Between Model Size. We use Qwen-2.5-0.5B, 3B, 7B, 14B and 32B for various scaled models.

Dialogue Quality Does Not Always Increase with Model Size   We initially hypothesized that a larger model would generate higher consistency and coherence; however, the experimental results do not fully support this assumption. For consistency, the 3B model actually achieves the highest average C score and Contd., irrespective of user polarity. Although performance improves somewhat as model size scales from 7B to 32B, the gains are not substantial relative to the increase in size. In terms of coherence, the performance peaks at 7B and then declines at larger scales. These findings align with (Chen et al., 2023a)’s conclusion that model size does not necessarily translate into higher downstream task performance.

C GPT-4o Results

We extended our analysis by incorporating GPT-4o—the leading closed-source LLM in many chatbot benchmarks—into our backbone set to test whether the phenomenon identified in RQ1 persists. The experimental setup mirrors that described in §3.1.1.

As shown in Table 6, the same pattern emerges with GPT-4o: the positive-pairing configuration yields the highest dialogue quality across all metrics, whereas configurations containing negative pairings or mixed sentiment polarities perform noticeably worse. In particular, the original-pairing configuration—which reflects real-world distributions—suffers a larger quality drop than we observed with open-source LLMs. This result reinforces the robustness of our findings and warns of potential, unforeseen performance degradation in downstream applications.

<table>
  <tr>
    <th rowspan="2">Pairing</th>
    <th colspan="3">Consistency</th>
    <th colspan="2">Coherence</th>
  </tr>
  <tr>
    <th>C score ↑</th>
    <th>Contd. ↓</th>
    <th>G-eval ↑</th>
    <th>PairEval ↑</th>
    <th>G-eval ↑</th>
  </tr>
  <tr>
    <td>Original</td>
    <td>0.385</td>
    <td>7.92</td>
    <td>4.14</td>
    <td>2.79</td>
    <td>4.56</td>
  </tr>
  <tr>
    <td>Negative</td>
    <td>0.626</td>
    <td>11.11</td>
    <td>4.27</td>
    <td>2.75</td>
    <td>4.44</td>
  </tr>
  <tr>
    <td>Positive</td>
    <td><b>0.651</b></td>
    <td><b>4.12</b></td>
    <td><b>4.42</b></td>
    <td><b>2.83</b></td>
    <td><b>4.69</b></td>
  </tr>
  <tr>
    <td>Mixed</td>
    <td>0.638</td>
    <td>8.60</td>
    <td>4.32</td>
    <td>2.76</td>
    <td>4.54</td>
  </tr>
  <tr>
    <td>Opposite</td>
    <td>0.589</td>
    <td>8.64</td>
    <td>4.23</td>
    <td>2.75</td>
    <td>4.44</td>
  </tr>
</table>

Table 6: Combined consistency and coherence results of the dialogues generated by GPT-4o.

D Evaluation

D.1 Consistency

C score   C score (Madotto et al., 2019) is a widely used metric for assessing the consistency of personalized utterances. Let the user profile for user \( n \) be \( U^{(n)} \) and its constituent personas be \( p_i^{(n)} \). Similarly, denote the dialogue by \( D \), composed of utterances \( u_i^{(n)} \) from user \( n \). We feed these persona-utterance pairs into a fine-tuned NLI model to obtain entailment scores, then sum these values for each utterance across all personas:

\[
U^{(n)} = \{ p_1^{(n)}, p_2^{(n)}, \ldots, p_k^{(n)} \} \quad ,
\]
\[
D = \{ u_1^{(1)}, u_1^{(2)}, u_2^{(1)}, u_2^{(2)}, \ldots \}
\]
\[
\mathrm{NLI}(u_i^{(n)}, p_j^{(n)}) =
\begin{cases}
1 & \text{entailment}, \\
0 & \text{neutral}, \\
-1 & \text{contradiction}.
\end{cases}
\]
\[
C(u_i^{(n)}) = \sum_{j=1}^k \mathrm{NLI}(u_i^{(n)}, p_j^{(n)})
\]

We fine-tune a BERT-large (Devlin, 2018) model on the DialogNLI dataset (Welleck et al., 2018) by utilizing AdamW optimizer (Loshchilov, 2017) for NLI, achieving near-state-of-the-art accuracy (88.9%) on the test set, using a learning rate of \( 1 \times 10^{-5} \) and a batch size of 16.

Originally, the C score was designed to measure the consistency of a single response given a dialogue context. To adapt it for entire dialogues, we
split the dialogue into individual utterances, match each utterance of speaker \( n \) to its user profile \( U^{(n)} \), compute all persona-utterance entailment scores, and sum them. We then average these sums across all utterances to obtain the dialogue-level C score:

\[
C(D) = \frac{1}{|D|} \sum_i \left( C(u_i^{(1)}) + C(u_i^{(2)}) \right),
\]

where \( |D| \) is the total number of turns in the dialogue.

Contd. Because the C score alone may be susceptible to Simpson’s Paradox (Simpson, 1951), we propose the Contradiction Ratio (Contd.) to provide a more detailed distribution of entailments versus contradictions. For example, suppose the entailment scores for a particular utterance across personas are [0, 0, 0, 1, 0] and [1, 1, 1, −1, 0]. These yield C scores of 1 and 2, respectively, even though the latter contains a contradiction. Consequently, Contd. measures the percentage of contradictions out of all entailments or contradictions. Formally:

\[
\# \text{ of Contradictions (C\#)} = \sum_{n=1}^2 \sum_i \sum_{j=1}^k 1\left( \mathrm{NLI}(u_i^{(n)}, p_j^{(n)}) = -1 \right),
\]
\[
\# \text{ of Entailments (E\#)} = \sum_{n=1}^2 \sum_i \sum_{j=1}^k 1\left( \mathrm{NLI}(u_i^{(n)}, p_j^{(n)}) = 1 \right).
\]
\[
\text{Contd.} = \left( \frac{\text{C\#}}{\text{C\#} + \text{E\#}} \right) \times 100,
\]

where \( 1(\cdot) \) is the indicator function.

D.2 Coherence

Q-DCE & PairEval Traditional reference-based metrics (e.g., BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005)) often fail to correlate well with human judgments of dialogue coherence (Ghazarian et al., 2022; Li et al., 2024), inspiring the emergence of reference-free automatic evaluation. Among these, Q-DCE uses a ranking-based loss function to pre-train an encoder model (Devlin, 2018), then applies knowledge distillation to produce quantitative coherence scores on a scale of 1 to 5.

With the rise of powerful LLMs, reference-free evaluation has increasingly shifted toward large language models (Liusie et al., 2023). We adopt PairEval (Park et al., 2024), wherein a fine-tuned LLM compares a target response against alternative responses given the same dialogue context. However, these studies often focus on single-turn evaluations, so we treat every utterance from the second to the last as a candidate response. All preceding utterances are considered the dialogue context for that response. Formally, let \( D = \{ u_1, u_2, \ldots, u_N \} \) represent the dialogue, and let \( M(c, u) = s \) denote an automated evaluation model that produces a coherence score \( s \) for a response \( u \) given context \( c \). We define the coherence of the dialogue Coh(\( D \)) as:

\[
\mathrm{Coh}(D) = \frac{1}{N-1} \sum_{i=2}^N M(u_{1:i-1}, u_i).
\]

E Prompt Templates

In Figure 7, we present the prompt templates for the two generation strategies introduced in §2.4 and employed in §4. Left part shows the LLM prompt used in §3 and §4 whereas right part illustrates the LLM prompt applied in §4. In the latter, two LLMs are personalized with distinct user profiles, and additional instructions are included to limit the length of the responses.

Furthermore, Figure 8 depicts the GPT evaluation prompt template introduced in §2.5. Both templates provide detailed scoring criteria to facilitate more accurate evaluations by the models. Since it was common for dialogues to receive a perfect score of 5, we added the instruction “BE STRICT TO YOUR EVALUATION” to promote a wider distribution of scores. Additionally, to distinguish our approach from conventional metrics, a stronger penalty for contradictions was imposed.

F Dialogue Statistics

Table 7 presents statistics for the ConvAI2 source dataset, the user profiles constructed in §2.2, and the datasets used in experiments addressing each research question. The symbol “*” indicates that the omitted value deviates only marginally from the reported statistic, while “—” is used when a statistic is either difficult to compute or not applicable. For §3.1, only statistics for Llama-3.1-8B are provided for convenience, as its trends are consistent with those observed for §3.2.
LLM
System Prompt:
User 1: {User Profile 1}
User 2: {User Profile 2}

User Prompt:
Generate a dialogue between User1 and User 2.

LLM 1
System Prompt:
User 1: {User Profile 1}

User Prompt:
You are User 1. Generate the next line of dialogue based on the previous context in 1~2 short sentences. Dialogue context: {Dialogue}

LLM 2
System Prompt:
User 2: {User Profile 2}

User Prompt:
You are User 2. Generate the next line of dialogue based on the previous context in 1~2 short sentences. Dialogue context: {Dialogue}

Figure 7: Prompt Template with Simple Framework Illustration for Two Generation Strategies. First one indicates dual-persona joint generation, and second one indicates turn-based approach.
You will be given two of user persona descriptions and a dialogue between these users.
Your task is to rate the dialogue on one metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:
Consistency (1-5) - the factual alignment between the Persona descriptions and the utterances in the dialogue. A factually consistent utterance contains only statements that are entailed by the source persona descriptions. Annotators were also asked to penalize dialogue that contained utterances contradicted by Persona descriptions.

- 1: Dialogue misses all persons in Persona descriptions, or contains many utterances contradicted by Persona descriptions.
- 2: Dialogue misses three or four persons in Persona descriptions, or contains several utterances contradicted by Persona descriptions.
- 3: Dialogue misses two persons in Persona descriptions, or contains few utterances contradicted by Persona description.
- 4: Dialogue misses one person in Persona descriptions.
- 5: Dialogue reflects all of Persona descriptions perfectly.

Evaluation Steps:
1. Read the persona descriptions carefully and identify the main facts and details it presents.
2. Read the dialogue and compare it to the descriptions. Check if the dialogue contains any utterances contradicted by given descriptions.
3. Assign a score for consistency based on the Evaluation Criteria. BE STRICT IN YOUR EVALUATION.
4. Justify the rating by referring to specific aspects of the conversation that demonstrate its coherence or lack thereof.

Example:
Persona descriptions:
User 1 | [User Profile 1]
User 2 | [User Profile 2]

Dialogue:
[dialogue]

Evaluation Form (scores ONLY):
- Consistency:

(a) G-eval Prompt for Evaluating Consistency.

You will be given a pair of user personas. You will then be given one conversation between this persona pair.
Your task is to rate the conversation on one metric. Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:
Coherence (1-5) - the collective quality of all utterances. The conversation should be well-structured and well-organized. The conversation should not just be a heap of related information, but should build from utterance to a coherent body of conversation about a topic and previous context.

- 1: All utterances in dialogue are unrelated to each other, or the dialogue contains many utterances contradicted to the previous context.
- 2: Dialogue contains some utterances related to the previous context.
- 3: Dialogue contains most utterances unrelated to the previous context.
- 4: Dialogue is somewhat fluent, but the flow of the topic is not smooth.
- 5: Dialogue is perfectly fluent and well-organized.

Evaluation Steps:
1. Read and understand the given conversation.
2. Evaluate the conversation based on the coherence of the utterances.
3. Rate the conversation on a scale of 1 to 5 based on Evaluation Criteria. BE STRICT IN YOUR EVALUATION.
4. Justify the rating by referring to specific aspects of the conversation that demonstrate its coherence or lack thereof.

Example:
Conversation: [dialogue]

Evaluation Form (scores ONLY):
- Coherence:

(b) G-eval Prompt for Evaluating Coherence

Figure 8: G-eval Prompts for Evaluating Dialogue Quality.
<table>
  <tr>
    <th>Source</th>
    <th>Personas (profiles)</th>
    <th># of samples</th>
    <th># profile words</th>
    <th># dialogue turns</th>
    <th># dialogue words</th>
  </tr>
  <tr>
    <td rowspan="10">ConvAI2</td>
    <td>unique profiles</td>
    <td>2310</td>
    <td>63.34</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>unique personas</td>
    <td>6126</td>
    <td>6.59</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>negative personas (confidence score \( \leq 0.01 \))</td>
    <td>1006</td>
    <td>7.08</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>0.01 < confidence \( \leq 0.1 \)</td>
    <td>652</td>
    <td>6.88</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>0.1 < confidence \( \leq 0.2 \)</td>
    <td>158</td>
    <td>7.02</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>0.2 < confidence \( \leq 0.4 \)</td>
    <td>151</td>
    <td>6.53</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>0.4 < confidence \( \leq 0.6 \)</td>
    <td>150</td>
    <td>6.63</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>0.6 < confidence \( \leq 0.8 \)</td>
    <td>194</td>
    <td>6.47</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>0.8 < confidence \( \leq 0.9 \)</td>
    <td>191</td>
    <td>6.76</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>0.9 < confidence \( \leq 0.99 \)</td>
    <td>933</td>
    <td>6.57</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>positive personas (confidence score > 0.99)</td>
    <td>2091</td>
    <td>6.42</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="2">Synthesized user profiles</td>
    <td>Negative Profiles</td>
    <td>10000</td>
    <td>35.68</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Mixed Profiles</td>
    <td>10000</td>
    <td>34.65</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Profiles Profiles</td>
    <td>10000</td>
    <td>35.07</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="18">$3.1$</td>
    <td>Original pairings (llama)</td>
    <td>2296</td>
    <td>63.27</td>
    <td>16.01</td>
    <td>414.09</td>
  </tr>
  <tr>
    <td>Negative pairings (llama)</td>
    <td>2898</td>
    <td>70.26</td>
    <td>14.53</td>
    <td>390.15</td>
  </tr>
  <tr>
    <td>Mixed pairings (llama)</td>
    <td>2879</td>
    <td>69.34</td>
    <td>15.36</td>
    <td>417.49</td>
  </tr>
  <tr>
    <td>Positive pairings (llama)</td>
    <td>2864</td>
    <td>67.29</td>
    <td>14.86</td>
    <td>423.67</td>
  </tr>
  <tr>
    <td>Opposite pairings (llama)</td>
    <td>2826</td>
    <td>69.47</td>
    <td>15.82</td>
    <td>411.73</td>
  </tr>
  <tr>
    <td>Original pairings (qwen)</td>
    <td>2301</td>
    <td>**</td>
    <td>10.42</td>
    <td>355.75</td>
  </tr>
  <tr>
    <td>Negative pairings (qwen)</td>
    <td>2987</td>
    <td>**</td>
    <td>10.05</td>
    <td>393.20</td>
  </tr>
  <tr>
    <td>Mixed pairings (qwen)</td>
    <td>2916</td>
    <td>**</td>
    <td>10.43</td>
    <td>390.12</td>
  </tr>
  <tr>
    <td>Positive pairings (qwen)</td>
    <td>2851</td>
    <td>**</td>
    <td>10.46</td>
    <td>374.32</td>
  </tr>
  <tr>
    <td>Opposite pairings (qwen)</td>
    <td>2895</td>
    <td>**</td>
    <td>10.52</td>
    <td>381.12</td>
  </tr>
  <tr>
    <td>Original pairings (minstral)</td>
    <td>2288</td>
    <td>**</td>
    <td>10.73</td>
    <td>287.17</td>
  </tr>
  <tr>
    <td>Negative pairings (minstral)</td>
    <td>2940</td>
    <td>**</td>
    <td>9.88</td>
    <td>299.77</td>
  </tr>
  <tr>
    <td>Mixed pairings (minstral)</td>
    <td>2851</td>
    <td>**</td>
    <td>10.10</td>
    <td>295.07</td>
  </tr>
  <tr>
    <td>Positive pairings (minstral)</td>
    <td>2790</td>
    <td>**</td>
    <td>10.13</td>
    <td>299.75</td>
  </tr>
  <tr>
    <td>Opposite pairings (minstral)</td>
    <td>2801</td>
    <td>**</td>
    <td>10.52</td>
    <td>299.20</td>
  </tr>
  <tr>
    <td>Original pairings (gamma)</td>
    <td>2167</td>
    <td>**</td>
    <td>13.66</td>
    <td>227.85</td>
  </tr>
  <tr>
    <td>Negative pairings (gamma)</td>
    <td>2710</td>
    <td>**</td>
    <td>13.43</td>
    <td>233.62</td>
  </tr>
  <tr>
    <td>Mixed pairings (gamma)</td>
    <td>2793</td>
    <td>**</td>
    <td>13.97</td>
    <td>239.38</td>
  </tr>
  <tr>
    <td>Positive pairings (gamma)</td>
    <td>2863</td>
    <td>**</td>
    <td>12.72</td>
    <td>230.19</td>
  </tr>
  <tr>
    <td>Opposite pairings (gamma)</td>
    <td>2672</td>
    <td>**</td>
    <td>15.12</td>
    <td>236.88</td>
  </tr>
  <tr>
    <td>Negative pairings (K=1) (llama)</td>
    <td>500</td>
    <td>13.61</td>
    <td>10.29</td>
    <td>281.76</td>
  </tr>
  <tr>
    <td>(0.01 < conf \( \leq 0.1 \)) pairings (llama)</td>
    <td>500</td>
    <td>13.64</td>
    <td>11.46</td>
    <td>289.12</td>
  </tr>
  <tr>
    <td>(0.1 < conf \( \leq 0.2 \)) pairings (llama)</td>
    <td>500</td>
    <td>14.36</td>
    <td>11.40</td>
    <td>275.91</td>
  </tr>
  <tr>
    <td>(0.2 < conf \( \leq 0.4 \)) pairings (llama)</td>
    <td>500</td>
    <td>13.41</td>
    <td>11.27</td>
    <td>228.41</td>
  </tr>
  <tr>
    <td>(0.4 < conf \( \leq 0.8 \)) pairings (llama)</td>
    <td>500</td>
    <td>12.84</td>
    <td>10.95</td>
    <td>261.1</td>
  </tr>
  <tr>
    <td>(0.6 < conf \( \leq 0.8 \)) pairings (llama)</td>
    <td>500</td>
    <td>13.49</td>
    <td>11.45</td>
    <td>231.23</td>
  </tr>
  <tr>
    <td>(0.8 < conf \( \leq 0.9 \)) pairings (llama)</td>
    <td>500</td>
    <td>13.02</td>
    <td>11.27</td>
    <td>253.59</td>
  </tr>
  <tr>
    <td>(0.9 < conf \( \leq 0.99 \)) pairings (llama)</td>
    <td>500</td>
    <td>12.96</td>
    <td>12.64</td>
    <td>241.78</td>
  </tr>
  <tr>
    <td>Positive pairings (K=1) (llama)</td>
    <td>500</td>
    <td>12.87</td>
    <td>9.72</td>
    <td>284.83</td>
  </tr>
  <tr>
    <td rowspan="14">$4$</td>
    <td>Original pairing (llama-3B)</td>
    <td>2203</td>
    <td>63.26</td>
    <td>10.33</td>
    <td>231.17</td>
  </tr>
  <tr>
    <td>Original pairing (qwen-3B)</td>
    <td>2254</td>
    <td>**</td>
    <td>11.55</td>
    <td>246.13</td>
  </tr>
  <tr>
    <td>Original pairing (llama-turn-based)</td>
    <td>2032</td>
    <td>**</td>
    <td>17.68</td>
    <td>1020.30</td>
  </tr>
  <tr>
    <td>Original pairing (qwen-turn-based)</td>
    <td>2190</td>
    <td>**</td>
    <td>17.03</td>
    <td>362.83</td>
  </tr>
  <tr>
    <td>Original pairing (llama-turn-based + ascending)</td>
    <td>2056</td>
    <td>**</td>
    <td>17.26</td>
    <td>1048.87</td>
  </tr>
  <tr>
    <td>Original pairing (llama-turn-based + descending)</td>
    <td>2060</td>
    <td>**</td>
    <td>17.86</td>
    <td>1056.91</td>
  </tr>
  <tr>
    <td>Original pairing (llama-turn-based + center-out Ascending)</td>
    <td>2060</td>
    <td>**</td>
    <td>17.43</td>
    <td>1031.42</td>
  </tr>
  <tr>
    <td>Original pairing (qwen-turn-based + ascending)</td>
    <td>2148</td>
    <td>**</td>
    <td>12.09</td>
    <td>256.13</td>
  </tr>
  <tr>
    <td>Original pairing (qwen-turn-based + descending)</td>
    <td>2181</td>
    <td>**</td>
    <td>12.09</td>
    <td>266.45</td>
  </tr>
  <tr>
    <td>Original pairing (qwen-turn-based + center-out Ascending)</td>
    <td>2155</td>
    <td>**</td>
    <td>12.09</td>
    <td>262.82</td>
  </tr>
  <tr>
    <td>Negative pairings (K=1)</td>
    <td>500</td>
    <td>13.61</td>
    <td>10.29</td>
    <td>281.76</td>
  </tr>
  <tr>
    <td>Opposite pairings (K=1)</td>
    <td>500</td>
    <td>13.24</td>
    <td>10.04</td>
    <td>286.09</td>
  </tr>
  <tr>
    <td>Positive pairings (K=1)</td>
    <td>500</td>
    <td>12.87</td>
    <td>9.72</td>
    <td>284.83</td>
  </tr>
  <tr>
    <td>Negative pairings (K=2)</td>
    <td>992</td>
    <td>29.34</td>
    <td>11.67</td>
    <td>324.25</td>
  </tr>
  <tr>
    <td>Mixed pairings (K=2)</td>
    <td>994</td>
    <td>27.44</td>
    <td>11.89</td>
    <td>333.66</td>
  </tr>
  <tr>
    <td>Positive pairings (K=2)</td>
    <td>987</td>
    <td>27.29</td>
    <td>11.75</td>
    <td>334.45</td>
  </tr>
  <tr>
    <td>Negative pairings (K=5)</td>
    <td>2898</td>
    <td>70.26</td>
    <td>14.53</td>
    <td>390.15</td>
  </tr>
  <tr>
    <td>Mixed pairings (K=5)</td>
    <td>2879</td>
    <td>69.34</td>
    <td>15.36</td>
    <td>417.49</td>
  </tr>
  <tr>
    <td>Positive pairings (K=5)</td>
    <td>2864</td>
    <td>67.29</td>
    <td>14.86</td>
    <td>423.67</td>
  </tr>
  <tr>
    <td>Negative pairings (K=10)</td>
    <td>913</td>
    <td>151.94</td>
    <td>16.68</td>
    <td>387.12</td>
  </tr>
  <tr>
    <td>Mixed pairings (K=10)</td>
    <td>905</td>
    <td>150.24</td>
    <td>18.02</td>
    <td>574.97</td>
  </tr>
  <tr>
    <td>Positive pairings (K=10)</td>
    <td>890</td>
    <td>146.09</td>
    <td>16.56</td>
    <td>475.81</td>
  </tr>
  <tr>
    <td>Positive Ratio 1%</td>
    <td>125</td>
    <td>**</td>
    <td>**</td>
    <td>**</td>
  </tr>
  <tr>
    <td>Positive Ratio 20%</td>
    <td>479</td>
    <td>**</td>
    <td>**</td>
    <td>**</td>
  </tr>
  <tr>
    <td>Positive Ratio 40%</td>
    <td>885</td>
    <td>**</td>
    <td>**</td>
    <td>**</td>
  </tr>
  <tr>
    <td>Positive Ratio 60%</td>
    <td>869</td>
    <td>**</td>
    <td>**</td>
    <td>**</td>
  </tr>
  <tr>
    <td>Positive Ratio 80%</td>
    <td>415</td>
    <td>**</td>
    <td>**</td>
    <td>**</td>
  </tr>
  <tr>
    <td>Positive Ratio 100%</td>
    <td>106</td>
    <td>**</td>
    <td>**</td>
    <td>**</td>
  </tr>
  <tr>
    <td>Negative Pairings (0.5B)</td>
    <td>974</td>
    <td>70.19</td>
    <td>10.33</td>
    <td>231.17</td>
  </tr>
  <tr>
    <td>Mixed pairings (0.5B)</td>
    <td>970</td>
    <td>69.32</td>
    <td>11.55</td>
    <td>246.13</td>
  </tr>
  <tr>
    <td>Positive pairings (0.5B)</td>
    <td>960</td>
    <td>68.24</td>
    <td>14.25</td>
    <td>300.61</td>
  </tr>
  <tr>
    <td>Negative pairings (3B)</td>
    <td>982</td>
    <td>**</td>
    <td>10.81</td>
    <td>334.58</td>
  </tr>
  <tr>
    <td>Mixed pairings (3B)</td>
    <td>984</td>
    <td>**</td>
    <td>11.24</td>
    <td>328.64</td>
  </tr>
  <tr>
    <td>Positive pairings (3B)</td>
    <td>985</td>
    <td>**</td>
    <td>11.54</td>
    <td>327.88</td>
  </tr>
  <tr>
    <td>Negative pairings (7B)</td>
    <td>2987</td>
    <td>**</td>
    <td>10.05</td>
    <td>393.20</td>
  </tr>
  <tr>
    <td>Mixed pairings (7B)</td>
    <td>2916</td>
    <td>**</td>
    <td>10.43</td>
    <td>390.12</td>
  </tr>
  <tr>
    <td>Positive pairings (7B)</td>
    <td>2851</td>
    <td>**</td>
    <td>10.46</td>
    <td>374.32</td>
  </tr>
  <tr>
    <td>Negative pairings (14B)</td>
    <td>995</td>
    <td>**</td>
    <td>12.51</td>
    <td>333.07</td>
  </tr>
  <tr>
    <td>Mixed pairings (14B)</td>
    <td>987</td>
    <td>**</td>
    <td>12.68</td>
    <td>335.39</td>
  </tr>
  <tr>
    <td>Positive pairings (14B)</td>
    <td>987</td>
    <td>**</td>
    <td>12.21</td>
    <td>317.31</td>
  </tr>
  <tr>
    <td>Negative pairings (32B)</td>
    <td>999</td>
    <td>**</td>
    <td>10.83</td>
    <td>339.31</td>
  </tr>
  <tr>
    <td>Mixed pairings (32B)</td>
    <td>980</td>
    <td>**</td>
    <td>10.64</td>
    <td>328.76</td>
  </tr>
  <tr>
    <td>Positive pairings (32B)</td>
    <td>959</td>
    <td>**</td>
    <td>10.15</td>
    <td>310.81</td>
  </tr>
</table>

Table 7: Personas and Dialogues Statistics table combining ConvAI2, synthesized user profiles, and RQ1–RQ4 data.

Reviewer paper summaries: - The paper examines how the sentiment of assigned user personas affects the coherence and consistency of dialogues generated by LLMs.
- Experiments show that negatively polarized personas often overemphasize persona traits, leading to contradictions and reduced coherence. In contrast, positively polarized personas integrate traits more selectively, resulting in smoother, more coherent dialogues.
- The study also reveals that personas with weak or neutral sentiment generally produce lower-quality dialogues
- Motivated by these findings, the paper proposes a dialogue generation approach that explicitly considers persona polarity by combining a turn-based generation strategy with a profile ordering mechanism
- The proposed approach aims to improve consistency and coherence by positioning neutral or weakly sentimental personas earlier in the dialogue and positive personas later.
Summary of strengths: - The paper is well-organized and well-written. 
- It is interesting that the paper focuses on the underexplored influence of persona sentiment on dialogue quality in LLMs.
- The authors conduct a comprehensive study using a range of polarized user profiles and several backbone LLMs, providing robust evidence for their findings.
- The findings are well-supported by empirical evidence:
  - The paper clearly identifies how different persona polarities impact dialogue consistency and coherence, revealing nuances in how LLMs handle positive and negative personas.
  - The paper examines how different user pairings and polarity levels affect dialogue quality, offering insights into key influencing factors.
  - It also explores how LLM size impacts polarity-based trends, revealing notable variations and questioning the assumption that bigger models always perform better.
Summary of weaknesses: - The authors evaluated the dialogue model using various automatic metrics, but human evaluation is necessary for a more accurate assessment. Currently, there is no human evaluation provided.
- Although two methods were proposed to make LLMs more robust to polarity, it is unclear how novel or effective these methods actually are:
  - While the turn-based approach showed some improvement over the joint approach, this setup is not particularly novel, especially considering the widespread use of agent-based systems and agent simulations today. As such, the turn-based method used here seems like a fairly typical baseline, and it remains uncertain whether it genuinely contributes to improving robustness to polarity in LLMs.
  - The use of user profiling and sorting had some effect, but the impact was relatively small. It would be more compelling if the paper proposed a more effective method for enhancing LLM robustness to polarity.
- The implications derived from the findings could be more clearly articulated. For instance, the findings, such as “a strongly polarized persona enhances dialogue quality” and “positively polarized personas result in better dialogue quality than negatively polarized ones”, could be further connected to real-world applications. How can these insights help in building better dialogue agents in practice? Simply forcing a persona to be strongly polarized may feel unnatural or may fail to accurately reflect the user’s true characteristics. It would strengthen the paper if the authors could discuss how their main findings could be applied to the development of more effective and realistic dialogue models.
Comments, suggestions and typos: [Question]
- Regarding the dataset filtering process (from 60k to 58k), who performed the filtering based on quality? Was it done by a human annotator or by a model?

[Suggestion]
- Incorporate human evaluation to supplement automated metrics and to gain deeper qualitative insights into how different persona polarities and the proposed methods affect dialogue quality.
- In Figure 7, for the turn-based approach, there does not appear to be a specific prompting technique used to better integrate persona sentiment into the dialogue generation process. The authors could consider employing more advanced techniques, such as sentiment-aware prompting, to more effectively incorporate persona sentiment into responses.
Overall assessment scores: 3.5