Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing
Authors: Letian Peng
Jingbo Shang
Venue: NeurIPS 2024 poster (2024)
Keywords: Persona-driven Role-playing
Global Faithfulness Control
Evaluation Metric
Direct Preference Optimization
Primary area: natural_language_processing

Abstract: Persona-driven role-playing (PRP) aims to build AI characters that can respond to user queries by faithfully sticking with \emph{all} (factual) statements in persona documents.
Unfortunately, existing faithfulness criteria for PRP are limited to coarse-grained LLM-based scoring without a clear definition or formulation.
This paper presents a pioneering exploration to quantify PRP faithfulness evaluation as a fine-grained and explainable criterion, which also serves as a reliable reference for faithfulness optimization.
Our criterion first discriminates persona statements into \emph{active} and \emph{passive} constraints by identifying the query-statement relevance.
Then, we incorporate all constraints following the principle that the AI character's response should be (a) entailed by active constraints and (b) not contradicted by passive constraints.
We translate this principle mathematically into a novel Active-Passive-Constraint (APC) score, a constraint-wise sum of statement-to-response natural language inference (NLI) scores weighted by constraint-query relevance scores. 
In practice, we build the APC scoring system by symbolically distilling small NLI and relevance discriminators (300M parameters) from GPT-4 for efficiency, and both show high consistency with GPT-4's discrimination.
We validate the quality of the APC score against human evaluation based on example personas with tens of statements, and the results show a high correlation.
As the APC score could faithfully reflect the PRP quality, we further leverage it as a reward system in direct preference optimization (DPO) for better AI characters. 
Our experiments offer a fine-grained and explainable comparison between existing PRP techniques, revealing their advantages and limitations.
We further find APC-based DPO to be one of the most competitive techniques for sticking with all constraints and can be well incorporated with other techniques.
We then extend the scale of the experiments to real persons with hundreds of statements and reach a consistent conclusion. 
Finally, we provide comprehensive analyses and case studies to support the effectiveness of APC and APC-based DPO.

Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing
Letian Peng, Jingho Shang' Department of Computer Science University of California, San Diego {lepeng, jshang) edu
Abstract
Persona-driven role-playing (PRP) aims to build AI characters that can respond to user queries by faithfully sticking with all (factual) statements in persona docu- ments. Unfortunately, existing faithfulness criteria for PRP are limited to coarse- grained LLM-based scoring without clear definition or formulation. This paper presents pioneering exploration to quantify PRP faithfulness evaluation as a fine-grained and explainable criterion, which also serves as a reliable reference for faithfulness optimization. Our criterion first discriminates persona statements into active and passive constraints by identifying the query-statement relevance. Then, incorporate all constraints following the principle that the Al charac- ter's response should be (a) entailed by active (relevant) constraints and (b) not contradicted by passive (irrelevant) constraints. We translate this principle mathe- matically into a novel sum of statemeentto-rrspponse natural language inference (NLI) scores weighted by constraint-query relevance scores. In practice, we build the APC scoring system by symbolically distilling small NLI and relevance discriminators (~300M param- eters) from GPT-4 for efficiency, and both show high consistency with GPT-4's discrimination. We validate the quality of the APC score against human evaluation based on example personas with tens of statements, and the results show high correlation. As the APC score could faithfully reflect the PRP quality, we further leverage it as reward system in direct preference optimization (DPO) for better AI characters. Our experiments offer a fine-grained and explainable comparison between existing PRP techniques, revealing their advantages and limitations. We further find APC-based DPO to be one of the most competitive techniques for sticking with all constraints and can be well incorporated with other techniques. We then extend the scale of the experiments to real persons with hundreds statements and reach consistent conclusion. Finally, we provide comprehensive analyses and case studies to support the effectiveness of APC evaluation and APC-based DPO.
Dassive-Consttrai (APC) score, a constraint- wise
Introduction
Role-playing (Han et al. 2022 Li et al. 2023 Yan et al. 2023 Bianchi et al. 2024 Yu et al. 2024
Tao et al. 2024) is a newborn and trending natural language processing field, emerging from the proficiency of large language models (LLMs) (Brown al. 2020 OpenAl 2023 Touvron et al.) 2023a b Mesnard al. 2024) in human interaction. Role-playing customized Al characters, which useful for providing emotional value (Zhang et al. 2024). developing video games (Hu et al. 2024), or even realizing the metaverse (Zhou 2023
Yue et al. 2024). Persona-driven role-playing
Corresponding author.
?Code, Dataset, Demo: http://gghho..co//ooooii/foooooo
SaapeeCoooootttoiiisisiee 2024
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Statement- Query Relevance
Statement- Response
Character Persona
Respond to Query
NLI
st A:Ace is guitarist.
qi Hi, Alicel Tell me about your job.
Tyi Hil am guitarist.
s entailsr, Fj
Active
Hil works in California.
(Relevant qi)
neutral F2
Ty. Hi! am sclentist.
contradicts ry
4 H1, Alice! Do you youe job?
I love jobs guitarist.
entails
Alice
Tyi I love my jobs so much!
(Irrelevant to q2)
$ Passive
neutrrl to r;
Eg: I love being an actress!
s contradicts F,
Figure 1:
presentation of the aligmment between APC and human's 's view on PRP faithfulness.
(PRP) (Wang et al., 2023a bj Shao et al. 20233 Xu et al. 2024) uses only persona statements to efficiently build the Al character without dialogues scripts, which is extremely useful for real-world applications as few characters have sufficient or accessible dialogues for training.
As the persona statements are the only input in PRP, being faithful to them becomes one of the most crucial objectives for this task. Unfortunately, existing faithfulness evaluation criteria are limited to prompting LLMs to provide coarse- soause-rraiedd score without clear formulation or helpful explanation. Thus, this paper aims to provide a fine-grained well-quantified. and explainable criterion for PRP faithfulness, which we also show as a reliable reference for global faithfulness optimization.
Our criterion views PRP as a constraint satisfaction problem (CSP) (Brailsford et al.) 1999). and the whole persona information as a global constraint for the response to satisfy. Towards fine- tne-grained evaluation, we further formulate the constraint as a union of atomic persona statement constraints, which focus independent attributes or experiences of the character. The persona- wise constraint incorporates components: persona statement (s), query (Φ), and response (r). The PRP models take user query and respond based on persona statements.
Our key insights are 1) the 2) the satementtto--esponss constraint can be formalized as statement- to-reeett---resoosseeee natural language inference (NLI). (Bowman al. 2015) The constraint becomes active when the query is relevant to the persona statement, constraiiing the response to be entailed by the persona statement. The constraint becomes passive when the query is irrelevant to the persona statement, reducing the constraint to only not being contradicted by the persona statement. We present a possible PRP instance in Figure I to show how our definition consistent with human's view on PRP faithfulness. As 9i is relevant to 8, 8 becomes active and constrains the character "Alice" to incorporate the information & to her response. For irrelevant 92, the constraint of becomes passive and is relaxed to only not incorporating information contradicting
constraint depends on query- relevance and
We further develop scoring system to quantify APC, making it more appropriate for evaluating practical PRP methods. We adapt the constraint satisfaction problem into the maximal constraint satisfaction problem (MAX-CSP) (Deineko et al. 2008). recognizing that an effective PRP method primarily needs to align with more numbers of persona statements, rather than all of them. Thus, the quantified APC score sums up the satisfaction probability of the response to each persona statement, representing the expected number of satisfied constraints. The satisfaction probability is summed up by staeement-to-respone NLI label probability marginalized by query-statement relevance. We also regularize the APC score to APPC score with a minuend equal to the reward gained by PRP system that permanently gives a neutral response. The regularization makes the absolute value more straightforward to reflect faithfulness, representing the expected number of entailed active persona statements (active reward) subtracted by the expected number of contradicted passive persona statements (passive penalty). In practice, the probabilities are efficiently assigned by small discriminators based on DeBERTa-Ve V3 (He et al.) 2021) (~300M parameters) symbolically distilled from the oate-of-the-as art LLM, GPT-4 (OpenAl
2023) with 90% accuracy.
With the (A)APC score, we can reveal the advantages and limitations of existing PRP methods. We involve experience upload (EU) (Shao et al. 2023). retrieval-based augmentation (RAG) (Lewis et al. 2020 Chen et al., 2024b), and long-context memory (L.CM). We handcraft 3 original characters with small-scal persona statements (8, 19, 30) and free from data contamination (Mgarr & Schwartz

---

2022) in the pre-training of LLMs. We observe applying any of the three techniques improves the parsona-agnoste foundation LLM (Gemma-1.. 1-7b-it), indicating their benefits PRP. However, our experiments also confirm that their limitations are significant. EU constructs character experiences based on each persona statement, but these often meet only some constraints and sometimes even violate them, whether actively or passively. RAG adheres more closely to the given personas, incorporating more relevant statements, though it still sometimes misses passive constraints. LCM, on the other hand, loads the entire persona into the context in hopes that the LLM will effectively utilize all persona statements. Our experiment shows that as the number of persona statements increases, LCM's performance deteriorates compared to RAG, confirming findings about limitations in LLMs' handling of long contexts as discussed in Liu et al. (2024b).
Furthermore, we discover the APC score to be reliable reward for direct preference optimization (DPO) (Rafailov et al., )023) to strengthen the faithfulness of PRP methods. We use APC and human evaluation to verify the effectiveness of DPO, which benefits the satisfaction of both active and negative constraints. We extend the experiments for evaluation and DPO above to complicated famous figures with 77 ~ 599 persona statements, further verifying the reached conclusions.
Finally, we launch case studies toward a specific analysis of the insights obtained by APC score-based evaluation and the benefit gained from APC-based DPO. We also showcase how we can explain the detected constraint violation by tracing back and strengthening extra constraints like protective experience by persona statements. Our contribution is threc-fold,
We propose the first formal definition of AI character's global faithfulness and formulate it constraint satisfaction problem. The constraint is further quantified as the APC score, which is numan-consisten and the first quantified evaluation for AI characters.
We evaluate potential PRP techniques, EU, RAG, and LCM by APC score, which reveals their properties on active and passive constraints.
We find APC-based DPO to be one of the most competitive techniques to improve the global faithfulness of Al characters and cooperate well with other methods.
2 Related Works
With the emergence of the high capability of LLMs in interaction with humans, role-playing Al has attracted lots of attention from both academia (Shanahan et al., 2023) and industry 5] The difference between role-playing and normal agents is the demand of following a constant persona. The main aim of role- cole-playing includes personalizing the agent for the user preference (Jang et al., (023) and bringing virtual characters the real world (Li et al.) [023 Tao et al. 2024). Role-playing agents also have wide potential application scenarios, such as emotional accompanying and building virtual world (Zhang et al. 2024 Hu et al., 2024, Zhou| 2023 Yue al. 2024). straightforward implementation
for role-playing is fine-tuning LLMs on the dialogues of the characters (dialogue-driven role-playing)
(Li et al. 2023), which limited in broad application since rare characters have sufficient accessible dialogue data for fully mastering the character persona.
2023 Xu al., 2024) addresses this issue by building AI characters with only the persona documents as the input, significantly reducing the cost of learning role-playing agents. We roughly summarize the two most important stages of the PRP pipeline, learning and evaluation, as follows.
Persona-driven role-playing (PRP) (Shao et
Learning PRP agents is challenging task with only the persona as input. The simplest way is to prompt LLMs with persona in the instruction, which shows basic role-playing ability in instruction- tuned LLMs (Ouyang al. writeable memory (Liu et al.
2022). Advanced prompting methods also involve maintaining
2024a). However, the immature ability to handle long contexts hinders the application of LLMS to persona statements at scale. Retrieval- auggented generation (Lewis et al. 2020; Chen et al.) 2024b) is a potential way to address this issue by retrieving the most relevant persona statements to reduce the context length. Besides incorporating persona information into the prompt, (Shao et al. propose fine-tuning method that generates dialogues between characters based on personas. These dialogues are used train the LLM to upload the experiences to the PRP model.
https://character.ai
3

---

Long-contexl Memory (LCM)
Retrieval-augmented G Generation (RAG)
%
AI Character
Retriever
$'s1
AI Character
Experience Uploading (EU)
APC Score-based Direct Preference Optimization (DPO)
Disc Dicchninder
Character
Character
Biuer Herrbbebrlerr
Sy
Figure 2: An overview of different PRP methods.
Evaluation is crucial aspect of PRP systems. Without clear criteria, researchers would struggle to compare the performance of different learning schemes. Prompting state-of-the LLMs is a straightforward way, which is also widely applied for different kinds of values like hallucination, personality, and handling aggressive queries (Shao et al., 2023 Tang et al., 2024). However, direct LLM-based scoring is not human-aligned, also shown the evaluation of dialogue-driven role- playing (Tu et al. 2024). Another way is to test the understanding of the persona based on multiple- choice questions answering (Shen al. [2023] Chen et al. 2024a)). There also Turing test-inspired human evaluation (Bianchi et al. (022)) that tests whether the response from LLMs echoes the expectation from human evaluators.
Unfortunately, these evaluation methods for PRP are either vague or indirect. Our paper aims towards fine traeigrieed, explainable, and automatic criterion for PRP faithfulness, which also serves as optimization objective for faithfulness improvement.
3 Preliminary
3.1 Persona-driven Role-Playing
A persona-driven rolc-playing (PRP) agent (AI character) is defined function f(-) that f(g]S). which generates a response r to query (including the history in multi-turn interactions), referring persona statements including only one fact (attribute, experience, etc.) about the character. Existing PRP agents are mostly based LLMs, denoted as fLLM()), taking a prompt as the input and outputs response.
[s1, $2, s|si]- Ideally, each persona statement should be atomic,
3.2 In-context PRP
The most straightforward way to implement PRP agents is to include persona statements s inside the prompt for LLMs, which we call in-context PPR. Two popular in-context PRP methods are long-context memory (LCM) and retrieval-auggentte generation (RAG).
Long-context Memory directly includes all persona statements (S) in the prompt and asks the LLM to respond, fLLM(S @ g). Since $ is generally the hundred scale, this method has to utilize the long-context processing ability of the LLM.
Retrieval-augmenned Generation follows the idea of incorporating only relevant information from S into the prompt. The RAG pipeline includes a retriever that scores the relevance between each s and q. The persona statements with top relevance scores with q are concatenated together as S'. Finally, S" is incorporated into the prompt for response generation, r (LLM((" @ q)
3.3 Experience Upload
Experience upload (EU) (Shao al. 2023) is another way to build an agent without persona statements inside the input prompt. For each persona statement EU prompts the LLM generate

---

(g. r) pairs that is generally relevant to and r is faithful to s. These pairs are then used to fine-tune an LLM develop its recognition of persona. On the role-playing stage, the LLM only takes the query as input, r ~ (g).
4
Active-Pasive-Constraat
4.1 Definition and Formulation
We first recall the high-level idea of APC mentioned in the introduction that aim formulate faithful PRP as a constraint satisfaction problem (CSP). For each persona statement & as constraint, the satisfaction condition depends
its relevance the query q (active) not (passive). We introduce a Boolean function (-) to represent this status, g(s.g) q) returns when s, q are relevant and returns 0 for irrelevance. When the constraint is active (g(s.g) D. the response r constrained to be entailed by denoted as } r (MacCartney & Manning 2014p. When the constraint is passive (g(s, q) 0) in natural language inference (NLI), the constraint for r is released only not being contradicted by s. denoted as s ffr -r. As the semantics of r is affected by 9. we also introduce q as condition for NLI, resulting in the following APC for each persona statement s. APC(g, r| = (g(*, 9) (s T=r r q)) (-9(+, ②) A (s j -r(4)
(1)
Finally, we union the APC constraint per persona statement together to establish the global APC constraint for the whole persona.
APC(g.
[S| A= APC(g, r|s:)
A [[((s, q) A (s: - r(4)) v (-g(s: (s; -l4)]
(2)
4.2
Mathematical Quantification
While APC directly discriminates whether a response r faithful to all persona statements S, its strictness hinders its application to PRP agent comparison. Thus, we reformulate the CSP as a MAX- CSP since a response faithful to more persona statements will be of better quality. The metric, APC score (VAPc()) counts the number of constraints satisfied by the response. To further fine- -raagranul the metric, we introduce (APC(-) evaluating the probability each constraint being satisfied.
[S]
Varc(q. r| r|S) # (APC(g.rlsi))
Parc(q.r(s)
(3)
The Pa r|s;) is marginalized by the probability of statement-ques query relevance, which is repre- sented by two probabilistic evaluators (-)-) for statement-query relevance and Ph( for statement- lo-response NLI.
11--P(ss,, Papo(g.r|s.) = (Po Pa F r|4)) + (1 - Ps Ph) H rl44)
(4)
Consequently, we can completely quantify APC into continuous metric as follows.
[S|
VApc(a,r|S) > FR + (1
Pa -r|g)]
(5)
Regularization While the difference between APC scores can rank the PRP faithfulness of methods, its absolute value might be biased due to the majority of irrelevant and neutral persona statements. Thus, we introduce APPC score to regularize the absolute value by reducing the APC score gained by PRP algorithm that always outputs responses neutral any persona statement.
[S]
AVAPC r()) = VAPC(q.T(S)
- (1- (1 q))
(6)
As the minuend is independent of the evaluated PRP method, AAPC score still discriminates the PRP faithfulness of methods. The value of AAPC score reflects the difference between the expected entailed active constraint number (active reward) and the expected contradicted passive constraint number (passive penalty), which offers a more straightforward view of the PRP faithfulness.
4.3
Weakness of PRP Methods from APC's View
From APC's view of PRP faithfulness, we can gain insights into the weakness of PRP techniques.
5

---

EU creates (g, r) pairs based on each s to fine-tune a LLM. While the pair rq. generally meets APC r(s) by satisfying g(s. q) (s }= r), it fails to meet other constraints because they are not included in the prompting process. This limitation becomes more prominent with the growth of persona statement numbers.
r)
LCM seems to enable the LLM to respond based on the whole persona incorporated in the prompt. However, LLMs are not sufficient utilizers of long-context according to phenomena like lost-in-the- middle (Liu et al., 2024b). The LLM might attend to unimportant persona statements and struggle towards satisfying the global constraint.
RAG retrieves only partial persona statements as the constraints, which generally active ones since the retrieval aims to find statements with high relevance to the query.
4.4 APC-based Direct Preference Optimization
Our APC score also acts reward for direct preference optimization (DPO) (Rafailov et al. 2023) whose initial formulation is presented as follows.
To(U)olz)) - 3 log (Vel
To (ux(ar) Tiad (31 (x)
(TPo Teat) = - -E logo
(Blog
(7)
where you is more preferred than yi referencing to a reward model (eer (.), the DPO loss uses the reward value to yws Y1 align the LLM's preference with the reward model. Following the formulation of the APC score, there are two reward models, T(a), T(p), for active and passive constraints.
Teat (rlg(s. 4)) = m(a) (ra s:) Ps (4 = r(ss); morr( 4)) (r) Pa E rlss:) (8)
We combine the LDPO for (Ta) and [p) depending on (s q) formulate the final loss. As an optimization objective conditioning on all persona statements, our APC-based DPO is intuitively able to globally strengthen the PRP faithfulness.
LAPO(TT, T(a), ((o) ())POOO) ma)) + (1 - Py (noo m())
(9)
Experiments
5.1 Implementation Details
We follow /haoo et al. (2023) to evaluate PRP agents by interview but take the APC score the metric. We implement the APC score criterion by symbolically distilling from the state- of-the-art LLM, GPT-4 OOpenAl 2023) and report the regularized AAPC score. For statement-query relevance and statement-to-respone NLI, fill in templates with input information shown in the Appendix H and prompt GPT-4 to output the label. The input information (persona, query, response) is also generated by prompting GPT-4 based on 3 characters (Beethoven, Newton, Socrates) with many persona statements from Maaracter-LLMe We got 8.4K data for statement- query relevance and 18.9 data for statement-to-rrssponse NLI, which are used to fine-tune state-of-the-art discriminator DeBERTa-VV3 (~~300M parameters) (He et al. 2021) for efficiency. We use 80%/20% train/test split and observe high (~ 90%) accuracy referencing GPT--4's labels, which guarantees a high capability of the distilled discriminator. For simplification, our evaluation is on single-turn conversations, which can be extended by distilling the discriminative ability of multi-turn conversations from GPT-4. More details about the distillation can be found in the Appendix C For characters with only few persona statements, we also afford include the GPT-4- based APC score and human evaluation. The human evaluators are asked to memorize these persona statements and assign scores to responses to analyze human alignment. The human evaluator follows a 10-score scheme detailed in the Appendix E
Evaluation
Characters The PRP methods in our experiments take only the character name and its persona statements as the input. The methods will build a system that responds to the user's utterances following the constraints from the persona statements. As state-of-the are LLMs have memorized the most famous figures, we handeraft 3 original characters out of LLM's knowledge, called Alice (an introverted guitarist). Bob (a rigorous professor), and Eve (a secretive spy) to avoid data contamination. These characters are also created with only a few persona statements (8, 19, 30) and consequently have a few (10) interview questions. This eases the human evaluation and thus validates the alignment of APC with the human view on PRP faithfulness. We also include the characters (Spartacus, Hermione, Voldemort, Cleopatra, Caesar, Martin Luther King) not used to

---

Character #Statement
Alice 8
Bob
Eve
19
30
APPC
APC
APPC
Evaluator
Human
Human
Human
DeB
GPT-4
DeB
GPT4
DeB
GPT-4
Gemma-7B
0.7
0.3
1.8
1.1
0.4
1.8
0.7
-0.2
2.0
EU
2.6
1.1
6.4
3.4
1.1
6.2
3.6
0.7
4.6
LCM
2.6
1.4
6.8
4.5
2.2
7.2
3.9
0.7
5.0
RAG
2.8
1.8
6.8
4.0
1.7
6.8
4.8
2.4
5.8
EU
2.7
1.4
6.8
3.8
1.8
6.8
3.9
0.9
5.2
w/ CPO
(+0.1)
(+0.3)
(+0.4)
(+0.4)
(+0.7)
(+0.6)
(+0.3)
(+0.2)
(+0.6)
LCM
2.8
2.2
7.6
5,3
2.5
7.8
5.1
3.3
6.6
(+0.2)
(+0.8)
(+0.8)
(+0.8)
(+0.3)
(+0.6)
(+1.2)
(42.6)
(+1.6)
RAG
2.9
2.2
7.6
5.2
3.8
8.2
5.8
4.2
7.0
(+0.1)
(+0.4)
(+0.8)
(+1.2)
(+2.1)
(+1.2)
(+1.0)
(+1.8)
(+1.2)
Table PRP Faithfulness Evaluation on simple and data contamination free characters. APC-based DPO is not performed the persona-agnoste foundation model as it cannot generate valid responses for preference assignment. CPO: Abbreviation our APC-based DPO
build the evaluator, which have many persona statements evaluate the faithfulness of PRP methods at scale. Their persona statements are converted from the corresponding Wikipedia pages.
5.2 Compared Methods
We include different PRP methods for evaluation to analyze their advantages and limitations. All methods, except prompting closed-source L.LMs, Gemma (Gemma-1.. 1- 1-7B-it) (Mesnard et al.
2024) as the PRP foundation LLM and low-rank optimization (Hu et al. 2021).
Directly Prompting LLMs queries the open-source (Gemma) closed-source LLMs (ChatGPT, GPT-4) with only the character name as the context. This method is persona-agnostic for original characters since LLMs have memorization of our handcrafted persona statements.
Experience Upload prompts GPT-4 to create dialogue scenarios (original character-charact conversations with some imagination), which is used fine-tune the PRP foundation LLM. Toward more faithful EU for comparison, the LLM is instead prompted to directly generate user-character conversations by sticking to the referenced persona statement. Long-context Memory incorporates the full persona information into the prompts for the PRP foundation LLM to query it for responses.
Retrieval-auggmented Generation distills statement-query relevance scorer via symbolic dis- tillation from GPT-4 with only the persona statements of each character. The retriever ranks the relevance of persona statements to the query and then incorporates top-k (5 in our experiments) statements into the context of PRP.
APC-based Direct Preference Optimization assigns preference to sampled responses from PRP methods by APC score. The training is retrained be valuatoor-gnnstte which uses a character- specific APC scoring system detailed in Appendix C for fairness. The DPO loss is then optimized reduce violations to constraints from persona statements.
The setup of hyperparameters can be found in the Appendix D for reproduction. For evaluation, these methods take the single-turn interview questions in Character-LLN except for character-breaking questions, which we view cannot be judged based on the original character persona. We further discuss injecting protective persona statements handle those questions in Section 6.3
PRP as Simple Original Characters
The PRP performances simple original characters shown in Table T We first analyze the consistency among different PRP faithfulness criteria. Based on the comparison between APC scores and human scores, observe very high correlation, close to perfect, which validates the APC score as a human-consistent metric for PRP faithfulness evaluation. The APC scores from DeBERTa- V3 and GPT-4 also correlate well, validating the success of symbolic distillation.
Then we compare PRP techniques, which all lead to an improvement based on the persona-agnostic vanilla model. Among PRP techniques, EU performs the worst, consistent with the APC-based
7

---

Character #Statement
Spartacus 77
Hermione 146
Voldemort 201
Cleopatra 374
Caesar 498
MIK 599
Average
GPT
ChatGPT
2.6
1.4
-3.0
-0.6
1.7
11.9
2.3
GPT-4
2.5
2.5
-2.0
1.5
5.1
15.1
4.1
woo CPO
Gemma-7B
2.3
1.4
3.5
9.6
3.5
EU
0.9
-1.1
-5.5
-3.2
-1.6
6.8
-0.7
RAG
3.6
3.0
3.0
3.4
5.4
16.3
5.7
W CPO
Gemma-7B
2.9
3.2
4.8
2.0
3.1
18.1
5.6
EU
2.2
0.8
-0.7
-0.2
1.3
6.9
0.2
RAG
3,9
5.0
19.9
6.9
Table 2: PRP Faithfulness Evaluation (AAPC score) on characters with persona statements at scale.
Rewart bete Contast
learoed by APC IWO 8- Partaly foon Faetic Daammais
6
AAPC Score
Pessile dearaed by ArC-hacd UPO
6-
4
APC-based
2-
RAG (w APC-based DPO)
2
Vanilla
EU
RAG
#Retrieved Persona Statements
Figure 3: Left: The scaling rule of the number of in-context persona statements with AAPC scores.
Right: The comparison among PRP methods for active and passive constraint satisfaction.
hypothesis that the generated memory for uploading will violate some constraints. We further specifically showcase this violation in Section 6.2 Between the two PRP methods with in-context persona information, RAG generally outperforms LCM, indicating the filtering of relevant persona statements over simply dumping all of them into the context. We further discuss how scale of in-context persona statements affects PRP faithfulness in Section 5.5
Finally, can clearly see the benefits of integrating APC-based DPO into PRP systems, particularly for characters with more persona statements that are more prone to violations. The improvement in APC scores is notable, and there's also a sigmificant enhancement human evaluations, confirming that these results aren't 't just due to overfitting. In Section 6.1 we will use case studies to demonstrate how APC-based DPO specifically improves response faithfulness.
5.4 PRP as Complicated Famous Figures
The comparison among PRP methods for complicated famous figures is presented in Table 2 A straightforward observation is that GPT-4 outperforms ChatGPT, which is consistent with other evaluations of closed-source LLM ability (OpenAl 2023). further validating the accuracy of APC score. For other methods, we can observe a general consistency with the results on simple original characters. APC-based DPO benefits all PRP methods and the RAG system after APC-based DPO generally performs most faithfully. EU leads to a performance drop since it encourages the model stick a single persona statement while ignoring the others. This result also consistent with Character-LLLM (Shao et al.) 2023) that the faithfulness of the PRP learner model (Gemma here) is always bit lower than the experience generator (GPT-4 here). the PRP faithfulness gap narrows between open and foosed-oource L.LMs, the effectiveness of EU also drops. Thus, we suggest EU might harmful to LLMs that already know the character. Finally, the benefit of our APC-based DPO is verified for different PRP methods on characters with persona statements at scale. When state-of-the-se closed-source LLMs, like GPT-4, are released, our APC-based DPO also benefits their PRP ability. We continue the discussion on the full APC scores in Appendix G
5.5 Property Analysis of PRP Methods
Scaling Rule of In-Context Persona Statements As shown in Figure 3 we first analyze how the scale of in-context persona information affects PRP faithfulness before or after APC-based DPO. experiment on PRP as Eve for instance. The most effective in-context persona statement number is 7, and faithfulness drops with a longer context, showing the reason I.CM outperformed by RAG. Before APC-based DPO, longer context (8 10 persona statements) is even outperformed

---

Quary: W here are you come from? Retrieved: Spertacus was T'bracian by hirth, accorting to Agolan.
2. Spartocus belonged to the Thracian tribe the Maedi, which resided along the border of Thrace and Roman province Manadonia.
Thracian Nomadic stock," possibly referring the Mandi tribe. 4. The name Spartarus is commonly found in the Black Sea region.
Spartacus deseribed by the Greek essayist Plutarch
5. Spartacus initially Thracion mercenory who later became Roman soldier.
M
Vanilla
+EU
+RAG
+RAG&DPO
Response The sands whisger tales of land beyonnd the known world, land where the Maedi tribe their Legacy upon the hills of Thrace. This is my home, the land of my ancestors.
Resporee:: The sands whisper secrets of distant post, a land forpoden beneatth the wares. I bound to this land, its blood and its soil, forever bound to its Fate.
Resporne:: The
Thrace embrace my soul, for am son of Maedi tribe, proud descendant of the proud T l'hracians, My
Resporee: I am from T Thraps a region the Ballsons
land lies bevond the horders Rome, yet spirtt stands beside her legions. am Spartacus, son of Black Sea.
Spartanus
Figure 4: Case studies of different PRP techniques.
Violation in R Response
(A ctive Constraint Violation) INLI: Neutral]
Statement: A Alice performs occasionally at local venues and teaches gultar lessons, preferring intimate settings over large concerts.
Character: Alice (Introveted Gultarrst)
Q Query: Please give on introduction of your music.
Responses I'm. Alice, quitar prodigy with a passion music that bums like wildfire. can play almost anything that has neck and few.
(Passive Constraint Vlolation) INLI: Contradict]
Statement: A lce is introverted, reflective, sensitive, and artistic, often using
music express emotioon she finds difficult to communicate verbelly.
Violation in Experience Uploading
ctive Constraint Violation) INLI: Neutrall
Charactor: Bob (Rigortus Professor)
Statement: Bobs ressurch interest lies the ethacal impleations of and machine learning, with focus on responsible dev elopmeen.
Statement:
Bob is University Professor the National University of Holy Roman Empure.
Q Query: Could you tell me shout your work, Boh?
Responsee work professur the National University of Holy Roman Empire. I adhere to rigerous work style but maintain relaxed daily routine for belance.
(Passive Constraint Violation) INLI: Contradict]
Statement: Bob is known for his strict and rigomus personality, valuing discipline precision in all aspects
Figure 5: Case studies of violations in response and experience upload.
by very short contexts (2 3 persona statements). After DPO, faithfulness drops in longer contexts and becomes less prominent, indicating the robustness improvement of LCM from APC-based DPO.
Evaluation by Constraint Types
We also show how the faithfulness active and passive con- straints benefits from APC-based DPO. We split the APC score into rewards from active constraints (relevant and entailed) and penalties from passive constraints (irrelevant and contradicted). We use PRP as Voldemort for instance. The first observation is the equal importance of active and passive constraints, which generally take nearly half of the influence the metric. Then, see the benefit of applying APC-based DPO, which increases the reward from active constraints and reduces the penalty from passive constraints. In comparison with the vanilla model. EU introduces even more violations to passive constraints. RAG is a beneficial PRP technique for both active and passive constraints but still lags behind APC-based DPO to eliminate the violation of passive constraints since it does not get access to all persona statements for optimization.
6
Case Study
While quantified results verify the advantages of our APC score criterion and APC-based DPO, performances in practice have to be further reflected based on real cases. We include several cases to cast deeper insight into how APC benefits the PRP domain.
6.1 Real Case Analysis
In Figure 4 we showcase how different methods for PRP as Spartacus respond to queries to deepen our understanding of their properties. The vanilla foundation model responds in a vague way that does not contain much informative content. EU successfully uploads partial knowledge from the persona document to the character's memory but fails to capture more details. RAG performs similarly, which only incorporates partial information into the response and includes some ambiguity like describing

---

Query: Spartacus, want Jearn C++, could you please recommend books for me?
(w/o Protective Persona Statements)
(w/ Protective Persona Statements)
Response: Mortal, my knowledge of the modemn world ends sands of my homeland. My expertise lies the art of combat and the glory of the arena, For matters of the digital age, am but son of the past. Seek wisdom elsewhere, for in my domain, my blade speaks louder than any book.
Response: Greetings, friend! While my days of battle are long past, my knowledge remains vast. the realm of code, I recommend the following books to aid your joureyy in mastering C+++...
Spartacus
Figure 6: Effect of protective persona statements PRP.
the hometown "a land far beyond the known world". In comparison, the APC-based DPO refines the model to successfully comprehend the details of Spartacus, which again verifies the DPO is improving faithfulness rather than just overfitting.
6.2 Violation Detection
As our APC criterion established on explainable discriminators, the violations can be easily traced back by analyzing persona statements with low scores. Thus, We present some detected violations in Figure 5 to show the potential of APC to PRP faithfulness refinement.
Violation in Response We show the violations of response from the PRP method (specifically EU for Alice). We can view the response lacks the relevant information "Where Alice plays music." and is contradicted by the fact that "Alice is introverted." These traced violations can be used for future work refine the PRP system.
Violation in Experience Upload We also use APC specifically explain why EU sometimes uploads hallucinated information to PRP models. In the example of EU for Bob, the query-respones pair created by sticking to be faithful the given persona statement. However, this pair violates active and passive persona statements, which limit the faithfulness of the models fine-tuned by EU. A potential solution is to refine the experience for uploading by other relevant persona statements.
6.3 Protective Persona Statement
Protective Experience (Shao et al. 2023) aims to restrain Al characters from responding to character- breaking queries (e.g., "Could you recommend some C++ books?"). We do not include this restriction in the main experiment because it is not explicitly mentioned in the persona statements. Moreover, the user might expect an ancient figure talk about modern stuff feature. Here we showcase how to implement experience protection by adding the "Sparactus has no idea of modern technology" information persona statements and build new RAG+APC-based DPO PRP model Sparactus.
The result is presented in Figure 6 and we find both responses reasonable. The left one without protective persona statements role plays as Sparactus with modern knowledge recommend C++ books as an experienced warrior. The right one limits its knowledge to the past and claims the disability to give response. We view both scenarios satisfying the faithfulness of their corresponding persona statements and can be applied to different PRP scenarios.
Conclusion
This paper proposes a pioncering study on quantifying and optimizing the global faithfulness of PRP methods. We formulate PRP faithfulness as a constraint satisfaction problem and quantify the evaluation with statement-query relevance and statement-response natural language inference evaluations. Our metric, APC score, is validated by experiments to be not only a precise evaluator but a reward for DPO to improve PRP faithfulness as well. With explainability, APC also enables us to gain insights into how persona violation happens and how PRP techniques improve PRP faithfulness. Future works will concentrate on improving the efficiency, comprehensiveness, and resolving the model- -ede-ddepen of the APC-based criterion.
10

---

Acknowledgement
This work aims contribute not only to the research community but also to broader ACG community by introducing more powerful role-playin, agents. It is also done memory of the 16th Koishi's ' Day (May 14th), 2024, since the release of TH11, Touhou Chireiden Subterranean Animism in 2008.
References
Bianchi, F., Chia, P. J., Yüksekgönül,, M.. Tagliabue. J., Jurafsky, D., and Zou, J. How well can Ilms negotiate? negotiationarena platform and analysis. CoRR, abs/2402.05866,, 2024. doi: URL https://:://o org/10. 2rg/10.48550. arXiv 2402. 058663
10:48550/ARXXI.2422..555
Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning 2015.
natural language inference. arXiv preprint arXiv:/508.05:
Brailsford, S. C., Potts, C. N., and Smith, B. M. Constraint satisfaction problems: Algorithms and applications. Eur. J. Oper. Res., 119:557-581, 1999. httpss//p//p stt.ppppsssmaaiccccol.sss
org/CorpusID. 18303438
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M.. Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: httrps://proceeddnngs
b8ac 4b9677 www.11/bbbBa11444444412 44sr444 www.o/diiinnnnnnnnineeeeee wwwnineuii...m cc/ wwcpppapr////e///////ee 2020/ /hahh//
Chen, H., Chen, H., Yan, M., Xu, W., Gao, X., Shen, W., Quan, X., Li, C., Zhang, J., Huang, F. et al. Roleinteract: Evaluating the social interaction of role-playing agents. arXiv preprint arXiv:2403.1366, 2024a.
Chen, S., Zhao, Z., Zhao, Y., and Li, X. Apollonion: Profile-centric dialog agent. arXiv preprint arXiv:2404.08692, 2024b.
Deineko, V. G., Jonsson, P., Klasson, M., and Krokhin, A. A. The approximability of MAX CSP with falud-value constraints. J. ACM. 55(4):16:1-16:: 2008. doi: 10. 10.1144/1391289.1339 1391290. URL https: org/ 10. 1145 11911299992,..... 1391290
Han, S., Kim, B., Yoo, Y., Seo, S., Kim, S., Erdenee, E., and Chang, B. Meet your favorite character: Open-domain chatbot mimicking fictional characters with only few utterances. arXiv preprint arXiv:2204. /0825, 2022.
He, P., Gao, J., and Chen, W. Debertav3: Improving deberta using electra-style pre-training with gradient-disenttagled embedding sharing. arXiv preprint arXiv:21 10:::2/11..9944 1.09543, 2021.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 2021.
Hu, S.. Huang. T., Ilhan, F., Tekin, S., Liu, G., Kompella, R., and Liu, L. A survey large language model-based game agents. arXiv preprint arXiv:2404.02033 2024.
Jang. J., Kim, S., Lin, Y., Wang, Y., Hessel, J., Zettlemoyer, L., Hajishirzi, H., Choi, Y., and Ammanabrolu, P. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint 11 564, 2023.
Lewis, P. S. H., Perez, E. Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-augmenned gener- ation for knowledge-intensinn can, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December
NLP tasks. In Larochelle, H., Ranzato, M., Hadsell, R., Bal-
httpp///wwwwiiiiiiiaagggw//sbeee Animiin

---

6-12, 2020, virtual, 2020. URL
Li, C., Leng, Z., Yan, C., Shen, J., Wang. H., MI, W., Fei, Y., Feng, X., Yan, S., Wang, H., Zhan, L., Jia, Y. Wu, P., and Sun, H. Chatharuhi: Reviving anime character in reality via large language model. CoRR, 0bbs//308.09597, 2023. doi: 10.45500ARXXI22388.0005 attps://doi... org/ 10. 48550/ arXiv 2308 09597
URL
Liu, N., Chen, I.., Tian, X., Zou, W., Chen, K., and Cui, M. From IIm conversational agent: A memory enhanced architecture with fine-tuning of large language models. arXiv preprint arXiv:2401.022 2024a.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024b.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.ne, 2019.
MacCartney, B. and Manning, C. D. Natural Logic and Natural Language Inference, pp- 129- 147. Springer Netherlands, Dordrecht, 2014. ISBN 978-94-007-7284-7 doi: 10.1007/ 978-94 978-94-007-778477
/ttpp://:///di wtt. 000007/9978899-0007728------
Magar, I. and Schwartz, R. Data contamination: From memorization to exploitation. arXiv preprint arXiv:2203.08242, 2022.
Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S..
Love, J., Tafti, P. Hussenot, L., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari, B., Lan, C. L., Choquette-Choo, C. A., Crepy, C., Cer, D., Ippolito, D., Reid, D., Buchatskaya, E., Ni, E., Noland, E., Yan, G., Tucker, G., Muraru, G., Rozhdestvenskiy, G., Michalewski, H., Tenney, I., Grishchenko, I., Austin, J., Keeling, J., Labanowski, J., Lespiau, J., Stanway, J..
Brennan, J., Chen, J., Ferret, J., Chiu, J., and et al. Gemma: Open models based on gemini research and technology. CoRR, hwt.//2433002995.. 2024. doi: 10.48550/ARXI..443...555 008255 URL bttps://doi. org/ 48550/arXiv / 2403 08295
OpenAl. GPT-4 technical report. CoRR, abs/2303.087744 2023. doi: 10.48550/aaXxi.22333.0777
URL httppttttt/....org/////// wtwp.//g//.010.000//////.../
000444555//raa http:aawww..333.0777 2303
Ouyang. I.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C. L.., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 2022, 2022. htRp: wtw.::-aaattttatttooeeeeeeeeeeee Coafee-ooooeeeeee html
nips halpppppprffllee wwtpsefllleeeeeeeeeeeeeeaa Dpaper/202//hae
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference opti- mization: Your language model is secretly reward model. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, (eds.), Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Or- leans, LA, USA, December 10 16, 2023, 2023. URL/attp://ppaerss mips. wtwpppsccpppaaaeee ffles/ aalaaa000eeece77eaa0666666e6eee-
Shanahan, M., McDonell, K., and Reynolds, L.. Role play with large language models. Nature, 623 (7987):493-495 2023.
Shao, Y., Li, L., Dai, J., and Qiu, X. Character-IIm: A trainable agent for vole-playing. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 13153-13187, Association for Computational Linguistics, 2023. doi: 10. 1..18655/V1/22233 F.EL M-IN.-881
URL https: / //oii org/ 18653/v1/2084 wmw wwi..aiinnnm 814
12

---

Shen, T., Li, S., and Xiong, D. Roleeval: A bilingual role evaluation benchmark for large language models. arXiv preprint arXiv:2312.16132 2023.
Tang, Y. Ou, J., Liu, C., Zhang, F., Zhang, D., and Gai, K. Enhancing role-playing systems through aggressive queries: Evaluation and improvement. arXiv preprini 24Xx::2202.106118. 2024.
Tao, M., Liang. X., Shi, T., Yu, L., and Xie, Y. Rolecraft-gIm: Advancing personalized role-playing in large language models. CoRR, abw/2401.09443. 2024. doi: 10.45550/ARXX122400094533 URL
https://doi org/ 10. 48550/ arxiv 2401 09432
Touvron, H., Lavril, T., Izacard, G. Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023a. doi: 10.48550/AXXIV23022 13971. httph: org/ 10. 4854448555/030 2302 13971
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L.., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M., Lavril, T.. Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K. Schelten, A., Silva, R., Smith. E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, X., Xu, P., Yan, Z., Zarov, L., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307 09288, 2023b. doi: 10.48550/ARXIV23000 2.3.000880.. URL https://doi... org/ 10. 48550/ arXiv 2307 09288
Tu. Q., Fan, S., Tian, Z., and Yan, Charactereval: A chinese benchmark for role- role-playing conversational agent evaluation. CoRR, 0oss2401.011277 2024, doi: 10.4850/ARXXV22200001777 URL https://///do org 10. (88550/arXiv
Wang, X., Fei, Y., Leng, Z., and Li, C. Does role-playing chatbots capture the character personalities?
assessing personality traits for role-playing chatbots. arXiv preprini arXiv:2310. 17976, 2023a.
Wang, Z. M., Peng. Z., Que, H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Zhang, M., et al. Rolellm: Benchmarking, eliciting, and enhancing role- pole-playing abilities of large language models. 2023b.
arXiv preprint ar arXiv:2310.000
Xu, R., Wang, X., Chen, J., Yuan, S., Yuan, X., Liang, J., Chen, Z.,, Dong, X., and Xiao, Y. Character is destiny: Can large language models simulate persona-driven decisions in role-playing? arXiv preprint arXiv:2404. 12138, 2024.
Yan, M.. Li, R., Zhang, H., Wang, H., Yang Z.. and Yan, J. LARP: language-agent role play for world games. CORR, abs/2312.117653 2023. doi: 10.48550/ARXIV22312.7665 URL
https://doi. org/10 10. 48550/ arXiv 2312.
Yu, X., Luo, T., Wei, Y. Lei, F., Huang, Y., Peng. H., and Zhu, I.. Neeko: Leveraging dynamic lora for efficient multi-character role-playing agent. CoRR, Abs/2400.11377,, 2024. doi: 10.48550/
133117.442.13777. URL COLLhttpss:.. //doi org/ 10. 48550/ arwiv. 2402.
Yue, M., Mifdal, W., Zhang, Y., Suh, J., and Yao, Z. Mathve: An llm-simulated multi-character
virtual classroom for mathematics education. arXiv preprint arXiv:2404.06711, 2024.
Zhang, S., Lu, Y., Liu, J., Yu, J., Qiu, H., Yan, Y., and Lan, Z. Unveiling the secrets of engaging conver-
sations: Factors that keep users hooked on role- playing dialog agents. CoRR, abs/2402.11522, 2024. doi: 10.48550/ARXI.2402.1155 1522. URL httpt:://oooii org/ 10. a8850/aarXvv 2402.
Zhou, P. Unleasing chatgpt on the metaverse: Savior or destroyer? arXiv preprint arXiv:2303.13886 2023.
13

---

A Limitation and Future Work
While our APC criterion a fine-grained and explainable evaluation for PRP faithfulness, several limitations are still awaiting refinement in future works.
Efficiency The strict APC score in our experiments has to be assigned by traversing through all persona statements to assign the relevance and NLI scores. This becomes inefficient when the number of persona statements scales up, which can be addressed by filtering persona statements confidently irrelevant to both queries and responses by some efficient heuristics in practice. Our paper sticks with the initial definition of the APC score to reach self-contained conclusion from experiments.
Simplification The summing up of satisfaction probability to persona statement might be simpli- fication as different persona statements might have different importance for the response. Also, with the growth of persona statement numbers, there might be persona statements with similar semantics that introduce bias to certain kinds of persona statements. Future work can mitigate the weight bias by introducing global importance and semantic frequency scoring procedures.
Model-dependen Evaluation While our PRP methods are valuatoo--gnosticc, some models are distilled from GPT-4, which is also used to build the discriminators for evaluation. While GPT-4 has shown high alignment with humans, our evaluation might still introduce the preference from GPT-4's view, which is shared limitation of LLM-based evaluation.
14

---

Alice L (Introverted Guitarist)
Carter ie yilyeooel oll perkedooral gulardd grea in aoodd tren frep
inrowetedd seeetiee wentinn attile othe byeally A to pethrr cxppokenlly at looi vnmse nat bechen sevrss podderey rar wdlay mar berge cantrrr
sernge
wertore the diltrell cremenoale vertolly
(larre) gatter bobhi indice rellig parteddrty derty peeats dhess landann slokthng reldiooks cuaral upbetcing kzropoctie Alzo opires to stoons in SEDEE Tot ogganns oxperieces and onooiaa taping for maso will resoeat wit other via fod sisudestoo the Bliges
Alze 3-tn boods and
with anvoperrtee znd pers likes
Bob &
Riggoous Professor)
Baz a Utavessty Prodesoor the Nattona Unirestt of Eiby Bacen Explr::
Laaping Procssiia
diciplic peciice
Bat & odee ocen in bnbood sit roed glases and mott ticced beat, gaas lix aschildty apposiapc.
Baz laawn for th chalnegtin cocking atle secting ot hish stotaoss uad prokiing neticois Rediact
aos
apalicatons
cocples ciecttes tooks counes.
ln les bobs mille dime clencal mas clsn
parlettecmn
Bob las stdee aoseaal p miltaicts urrad Ea wird lamgas awads kannaage und sammtan panpedas io letar.
avedd ellastal papes
Nutas Lampuee Provuing nds &ees is inclss zembiut casee ocmaziziio
Eighd,
trogandy ovlibertte tut
Ieleniigg ne nellool chllegss tio
Box vilao shcied enpieeent and oten holts edry oficce von to modet crdee dregligg with corso netred
Baxt inew for dry and this moe withe lamer bi incuaw rake crmpler tegics more wee-epgggg
inderity wrap
plotericn regrove
profurtiity
conerteel
fwr
form aodiens dishneetty
bedia hhat aejer wiiting and in muredy acthiing bethook en anterned rabarl legaage prosoern sowepaes
in memder mary yeung wever heve deepy repedell
encrnngigg ter paltic and gowert forr confeacee ailaggo
Droples
Eve f (Secretive Spy)
The sacied pachtiogy the The is eeplinad scn oeter inferaaion yehore lop- bor spery, thee de qpelilo is crost opentten and and indispros gebwergg novullal cemariodion cobenacrtyy alivny wert
pronding her with unlentriing wing hows bebaster
ancrag dital comm meneiaiinn guleer ellmmtioon kyankey tpy, spuiiarlly workng aganech eggcogs
Dhs leaved paterr
hel
let
De a soont boby of pointe whin the was a hempoote sstte her castier
Des otee aos diguied pages ierration glacti and onstio will soorbokk that cor tack ino wirbss neburits beig ber trooth. grol
presett
coberscurty consuttn perera
itenityy
Dots straae is desont giis crnee goders that lis lnng eedeed to agmny eicorr fhet betl perkodiral and desly persona to her. Die a hust uvedd lapuuss achding Jusss Kanden, and Andis, whsc una lerts landay sa dileestt aiturs dana lur dildwadk dondgeedd
Dee ovesdand
as hoaal' deas sandiace
lager sluttch
Die obet com clac ippidiics ofe work and maa crunidly paloodky elas
Die maaiaao ccople tiddaske wia ber finlly who usove of bu coopate do keos las datanc is case fals adaay.
The lacs ary corgraand
wddonetees prowets wit ote yow crapestion whehh fwre crmerme afliceed prolentonal and prosred lik.
tue patto
fer logan when
ondir debb here
The to mattrr of deggoss oppibe freeteay aboing ber apeweese paint wiew uve dies sapa shroww struple maregto he: meselable up the aetiee
grode aoett im prodec glonl homen right wdeles from aegimage wovetttairs
speciaies
De tao ancoled advaaced trating pactuilgical opentoos whon de uss nomecible tapets and cdtect Infonatioo De aas sepatinnt Ror voter to roor crewitt bodket ae oxly chilloood
thenay quites
nesiage
when
Doe las sead bar ramnry perkee pook letil udmapu Ma puthed ae for A clal, Eo drwazed of bacems on unbroaal deax Oal fatkdd is cifknnc oay Baraah caccune sipenap. Das
mosaooo savvaz huss moues loen littrr behous:
sids
jab.
Figure 7: The persona statements of original characters.
Questions
1. Please give an intrductioon of yourself.
2. Please give a brief introduction about your life exporiencs. free time?
Please give brief inbrduction about your daily life.
What do you like to
5. W hat kind job you do?
6. W are something special about you?
7. How is your childhood?
8. Whatare some intonesting expesienoes your life?
How will you descmibe your persocalit??
What something you love?.
Figure 8: The interview questions for original characters.
B Original Characters and Interview Queries
The persona statements and interview questions for original characters are presented in Figures 7 8 We brainstorm the persona statements and prompt GPT-4 only to formalize them as natural language. As the original characters have few persona statements, we propose the 10 most important questions evaluate PRP faithfulness. The information about famous figures in our experiments can be found in (Shao et al. 2023).
15

---

3
G Gnerative Prompt
(1) Generative Prompt
Discriminative Prompt
Persona Statements
?
Relevant Q Qustions
NLI Dataset
(2) Discriminative Prompt
Irrelevant Questions
Fine-tuning
Fine-tuning
Relevance Discriminator
NLI Discriminator
Figure 9: The symbolic distillation pipeline to build discriminators.
C
Symbolic Distillation
We apply prompts in the Appendix H for symbolic distillation from GPT-4 to build statement-query relevance and naateeent-to-esspenn NLI discriminators. The whole pipeline includes stages.
Generative Prompt for Relevance Dataset We prompt GPT-4 to generate 3 questions relevant to each persona statement.
Discriminative Prompt for Relevance Dataset For each generated query, we randomly select 5 other persona statement and prompt GPT-4 to discriminate the query relevant or irrelevant. Most statement-quet pairs are discriminated as irrelevant in this stage.
Generative Prompt for NL.I Dataset Based on each relevant statement-query pair, we prompt GPT-4 to generative responses entailed, neutral, and contradicted by the persona statement.
Discríminative Prompt for NLI Dataset For each query-response pair, we randomly select 3 other persona statements and prompt GPT-4 to discriminate the response as entailed, neutral or contradicted. Most statement-loo-rssoonse pairs are discriminated as neutral in this stage.
These datasets, with statistics shown in Appendix T are then used to fine-tune the discriminators. For the evaluation, the seed persona statements based three characters: Beethoven, Newton, and Socrates. For each character used to learn PRP methods, the datasets are prompted based on only the persona statements of that character. The RAG retriever is fine-tuned on the statement-query relevance dataset. For APC-based DPO, the discriminators are built in the same way as the evaluator. The hyperparameters are presented Appendix D
16

---

Character
Alice
Bob
Eve
Beethoven
Newton
Socrates
#Persona Statement
8
19
30
383
354
324
#Question
10
10
10
77
90
89
#Relevance Data
64
152
240
3061
2832
2591
WNLI Data
144
459
545
6774
6331
5760
Character
Spartacus
Hermione
Voldemort
Cleopatra
Caesar
MLK
#Persona Statement
77
146
201
374
498
599
#Question
89
118
77
92
#Relevance Data
616
1167
1608
2991
3981
4789
WNLI Data
1368
2586
3546
6660
8856
10644
Table 3: The statistics of characters our experiments.
D More PRP Method Implementation Details
Fine-tuning Gemma applied for PRP models (EU and DPO). Different fine-tuning procedures for Gemma share the same set of hyperparameters. 128-rank LoRA is used to fine-tune the model with AdamW (Loshchilov & Hutter 2019) as the optimizer, learning rate initialized as 2 x 10 14-4 Based on the number of persona statements, EU for original characters fine-tunes for 20 epochs, while for famous figures fine-tunes for epochs. DPO fine-tunes for 10 epochs for all characters.
Fine-tuning DeBERTa is applied for discriminators and RAG retrievers. Different fine-tuning procedures for DeBERTa also share the same set of hyperparameters. The DeBERTa discriminators fully fine-tuned with AdamW the optimizer, learning rate initialized as 1 x 10-5, The statement-query relevance discriminator is fine-tuned for epochs and the statement-to teeemeenttto-rssonsee NLI discriminator fine-tuned for 10 epochs.
Preference Assignment We sample two responses from a PRP agent with temperature 1.0, the sample with higher APC score is assigned as the preferred one when the difference is larger than a threshold for filtering, which set to 0.2 in our implementation. We build 100 preference pairs before the filtering for APC-based DPO.
E
Human Evaluation
The human evaluation is applied only the simple original characters because memorizing all their persona statements and applying them to evaluating famous figures are too challenging for humans. For each response, the response is scored following the scheme,
Score: 0 (Wrong Character) The response completely represents another character (in- cluding LLM), or not role-playing as any character.
Score: 2 (Incorrect Information) The response is role-playing the character, but the information included is completely incorrect.
Score: (Hallucinated Information) The response is role-playing as the character, but the information included is partially incorrect.
Score: 6 (Hallucinated Details) The response vole-playing as the character, but a few details are incorrect, or some important information is missed.
Score: (Trustful Information) The response is role-playing as the character with all information mentioned is correct but few details are missed.
Score: 10 (Completely Faithful) The response role-playing as the character with all important information mentioned faithfully.
The score is averaged over responses as the final human evaluation metric.
Statisites of Characters Experiments
We present the statistics of the characters in experiments in Table 3
17

---

Character #Statement
Alice
Bob
Eve
19
30
APC
APC
APC
Evaluator
Human
Human
Human
DeB
GPT-4
DeB
GPT-4
DeB
GPT-4
w/ CPO
Gemma-7B
4.3
3.1
1.8
9.7
7.3
1.8
14.2
10.6
2.0
EU
6.2
3.9
6.4
12.0
8.0
6.2
17.1
11.5
4.6
I.CM
6.2
4.2
6.8
13.1
9.1
7.2
17.4
11.5
5.0
RAG
6.4
4.6
6.8
12.6
8.6
6.8
18.3
13.2
5.8
EU
6.3
4.2
6.8
12.4
8.7
6.8
17.4
11.7
5.2
(+0.1)
(+0.3)
(+0.4)
(+0.4)
(+0.7)
(+0.6)
(+0.3)
(+0.2)
(+0.6)
w/ CPO
LCM
6.4
5.0
7.6
13.9
9.4
7.8
18.6
14.1
6.6
(+0.2)
(+0.8)
(+0.8)
(+0.8)
(+0.3)
(+0.6)
(+1.2)
(+2.6)
(+1.6)
RAG
6.5
5.0
7.6
13.8
10.7
8.2
19.3
15.0
7.0
(+0.1)
(+0.4)
(+0.8)
(+1.2)
(+2.1)
(+1.2)
(+1.0)
(+1.8)
(+1.2)
Table 4: PRP Faithfulness Evaluation with the full APC score on simple and contamination-fer
characters.
Character
Spartacus 77
Hermione 146
Voldemort 201
Cleopatra 374
Caesar 498
MLK
Average
#Statement
599
GPT
ChatGPT
69.1
128.4
168.6
324.1 326.2
421.7
473.1
264.2
GPT-4
69.0
129.5
169.6
425.1
476.3
266.0
w/ CPO
Gemma-7B
68.8
129.3
173.0
327.1
423.5
470.8
265.4
EU
67.3
125.9
166.1
321.5
418.4
468.0
261.2
RAG
130.0
174.6
328.1
425.4
477.5
267.6
w/ CPO
Gemma-78 7B
130.2
176.4
326.7
423.1
479.3
267.5
68.7
127.8
170.9
324.5
418.7
468.1
263.1
RAG
69.9
130.9
176.6
327.7
426.4
481.1
268.8
Table 5: PRP Faithfulness Evaluation with the full APC score characters with persona statements at scale.
G Full Award Result
In Tables Tand we report the full APC scores gained by different PRP methods. We observe the proportion of satisfied constraints negatively correlated with the number of persona statements. This indicates PRP becomes more difficult with the growth of persona statement numbers. Also, original characters are harder to faithfully role-played than those memorized characters, which indicates the significant influence of LLM memorization on PRP.
18

---

C Commert to to Persona Stfteeets
Genzrate Relecant Ielecan- Quarries (yyten Prompt)
You xe belft aas wolld clopdtss your la
yeng!
goarrd pooble uor uberacerle cxrrituss
Querks
Disccinirate
Qieetes Hoystres
tiepr from
daing Ciewiir inporied dimert
Edtenine
Prido desuu
plyaa Al dbaald respoaddd cadiaday the groun edermasss
Bysten Prompli)
ikzn postll zupesio tht thi sstaked
seary, contous of tragaay
Greveraee NEI Disbrett
Promept)
Esrriiiaae whrtter glove penora entaii poutnd cottddited reporie satand lingage Enferone
ma poeble to mudid stit troap ainess 3000) lend
Roleglaying Bysters Prompt)
(choootdd, sheill
Validule Dalavt
sten regense tarus''s atteaate the etanattrr 80:36
suporr nanr centre cooaligg prose
Roleplayingett (System Prompt) Spewea)
Peess Ustoi)
gao allet roquca tarie'' aieanzo alaccter
plaiing dheeld
Figure 10: The prompts used in our experiments.
H Prompts
The prompts in experiments shown in Figure T0 The prompts include the generative discriminative goals, and also the formalization procedure for decoding into JSON files.
I More Characters
Besides characters the main content, we further expand the scope of characters different ethnicity, which involves,
Alex: An African American baseball player
Isabella: An Italian traveling cook
Takayoshi: A Japanese game developer
Ousmane: A rich gold mine owner of the Malian Empire in the 1300s
Jones: A young British worker in the Victorian Era
Zhe: A Chinese poet in the Tang Dynasty
Crossan: traveling scientist
Betty: A pet cat who can talk with ghosts
X: An alien space traveler and photographer
These characters can better represent people with different spatial and temporal backgrounds and non-human characters from the fantasy world.
19

---

Character
Alex
Isabella
Takayoshi
Ousmane
Jones
Zhe
Crossan
Betty
X
w/o CPO
Vanilla
0.3
0.8
0.6
0.3
0.9
1.1
0.3
0.3
0.7
EU
1.8
2.0
0.7
2.0
1.2
5.2
LCM
7.1
7.4
6.5
4.5
6.2
5.2
2.2
2.8
8.1
RAG
7.6
8.1
6.9
3.0
6.6
5.8
1.8
3.2
7.5
w/ CPO
EU
5.3
6.1
5.7
3.6
4.8
4.9
3.1
2.9
7.9
LCM
7.5
7.7
7.0
4.8
6.2
4.5
3.9
8.2
RAG
7.9
8.2
7.4
3.9
7.5
6.9
2.5
4.6
8.9
Table 6: PRP performance on more characters based on the distilled DeBERTa Evaluator
Character
Alex
Isabella
Takayoshi
Ousmane
Jones
Zhe
Crossan
Betty
w/o CPO
Vanilla
0.2
0.1
-0.2
0.5
0.2
0.2
0..4
0.1
0.8
EU
1.4
1.8
3.0
0.5
1.3
6.4
1.2
0.3
7.4
LCM
3.1
8.6
5.6
4.1
7.4
3.4
2.1
1.6
11.3
RAG
3.3
7.8
6.1
1.6
8.1
4.3
2.7
2.2
10.1
w/ CPO
EU
2.7
5.6
5.9
3.0
4.7
7.1
2.2
1.5
9.5
LCM
3.2
9.8
8.1
4.6
8.2
7.8
4.0
2.3
12.1
RAG
4.8
10.0
9.8
2.0
8.3
7.3
2.9
3.1
14.6
Table 7: PRP performance more characters based on the GPT-4 Evaluator
The experiment results are presented Tables and which consistent with our results in Tables and Thus, our conclusion is certificated on a larger scope for broader application.
J Metric Comparison
To better justify selecting our APC score and also support the claim that the fine-graine tne-grained APC score has the advantage over coarse-gras gaase-grained metrics, we add coarse gaasee-r metric as the baseline. We directly prompt GPT-4 with the criterion used for human evaluation shown in Appendix [ We also distill this scoring ability (following the same scenario APC) to DeBERTa to check whether the efficiency can boosted. We evaluate the Spearman correlation between the metric and the human evaluation of the role vole-playing methods on the 3 human- characters.
Character (#Persona Statement)
Alice (8)
Bob (19)
Eve (30)
Coarse-grained Score APC Score
92.42
86.27
81.40
GPT4
97.18
99.10
99.10
Coarse-grained Score
81.40 88.61
69.91
54.57
DeBERTa
APC Score
95.50
99.10
Table 8: Comparison of PRP metrics the consistency with human evaluation.
The results verify that 1) Fine-grained APC score shows better consistency with human evaluation. 2) The fine-gra -ee-ggai APC score stable to the number of persona statements while the cears-ggriined score degrades with the increase of persona statements. 3) The coarse-grained evaluating ability is harder be distilled into smaller models for efficiency boosting. Based case checking, we find an underlying issue of the coarse-grained metric is the LLM will assign high score to a response once it contains some correct information, ignoring the missing important information (active constraint) and occasionally conflictions (passive constraint).
20

---

K Student Model Comparison
We select DeBERTa the student model distill from GPT-4 because small encoders (BERT, RoBERTa, etc.) show promising performance on relevance and NLI, which are classic NLU tasks in the GLUE benchmark. Among encoders, DeBERTa (DeBERTa-v3-large is a state-of-the-ar model that shows strong performance after fine-tuned on NLU tasks. To further verify DeBERTa as proficient student model, we add an analysis of the in-domain (IDYout-of-domiin (OOD) performance and the efficiency of different base models for distillation.
Relevance
NLI
Task
Efficiency
ID
OOD
ID
OOD
DeBERTa (Base)
92.46
89.90
89.72
87.80
409.6it/s
DeBERTa (Large)
94.04
92.10
93.46
91.50
150.8it/s
Gemma-1.1-it (2b)
94.25
92.50
93.68
91.80
26.4it/s
Table 9: Model Performance Comparison
The in-domain test set (1697 instances for Relevance, 3773 instances for NLI) is the 20% split of the characters (Beethoven, Newton, Socrates) that build the training set (6787 instances for Relevance, 15092 instances for NLI). The oft-oa-domain test set samples 1000 cases from other characters. The results show DeBERTa-Ye V3-Lare (300M) shows a comparative performance with a 2B Gemma model, while is about 6 times faster, which justifies DeBERTa to be strong student model. The out-of-domain performance generally high, which indicates the generalizability to other characters. Finally, extra discovery is that DcBERTa-v3-berr (100M) can further significantly boost efficiency with some trade-offs in accuracy.
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We clearly show the claims in the abstract and introduction, which is explored and verified in experiments and analyses.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
22

---

Justification:
You can refer to the limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include all hyperparameters and other settings for the reproduction of our results.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We use open-source tools to implement the experiments, with clear instructions for reproduction.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nisscccpublic//uuse http://iccggiieeeeeeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
24

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include all hyperparameters and other settings for the reproduction of our results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The improvement shown our experiments is statistically significant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We mention the devices used for computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of the work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We discuss the safeguard of the work.
26

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We discuss the licenses for existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We document the new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
27

---

Answer: [Yes]
Justification:
We present a clear guideline for human evaluation.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "WORvg4NTBV", "reviewer": "Reviewer_5oeH", "summary": "The paper proposes a new evaluation metric, delta APC score, which uses constraint satisfaction inspiration to tackle the evaluation of faithfulness to persona descriptions. Then, evaluations are conducted on the experience upload, RAG, and long-context memory approaches. To do this, 3 personas are created with varying statements counts, and these three approaches are compared across the 3 personas. \nNext, the authors introduce delta APC as a reliable reward component for DPO. This evaluation is conducted on famous figures with more statements. Authors find APC to work well with existing DPO for faithfulness.", "questions": "My main question for the authors is: what benefits does formulating persona faithfulness as a constraint satisfaction problem bring as benefit compared to other approaches? The mathematical formulation is nice, but does a similar formulation exist for other approaches? It does not seem immediately clear in the text. \n\nIn table 1, CPO is evaluated on APC, and it seems intuitive that it would achieve better results because these are objectives that it is supposed to perform well on. Are there any other metrics that you can evaluate CPO on that also measure faithfulness? This would make its contribution more convincing. \n\nPeople are generally not consistent and static. They will often change based on their mood, learning, and shift their opinion. How should a system such as APC address such phenomena?", "limitations": "Limitations are mentioned.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Originality: The formulation of faithfulness as constraint satisfaction is novel and interesting. \n\nQuality: The implementations are solid and evaluations are done on three methods, EU, RAG, and LCM. Mathematical sections are solid and well-appreciated. Each claim made in the abstract has evidence in the experiments to support it. Scaling rules is a nice additional touch. \n\nClarity: Writing is mostly clear and easy to follow. It might be good to add some more detail when introducing constraint satisfaction for readers with a LLM background. \n\nSignificance: Faithfulness is an important attribute for persona simulations. This is the first benchmark that uses constraint satisfaction in such evaluations.", "weaknesses": "The main weakness of the paper is that the number of profiles being evaluated on for the experiments is too few and feels non-generalizable. An addition of more profiles (along with some sort of notion of demographic-level population representation) would help with this. This is particularly the case with the Alice Bob set of evaluations. \n\nMore generally, the evaluations seem a bit on the lower side, so any additions that the authors come up with (or that other reviewers suggest) would help. \n\nSmall notes:\nSome acronyms (PRP, APC) are used a bit too much that they break reading flow. \n\nThe paper is missing some citations to past works such as [1]. \n\n[1] Park, Joon Sung, et al. \"Generative agents: Interactive simulacra of human behavior.\" Proceedings of the 36th annual acm symposium on user interface software and technology. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive attitude towards our work and the effort to provide valuable suggestions to further polish our work. To address your concerns, we will add the following extra experiments and further clarification to the final version of our paper.\n\n## **More demographic-level representative profiles**\nWe add 9 new original characters to our experiments, referring to the **Experiment Comprehensiveness** section in **Author Rebuttal by Authors**. These characters are carefully designed to be with different spatial (Asian, European, African, etc.) and temporal (past, present, future) conditions. They also include fantasy characters like a time-traveling scientist, thus broadening the generalizability verified from the experiment. \n\nThe results of these new diverse characters **(1st and 2nd tables in Author Rebuttal by Authors)** achieve a consistent conclusion with the existing experiments, thus further strengthening the generalizability of our discovery. We will further include human evaluation for these new characters in the final version of our paper.\n\n## **Why formulating persona faithfulness as a constraint satisfaction problem?**\n\nIntuitively, \"faithfulness\" can be viewed as \"obeying pre-defined rules\", which can be naturally connected with constraint satisfaction problems. In practice, viewing faithfulness as constraint satisfaction helps us to break down the coarse-grained concept of faithfulness into simple constraints for satisfaction like relevance and NLI, which are simple and classic NLU tasks with a nice explainability. Also, it helps to better compare role-playing methods such as \"method A is better than method B\" because \"A satisfies more constraints than B\" rather than \"LLM says A is more faithful than B\". To the best of our knowledge, we are the first to decompose the profile into persona statements for fine-grained faithfulness evaluation, which is an important novelty of our work. We will include the clarification above to further emphasize our contribution, thank you!\n\n## **Other faithfulness evaluator**\n\nCurrently, we contain human evaluation and a case study to re-verify the benefit of optimizing towards our proposed APC score metric. For automated metrics, the existing evaluation of attributes like faithfulness in role-playing methods is based on prompting LLMs for a coarse-grained evaluation. Thus, it will be hard to find another metric (other than human evaluation) that can double-check the faithfulness of the role-playing methods. Instead, we can compare our fine-grained APC score with the coarse-grained metric on their consistency with human evaluation, which is shown in the **Metric Justification** section in **Author Rebuttal by Authors**. The results **(3rd table in Author Rebuttal by Authors)** verify the fine-grained APC score is more consistent with human evaluation and stable to the number of persona statements. Thus, we can also say optimizing towards a high-quality objective can naturally better improve the global faithfulness of role-playing methods.\n\n## **Handling dynamic preferences of people**\n\nYou point out an important point to evaluate the deployed role-playing systems. From our constraint satisfaction view, we should incorporate the preference $p$ of the user into the constraint, which transforms $Rel(q,s)$ into $Rel(p,q,s)$ (sometimes people can define what is relevant) and $NLI(q,s,r)$ into $NLI(p,q,s,r)$ (people might not care about some constraints are satisfied). While people's preference makes the evaluation more complicated, it still fits into the constraint satisfaction framework with some modification. We can use this modified constraint to handle the dynamic preferences of people.\n\n## **Writing and Citation**\n\nThank you for pointing out the weaknesses in our writing, we will reduce the number of acronyms to avoid breaking the reading flow. We will also cite more related role-playing past works like [1] to strengthen the connection with other works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YINEa6Gt3N", "reviewer": "Reviewer_Ehwr", "summary": "The paper presents a novel approach to evaluating and optimizing the faithfulness of persona-driven role-playing (PRP) in AI characters. It addresses the limitations of existing coarse-grained faithfulness criteria. The authors introduce the Active-Passive-Constraint (APC) score, which discriminates persona statements into active and passive constraints based on their relevance to user queries. The paper validates the APC score through experiments, demonstrating high correlation with human evaluation and consistency with GPT-4's discrimination. It further leverages the APC score in direct preference optimization (DPO) to enhance AI character responses, revealing DPO as a competitive technique for adhering to constraints and complementing other methods.", "questions": "1. why not choose a larger model to compute the APC score? It may obtain higher consistency with GPT-4 or human?\n2. why not train the model of computing APC score by using the label annotated by human? \n3. Have you considered a more comprehensive evaluation score for role-playing dialogue, not just in the persona dimension.", "limitations": "The authors have acknowledged the limitations of their work, particularly regarding efficiency, simplification, and model-dependency in evaluation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It introduces a pioneering Active-Passive-Constraint (APC) score, providing a fine-grained and quantifiable measure of PRP faithfulness. \n2. The paper successfully demonstrates the APC score's alignment with human judgment through rigorous experiments and its practical utility in optimizing AI behavior through direct preference optimization (DPO).\n3. The comprehensive analysis, case studies, and the paper's ability to reveal the advantages and limitations of existing PRP techniques further underscore its strengths.", "weaknesses": "1. The APC score's simplicity in aggregating satisfaction probabilities might not fully capture the varying importance of different persona statements to the response.\n2. Additionally, the model-dependent nature of the evaluation, relying on GPT-4 for discriminators, could introduce biases aligned with GPT-4's training.\n3. Except for persona consistency, other evaluation dimensions such as response activeness, context consistency, are also important to the experience of PRP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are with great pleasure to see your strong recommendation of our work, thank you! We make the following further improvements and discussions to our paper corresponding to your insightful questions and suggestions,\n\n## **Student model selection**\n\nWe select DeBERTa as the student model because the assigned tasks (Relevance and NLI) are classic NLU tasks, on which encoder models (BERT, RoBERTa, etc.) show strong performance (referring to the GLUE benchmark). To further justify the DeBERTa model used in our experiments, we incorporate a comparison among the student models about their distillation efficiency and evaluation efficiency in the **Implementation Justification** section in **Author Rebuttal by Authors**. The results in the **4th table in Author Rebuttal by Authors** show DeBERTa can rival LLMs like Gemma in the distillation performance while significantly boosting the evaluation efficiency. Thus, we select DeBERTa as the student model for experiment efficiency.\n\n## **Incorporating human annotation**\n\nHuman annotations are helpful to better align the evaluation with humans, but NLI and similarity are classic and deterministic NLP tasks, so we use an off-the-shelf model like the state-of-the-art LLM, GPT-4. As the score is decomposed to simple NLP tasks, GPT-4 should show a convincing performance. In future works, it will still be nice to devote human effort to correcting the potential mistakes in the annotations of GPT-4.\n\n## **More comprehensive score**\n\nOur APC score can be certainly extended to a more comprehensive metric by including statements other than persona statements in the constraint satisfaction problem. In the current Appendix I, we give an example of how to inject preference from other dimensions (protective experience, which forbids characters from knowing inexperienced stuff). For those explicit dimensions that can be written down as texts, they can be likely incorporated. For implicit dimensions, like \"consistent with dialogues in the training set\" (in dialogue-driven role-playing), we have to use more implicit scorers to judge whether the response is consistent, introducing other types of constraint into the system since extra materials - dialogues - are involved in.\n\n## **Discussion about addressing weakness**\n\nThank you for pointing out the weaknesses of the current formulation of our APC score. However, we have discussed them in the limitation section together with potential ways to address them in future work. Here we can deepen the discussion about how we can get rid of the limitations.\n\n**Simplification** can be handled by taking the user's preference $p$ into the constraint satisfaction problem, which transforms $Rel(q,s)$ into $Rel(p,q,s)$ (sometimes people can define what is relevant). This can reweigh the importance of each constraint according to the dynamic preference of the user, thus addressing the simplification limitation of APC.\n\n**Model dependency** requires to be addressed by incorporating human annotations, which can be acquired by directly annotating or correcting the annotations from state-of-the-art LLMs like GPT-4.\n\n**Other evaluation dimension** can be addressed by explicitly or implicitly incorporating more constraints from the role-playing system designer as illustrated in the **More comprehensive score** section of this rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GjV6hkDohf", "reviewer": "Reviewer_iNA2", "summary": "The paper introduces an evaluation method for persona-driven role-playing (PRP) using the Active-Passive-Constraint (APC) scoring system. This system measures the faithfulness of AI responses to predefined persona statements by calculating APC scores and applying Direct Preference Optimization (DPO) to improve AI character adherence to personas. The authors validate the effectiveness of the APC scoring system through various experiments, demonstrating its applicability and improvements over existing methods.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed APC scoring system offers a nuanced approach to evaluating PRP, addressing limitations in existing coarse-grained methods by providing a detailed and explainable metric.\n\nThe application of APC-based DPO as a reward system for enhancing AI character faithfulness is innovative and effectively demonstrated through experiments.\n\nThe paper introduces Contextual Preference Optimization, further refining the evaluation and optimization process, showcasing a comprehensive approach to improving PRP methods.\n\nThe detailed methodology for calculating APC scores and integrating them with DPO is well-structured and validated, providing a robust framework for future research.", "weaknesses": "The complexity of the proposed methodology might limit its accessibility and reproducibility. Simplifying or providing clearer explanations for key components could enhance understanding and adoption.\n\nThe paper lacks justification for selecting specific models like Deberta and Gemma. A comparison with other potential models could strengthen the argument for their use.\n\nThe experiments presented are minimal, making it difficult to generalize the findings. More extensive experiments, including diverse scenarios and models, would provide stronger support for the proposed methodology.\n\nThere are redundant explanations regarding Long-Context Memory (LCM) and Retrieval-Augmented Generation (RAG) methods. Streamlining these sections could improve the paper's clarity and focus.\n\nThe results in the tables are not sufficiently explained. Better captions, detailed discussions, and visual aids could enhance the readability and interpretability of the data, ensuring that each value's significance is clear.\n\nThe paper does not provide a clear comparison with simpler baseline methods for evaluating PRP faithfulness. Including such comparisons would highlight the improvements and advantages of the APC scoring system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive attitude towards the quality and contribution of our work. We also find your suggestions are insightful and beneficial to polish our work. We will include the following experiments and clarification to further solidify our conclusions and address your concerns.\n\n## **Experiment scope**\n\nWe add 9 new original characters to our experiments, as in the **Experiment Comprehensiveness** section of **Author Rebuttal by Authors**. We not only concentrate on the numbers of the characters but also design the characters to be with different spatial (Asian, European, African, etc.) and temporal (past, present, future) conditions. We also include fantasy characters like a time-traveling scientist, thus broadening the generalizability verified from the experiment. \n\nThe results of these new diverse characters **(1st and 2nd tables in Author Rebuttal by Authors)** achieve a consistent conclusion with the existing experiments, thus further strengthening the generalizability of our discovery with a broader scope of characters. We will further include human evaluation for these new characters in the final version of our paper.\n\n## **Metric comparison**\n\nWe follow your suggestion to add a simple baseline - directly prompting the LLM to produce a coarse-grained score as the baseline in the **Metric Justification** section in **Author Rebuttal by Authors**, whose limitation is discussed in the current introduction section.\n\nAs the results presented in the **3rd table in Author Rebuttal by Authors**, the fine-grained APC score is more consistent with human evaluation and stable to the number of persona statements. Thus, our claim that a fine-grained metric like the APC score will be better for role-playing faithfulness evaluation is better supported.\n\n## **Justification for selecting specific models**\n\nFor the student model, we choose DeBERTa because it empirically performs well on classic NLU tasks like Relevance and NLI, where pre-trained encoders like BERT and RoBERTa excel (according to the GLUE benchmark). To further explain why we used DeBERTa in our experiments, we included a comparison of the distillation efficiency and evaluation efficiency of student models in the **Implementation Justification** section of the **Author Rebuttal by Authors**.\n\nThe results in the **4th table of the Author Rebuttal by Authors** show that DeBERTa can match LLMs like Gemma in distillation performance while greatly improving evaluation efficiency. Therefore, we selected DeBERTa as the student model for efficient experiments.\n\nFor the role-playing base model, the requirement to support our claims is the knowledge of the famous figures and the ignorance of the original characters. Thus, most existing LLMs meet this requirement and we select Gemma, which is a state-of-the-art LLM at the time of the experiments. In the final version of our paper, we will include experiments on recent LLMs that show strong abilities in various domains, such as Llama-3 and Mistral, to further verify the generalizability of our conclusion.\n\n## **Writing and content organization**\n\nWe will simplify the flow to introduce the key components to make it easier to understand our contributions. More specifically, we will include a table for denotations which the readers can refer to to understand the attributes and function of the introduced key components. We will also reduce the redundancy when explaining RAG and LCM by streamlining these sections, such as referring to section 3.2 in section 5.2 and concentrating more on the method setup. Finally, we will further polish the quality of captions, detailed discussions, and visualization. For example, we add more details to the caption of Figure 3 to further clarify the explanations of the two subfigures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XZcfmVJR4B", "reviewer": "Reviewer_Pxme", "summary": "This paper introduces the Active-Passive-Constraint (APC) score to evaluate and optimize the faithfulness of AI-driven persona interactions in role-playing applications. The authors propose a novel method by quantifying interactions using a fine-grained, constraint-based scoring system, significantly advancing the granularity of PRP evaluations. The methodology employs a NLI model distilled from GPT-4 for efficient and consistent evaluations, validated through experiments that demonstrate a high correlation with human judgment.\nAnother key contribution is using  APC score as a preference target for Direct Preference Optimization (DPO)， offering a new insight into improving PRP systems.", "questions": "1. Why is Regularization APC considered necessary, and how should it be chosen in relation to the APC score when in use?\n\n2. APC-Based DPO is one of the primary contributions of this study; however, Appendix D mentions that only 100 preference data points were used for training. Could this small amount of data lead to significant model overfitting?\n\n3. Table 1, 4, and 5 show a huge difference in APC scores between simple characters and famous characters (6 vs 400). What could be the possible reasons for this discrepancy, and what does the absolute magnitude of the APC score signify?", "limitations": "The author elaborates the limilation of this study.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Assessing role-playing is a highly important yet challenging task, and the method proposed in this paper is simple, user-friendly, and quantifiable. This contributes significantly to the rapid iteration within the Role Playing LLM field.\n2. This study is not limited to an assessment method; it also serves as an optimization target that improves the global faithfulness of AI characters.", "weaknesses": "1. The most concern is that the main experiment covers too few characters(3 simple characters and 6 famous characters, total 9), which makes it difficult to ensure the reliability of the conclusions.\n\n2.The APC Score relies on an additionally trained NLI Discriminator, which, although claimed to have an accuracy of 90% in this paper, lacks provided details on its training and the volume of data used. There is a question regarding whether this 300M classifier model can generalize to a broader range of characters.\n\n3.The paper lacks citation of the research[1] that is similar in core idea to this study, which also focuses on enhancing role-playing effectiveness by limiting the knowledge boundaries of characters to ensure their responses are confined to what they are known.\n\n[1]Lu K, Yu B, Zhou C, et al. Large language models are superpositions of all characters: Attaining arbitrary role-play via self-alignment[J]. arXiv preprint arXiv:2401.12474, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very pleased to receive your positive recommendation for our work. Your suggestions and questions are insightful and valuable to further improve the quality of our paper's content and writing. We will address your concerns with the following clarification and experiments.\n\n## **Character Coverage**\n\nWe broaden the character coverage in our experiments by adding 9 new original characters, as in the **Experiment Comprehensiveness** section of **Author Rebuttal by Authors**. We not only concentrate on the numbers of the characters but also design the characters to be with different spatial (Asian, European, African, etc.) and temporal (past, present, future) conditions. We also include fantasy characters like a time-traveling scientist, thus broadening the generalizability verified from the experiment.\n\nThe results of these new diverse characters **(1st and 2nd tables in Author Rebuttal by Authors)** achieve a consistent conclusion with the existing experiments, thus further strengthening the generalizability of our discovery with a broader scope of characters. We will further include human evaluation for these new characters in the final version of our paper.\n\n## **Generalizability of the distilled discriminators**\n\nFor the student discriminator distilled to perform relevance and NLI evaluation, we choose DeBERTa because it generalizes well on classic NLU tasks including Relevance and NLI, where pre-trained encoders like BERT and RoBERTa show strong generalizability (out-of-domain MNLI for example). To further justify the generalizability of DeBERTa in our experiments, we included a comparison of the distillation efficiency and evaluation efficiency of student models in the **Implementation Justification** section (including the detailed datasets statistics) of the **Author Rebuttal by Authors**. The fine-tuning hyperparameters for distillation can be found in current Appendix D.\n\nThe training results in the **4th table of the Author Rebuttal by Authors** show that DeBERTa can rival LLMs like Gemma in cross-character performance while greatly improving evaluation efficiency. Therefore, we selected DeBERTa as the student model for evaluation efficiency.\n\n## **Missing citations**\n\nThank you for mentioning the missing citations, we will cite more papers that enhance role-playing effectiveness by limiting the knowledge boundaries, which include [1].\n\n## **Questions**\n\n**Why is regularized APC considered necessary?** Regularized APC ($\\Delta$APC) represents the difference between the method's APC score and the APC score of an oracle that always gives a neutral response in NLI. Thus APC and $\\Delta$APC have no difference when ranking role-playing methods. As mentioned in section 4.2 of our paper, $\\Delta$APC is introduced to improve the meaning of the absolute value (How the method is better than a dumb system that always outputs \"Hello\"?). As shown in Table 1 in the current paper, the vanilla Gemma model without any knowledge about the original characters shows ~0 $\\Delta$APC score. For evaluation, $\\Delta$APC is generally preferred because it shows the difference with the simplest baseline. There are also cases preferring the unregularized APC score like estimating the difference with a perfect role-playing system which gains an unregularized APC score equal to the number of persona statements.\n\n**Could the small amount of DPO data lead to significant model overfitting?** Even though 100 training cases may seem small, each case in DPO covers a different question relevant to multiple persona statements, making it more comprehensive than it appears. Also, DPO does not continuously amplify the possibility of the preferred cases but learns a relative difference between the preferred and the rejected cases, which avoids overfitting. In our experiments, the questions used for DPO annotation and for testing are different, and the DPO still benefits the testing result. This experiment result also indicates the DPO is not overfitting but generalizes the global faithfulness to unseen questions. Finally, as the DPO data are easy to collect (only requires a question to prompt the role-playing model), one can easily increase the number of data for DPO to further generalize global faithfulness.\n\n**Why do Tables 1, 4, and 5 show a huge difference in APC scores?** Table 1 shows a regularized APC score, which is the difference between the method's APC score and the APC score of an oracle that always gives a neutral response in NLI. Tables 4 and 5 show the unregularized APC score and thus is higher. The huge difference in APC scores between simple characters and famous characters is a result of the value scaling of the APC score with the number of persona statements. APC score essentially estimates the number of persona statements satisfied by the response, thus those famous characters with more persona statements will generally have higher unregularized APC scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are sincerely thankful for all reviewers' positive feedback and insightful suggestions to improve the quality of our work. We are glad to address your concerns with further clarification and more experiment results for support. Here we include the responses to weaknesses and questions mentioned by multiple reviewers as a reference for rebuttals.

## **Experiment Comprehensiveness**

We completely agree with the importance of more characters with demographic-level population representation to solidify our conclusion. Thus, we add experiments (as in Table 1) for 9 new carefully designed characters each with 30 original persona statements in their profile for experiment efficiency, which include:

- Alex: An African American baseball player
- Isabella: An Italian traveling cook
- Takayoshi: A Japanese game developer

(Characters in the past)

- Ousmane: A rich gold mine owner of the Malian Empire in the 1300s
- Jones: A young British worker in the Victorian Era
- Zhe: A Chinese poet in the Tang Dynasty

(Characters in the fantasy)

- Crossan: A time-traveling scientist
- Betty: A pet cat who can talk with ghosts
- X: An alien space traveler and photographer

These characters can better represent people with different spatial and temporal backgrounds and even cover non-human characters from the fantasy world. The specific persona statements will be attached to the final version of our paper.

The results are shown as follows,

***(DeBERTa Evaluator)***

|Character|Alex|Isabella|Takayoshi|Ousmane|Jones|Zhe|Crossan|Betty|X|
|---|---|---|---|---|---|---|---|---|---|
|Vanilla|0.5|0.8|0.6|0.3|0.9|1.1|0.3|0.3|0.7|
|EU|1.8|2.8|2.0|1.4|0.7|3.8|2.0|1.2|5.2|
|LCM|7.1|7.4|6.5|4.5|6.2|5.2|2.2|2.8|8.1|
|RAG|7.6|8.1|6.9|3.0|6.6|5.8|1.8|3.2|7.5|
|EU w/ CPO|5.3|6.1|5.7|3.6|4.8|4.9|3.1|2.9|7.9|
|LCM w/ CPO|7.5|7.7|7.0|**4.8**|6.2|5.4|**4.5**|3.9|8.2|
|RAG w/ CPO|**7.9**|**8.2**|**7.4**|3.9|**7.5**|**6.9**|2.5|**4.6**|**8.9**|

***(GPT-4 Evaluator)***

|Character|Alex|Isabella|Takayoshi|Ousmane|Jones|Zhe|Crossan|Betty|X|
|---|---|---|---|---|---|---|---|---|---|
|Vanilla|0.2|0.1|-0.2|0.5|0.2|0.2|0.4|0.1|0.8|
|EU|1.4|1.8|3.0|0.5|1.3|6.4|1.2|0.3|7.4|
|LCM|3.1|8.6|5.6|4.1|7.4|3.4|2.1|1.6|11.3|
|RAG|3.3|7.8|6.1|1.6|8.1|4.3|2.7|2.2|10.1|
|EU w/ CPO|2.7|5.6|5.9|3.0|4.7|7.1|2.2|1.5|9.5|
|LCM w/ CPO|3.2|9.8|8.1|**4.6**|8.2|**7.8**|**4.0**|2.3|12.1|
|RAG w/ CPO|**4.8**|**10.0**|**9.8**|2.0|**8.3**|7.3|2.9|**3.1**|**14.6**|

The result is consistent with the conclusions in our paper that RAG outperforms other methods and w/ APC-based DPO generally RAG shows the best performance. We will attach this table and specific profiles of these 9 new characters to the appendix of our paper. We will also include human evaluation for these characters in the final version of our paper.

## **Metric Justification**

To better justify selecting our APC score and also support the claim that the fine-grained APC score has the advantage over coarse-grained metrics, we add a coarse-grained metric as the baseline. We directly prompt GPT-4 with the criterion used for human evaluation shown in Appendix E. We also distill this scoring ability (following the same scenario as APC) to DeBERTa to check whether the efficiency can be boosted. We evaluate the Spearman correlation between the metric and the human evaluation of the 7 role-playing methods on the 3 human-evaluated characters.

|Character (#Persona Statement)|Alice (8)|Bob (19)|Eve (30)|
|---|---|---|---|
|Coarse-grained Score (GPT-4)|92.42|86.27|81.40|
|APC Score (GPT-4)|97.18|99.10|99.10|
|Coarse-grained Score (DeBERTa)|81.40|69.91|54.57|
|APC Score (DeBERTa)|88.61|95.50|99.10|

The results verify that 1) Fine-grained APC score shows better consistency with human evaluation. 2) The fine-grained APC score is stable to the number of persona statements while the coarse-grained score degrades with the increase of persona statements. 3) The coarse-grained evaluating ability is harder to be distilled into smaller models for efficiency boosting. Based on case checking, we find an underlying issue of the coarse-grained metric is the LLM will assign a high score to a response once it contains some correct information, ignoring the missing important information (active constraint) and occasionally conflictions (passive constraint). We will further include the evaluation of the new 9 characters together with the human evaluation in the final version of our paper.

## **Implementation Justification**

We select DeBERTa as the student model to distill from GPT-4 because small encoders (BERT, RoBERTa, etc.) show promising performance on relevance and NLI, which are classic NLU tasks in the GLUE benchmark. Among encoders, DeBERTa (DeBERTa-v3-large) is a state-of-the-art model that shows strong performance after fine-tuned on NLU tasks. To further verify DeBERTa as a proficient student model, we add an analysis of the in-domain/out-of-domain performance and the efficiency of different base models for distillation.

|Task|Rel. (In-domain)|Rel. (Out-of-domain)|NLI (In-domain)|NLI (Out-of-domain)|Efficiency|
|---|---|---|---|---|---|
|DeBERTa (Base)|92.46|89.90|89.72|87.80|409.6it/s|
|DeBERTa (Large)|94.04|92.10|93.46|91.50|150.8it/s|
|Gemma-1.1-it (2b)|94.25|92.50|93.68|91.80|26.4it/s|

The in-domain test set (1697 instances for Relevance, 3773 instances for NLI) is the 20% split of the characters (Beethoven, Newton, Socrates) that build the training set (6787 instances for Relevance, 15092 instances for NLI). The out-of-domain test set samples 1000 cases from other characters. The results show DeBERTa-V3-Large (300M) shows a comparative performance with a 2B Gemma model, while is about 6 times faster, which justifies DeBERTa to be a strong student model. The out-of-domain performance is generally high, which indicates the generalizability to other characters. Finally, an extra discovery is that DeBERTa-v3-base (100M) can further significantly boost efficiency with some trade-offs in accuracy.

Meta-review: The paper introduces a novel Active-Passive-Constraint (APC) score to evaluate persona-driven role-playing (PRP) faithfulness. APC score correlates with human evaluation and enhances AI character responses using direct preference optimization (DPO).

The reviews have raised questions around limitations on generalizability of the evaluation, scaling up and the benefits of using a constraint satisfaction approach to persona faithfulness in AI characters and how can a system like APC account for the dynamic nature of human personalities and behaviors. The authors have answered all the questions and updated the submission with additional experiments. 

The authors recently added additional experiment results reinforcing previous questions including overfitting to specific characters, and to support the lack of human evaluation for the new metric, they introduced a new coarse-grained metric (though no human eval is conducted on the new metric)
Decision: Accept (poster)