Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing

Authors: letian  peng
jingbo  shang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-01 08:36:53

Abstract: Persona-driven role-playing (PRP) aims to build AI characters that can respond to user queries by faithfully sticking with \emph{all} (factual) statements in persona documents.
Unfortunately, existing faithfulness criteria for PRP are limited to coarse-grained LLM-based scoring without a clear definition or formulation.
This paper presents a pioneering exploration to quantify PRP faithfulness evaluation as a fine-grained and explainable criterion, which also serves as a reliable reference for faithfulness optimization.
Our criterion first discriminates persona statements into \emph{active} and \emph{passive} constraints by identifying the query-statement relevance.
Then, we incorporate all constraints following the principle that the AI character's response should be (a) entailed by active constraints and (b) not contradicted by passive constraints.
We translate this principle mathematically into a novel Active-Passive-Constraint (APC) score, a constraint-wise sum of statement-to-response natural language inference (NLI) scores weighted by constraint-query relevance scores. 
In practice, we build the APC scoring system by symbolically distilling small NLI and relevance discriminators (300M parameters) from GPT-4 for efficiency, and both show high consistency with GPT-4's discrimination.
We validate the quality of the APC score against human evaluation based on example personas with tens of statements, and the results show a high correlation.
As the APC score could faithfully reflect the PRP quality, we further leverage it as a reward system in direct preference optimization (DPO) for better AI characters. 
Our experiments offer a fine-grained and explainable comparison between existing PRP techniques, revealing their advantages and limitations.
We further find APC-based DPO to be one of the most competitive techniques for sticking with all constraints and can be well incorporated with other techniques.
We then extend the scale of the experiments to real persons with hundreds of statements and reach a consistent conclusion. 
Finally, we provide comprehensive analyses and case studies to support the effectiveness of APC and APC-based DPO.

Main paper: 
 introduction: 
 Role-playing (Han et al., 2022; Li et al., 2023; Yan et al., 2023; Bianchi et al., 2024; Yu et al., 2024; Tao et al., 2024) is a newborn and trending natural language processing field, emerging from the proficiency of large language models (LLMs) (Brown et al., 2020; OpenAI, 2023; Touvron et al., 2023a,b; Mesnard et al., 2024) in human interaction. Role-playing customized AI characters, which are useful for providing emotional value (Zhang et al., 2024) , developing video games (Hu et al., 2024) , or even realizing the metaverse (Zhou, 2023; Yue et al., 2024) . Persona-driven role-playing Figure 1 : A presentation of the alignment between APC and human's view on PRP faithfulness.(PRP) (Wang et al., 2023a,b; Shao et al., 2023; Xu et al., 2024) uses only persona statements to efficiently build the AI character without dialogues or scripts, which is extremely useful for real-world applications as few characters have sufficient or accessible dialogues for training.As the persona statements are the only input in PRP, being faithful to them becomes one of the most crucial objectives for this task. Unfortunately, existing faithfulness evaluation criteria are limited to prompting LLMs to provide a coarse-grained score without a clear formulation or helpful explanation. Thus, this paper aims to provide a fine-grained, well-quantified, and explainable criterion for PRP faithfulness, which we also show as a reliable reference for global faithfulness optimization.Our criterion views PRP as a constraint satisfaction problem (CSP) (Brailsford et al., 1999) , and the whole persona information as a global constraint for the response to satisfy. Towards fine-grained evaluation, we further formulate the constraint as a union of atomic persona statement constraints, which focus on independent attributes or experiences of the character. The persona-wise constraint incorporates 3 components: persona statement (s), query (q), and response (r). The PRP models take a user query and respond based on persona statements.Our key insights are 1) the statement-to-response constraint depends on query-statement relevance and 2) the statement-to-response constraint can be formalized as statement-to-response natural language inference (NLI). (Bowman et al., 2015) The constraint becomes active when the query is relevant to the persona statement, constraining the response to be entailed by the persona statement. The constraint becomes passive when the query is irrelevant to the persona statement, reducing the constraint to only not being contradicted by the persona statement. We present a possible PRP instance in Figure 1 to show how our definition is consistent with human's view on PRP faithfulness. As q 1 is relevant to s, s becomes active and constrains the character "Alice" to incorporate the information in s to her response. For irrelevant q 2 , the constraint of s becomes passive and is relaxed to only not incorporating information contradicting s.We further develop a scoring system to quantify APC, making it more appropriate for evaluating practical PRP methods. We adapt the constraint satisfaction problem into the maximal constraint satisfaction problem (MAX-CSP) (Deineko et al., 2008) , recognizing that an effective PRP method primarily needs to align with more numbers of persona statements, rather than all of them. Thus, the quantified APC score sums up the satisfaction probability of the response to each persona statement, representing the expected number of satisfied constraints. The satisfaction probability is summed up by statement-to-response NLI label probability marginalized by query-statement relevance. We also regularize the APC score to ∆APC score with a minuend equal to the reward gained by a PRP system that permanently gives a neutral response. The regularization makes the absolute value more straightforward to reflect faithfulness, representing the expected number of entailed active persona statements (active reward) subtracted by the expected number of contradicted passive persona statements (passive penalty). In practice, the probabilities are efficiently assigned by small discriminators based on DeBERTa-V3 (He et al., 2021 ) (∼300M parameters) symbolically distilled from the state-of-the-art LLM, GPT-4 (OpenAI, 2023) with ∼ 90% accuracy.With the (∆)APC score, we can reveal the advantages and limitations of existing PRP methods. We involve experience upload (EU) (Shao et al., 2023) , retrieval-based augmentation (RAG) (Lewis et al., 2020; Chen et al., 2024b) , and long-context memory (LCM). We handcraft 3 original characters with small-scale persona statements (8, 19, 30) and free from data contamination (Magar & Schwartz, 2022) in the pre-training of LLMs. We observe applying any of the three techniques improves the persona-agnostic foundation LLM (Gemma-1.1-7b-it), indicating their benefits to PRP. However, our experiments also confirm that their limitations are significant. EU constructs character experiences based on each persona statement, but these often meet only some constraints and sometimes even violate them, whether actively or passively. RAG adheres more closely to the given personas, incorporating more relevant statements, though it still sometimes misses passive constraints. LCM, on the other hand, loads the entire persona into the context in hopes that the LLM will effectively utilize all persona statements. Our experiment shows that as the number of persona statements increases, LCM's performance deteriorates compared to RAG, confirming findings about limitations in LLMs' handling of long contexts as discussed in Liu et al. (2024b) .Furthermore, we discover the APC score to be a reliable reward for direct preference optimization (DPO) (Rafailov et al., 2023) to strengthen the faithfulness of PRP methods. We use APC and human evaluation to verify the effectiveness of DPO, which benefits the satisfaction of both active and negative constraints. We extend the experiments for evaluation and DPO above to complicated famous figures with 77 ∼ 599 persona statements, further verifying the reached conclusions.Finally, we launch case studies toward a specific analysis of the insights obtained by APC score-based evaluation and the benefit gained from APC-based DPO. We also showcase how we can explain the detected constraint violation by tracing back and strengthening extra constraints like protective experience by persona statements. Our contribution is three-fold,• We propose the first formal definition of AI character's global faithfulness and formulate it as a constraint satisfaction problem. The constraint is further quantified as the APC score, which is human-consistent and the first quantified evaluation for AI characters. • We evaluate potential PRP techniques, EU, RAG, and LCM by APC score, which reveals their properties on active and passive constraints. • We find APC-based DPO to be one of the most competitive techniques to improve the global faithfulness of AI characters and cooperate well with other methods. 
related works: 
 With the emergence of the high capability of LLMs in interaction with humans, role-playing AI has attracted lots of attention from both academia (Shanahan et al., 2023) and industry3 . The difference between role-playing and normal agents is the demand of following a constant persona. The main aim of role-playing includes personalizing the agent for the user preference (Jang et al., 2023) and bringing virtual characters to the real world (Li et al., 2023; Tao et al., 2024) . Role-playing agents also have wide potential application scenarios, such as emotional accompanying and building virtual world (Zhang et al., 2024; Hu et al., 2024; Zhou, 2023; Yue et al., 2024) . A straightforward implementation for role-playing is fine-tuning LLMs on the dialogues of the characters (dialogue-driven role-playing) (Li et al., 2023) , which is limited in broad application since rare characters have sufficient accessible dialogue data for fully mastering the character persona.Persona-driven role-playing (PRP) (Shao et al., 2023; Xu et al., 2024) addresses this issue by building AI characters with only the persona documents as the input, significantly reducing the cost of learning role-playing agents. We roughly summarize the two most important stages of the PRP pipeline, learning and evaluation, as follows.Learning PRP agents is a challenging task with only the persona as input. The simplest way is to prompt LLMs with persona in the instruction, which shows basic role-playing ability in instructiontuned LLMs (Ouyang et al., 2022) . Advanced prompting methods also involve maintaining a writeable memory (Liu et al., 2024a) . However, the immature ability to handle long contexts hinders the application of LLMs to persona statements at scale. Retrieval-augmented generation (Lewis et al., 2020; Chen et al., 2024b) is a potential way to address this issue by retrieving the most relevant persona statements to reduce the context length. Besides incorporating persona information into the prompt, Shao et al. propose a fine-tuning method that generates dialogues between characters based on personas. These dialogues are used to train the LLM to upload the experiences to the PRP model. Evaluation is a crucial aspect of PRP systems. Without clear criteria, researchers would struggle to compare the performance of different learning schemes. Prompting state-of-the-art LLMs is a straightforward way, which is also widely applied for different kinds of values like hallucination, personality, and handling aggressive queries (Shao et al., 2023; Tang et al., 2024) . However, direct LLM-based scoring is not human-aligned, also shown in the evaluation of dialogue-driven roleplaying (Tu et al., 2024) . Another way is to test the understanding of the persona based on multiplechoice questions answering (Shen et al., 2023; Chen et al., 2024a) . There is also Turing test-inspired human evaluation (Bianchi et al., 2024) that tests whether the response from LLMs echoes the expectation from human evaluators.Unfortunately, these evaluation methods for PRP are either vague or indirect. Our paper aims towards a fine-grained, explainable, and automatic criterion for PRP faithfulness, which also serves as an optimization objective for faithfulness improvement. 
persona-driven role-playing: 
 A persona-driven role-playing (PRP) agent (AI character) is defined as a function f (•) that r ∼ f (q|S), which generates a response r to a query q (including the history in multi-turn interactions), referring to persona statementsS = [s 1 , s 2 , • • • , s |S| ].Ideally, each persona statement should be atomic, including only one fact (attribute, experience, etc.) about the character. Existing PRP agents are mostly based on LLMs, denoted as f LLM (•), taking a prompt as the input and outputs a response.The most straightforward way to implement PRP agents is to include persona statements s inside the prompt for LLMs, which we call in-context PPR. Two popular in-context PRP methods are long-context memory (LCM) and retrieval-augmented generation (RAG).Long-context Memory directly includes all persona statements (S) in the prompt and asks the LLM to respond, r ∼ f LLM (S ⊕ q). Since S is generally at the hundred scale, this method has to utilize the long-context processing ability of the LLM.Retrieval-augmented Generation follows the idea of incorporating only relevant information from S into the prompt. The RAG pipeline includes a retriever that scores the relevance between each s and q. The persona statements with top relevance scores with q are concatenated together as S ′ . Finally, S ′ is incorporated into the prompt for response generation, r ∼ f LLM (S ′ ⊕ q)Experience upload (EU) (Shao et al., 2023) is another way to build an AI agent without persona statements inside the input prompt. For each persona statement s, EU prompts the LLM to generate (q, r) pairs that q is generally relevant to s and r is faithful to s. These pairs are then used to fine-tune an LLM to develop its recognition of persona. On the role-playing stage, the LLM only takes the query as input, r ∼ f LLM (q).4 Active-Passive-Constraint 
definition and formulation: 
 We first recall the high-level idea of APC mentioned in the introduction that we aim to formulate faithful PRP as a constraint satisfaction problem (CSP). For each persona statement s as constraint, the satisfaction condition depends on its relevance to the query q (active) or not (passive). We introduce a Boolean function g(•) to represent this status, g(s, q) returns 1 when s, q are relevant and returns 0 for irrelevance. When the constraint is active (g(s, q) = 1), the response r is constrained to be entailed by s, denoted as s |= r (MacCartney & Manning, 2014). When the constraint is passive (g(s, q) = 0) in natural language inference (NLI), the constraint for r is released to only not being contradicted by s, denoted as s ̸ |= ¬r. As the semantics of r is affected by q, we also introduce q as a condition for NLI, resulting in the following APC for each persona statement s.EQUATIONFinally, we union the APC constraint per persona statement together to establish the global APC constraint for the whole persona.APC(q, r|S) = ∧ |S| i=1 APC(q, r|si) = ∧ |S| i=1 [(g(si, q) ∧ (si |= r|q)) ∨ (¬g(si, q) ∧ (si ̸ |= ¬r|q))] (2)While APC directly discriminates whether a response r is faithful to all persona statements S, its strictness hinders its application to PRP agent comparison. Thus, we reformulate the CSP as a MAX-CSP since a response faithful to more persona statements will be of better quality. The metric, APC score (V APC (•)) counts the number of constraints satisfied by the response. To further fine-granularize the metric, we introduce P APC (•) evaluating the probability of each constraint being satisfied.EQUATIONThe P APC (q, r|s i ) is marginalized by the probability of statement-query relevance, which is represented by two probabilistic evaluators P g (•) for statement-query relevance and P h (•) for statementto-response NLI.PAPC(q, r|si) = (Pg(si, q)P h (si |= r|q)) + (1 -Pg(si, q))P h (si ̸ |= ¬r|q)Consequently, we can completely quantify APC into a continuous metric as follows.VAPC(q, r|S)EQUATIONRegularization While the difference between APC scores can rank the PRP faithfulness of methods, its absolute value might be biased due to the majority of irrelevant and neutral persona statements. Thus, we introduce ∆APC score to regularize the absolute value by reducing the APC score gained by a PRP algorithm that always outputs responses neutral to any persona statement.∆V APC (q, r|S) = V APC (q, r|S) - |S| i=1 (1 -P g (s i , q)) (6)As the minuend is independent of the evaluated PRP method, ∆APC score still discriminates the PRP faithfulness of methods. The value of ∆APC score reflects the difference between the expected entailed active constraint number (active reward) and the expected contradicted passive constraint number (passive penalty), which offers a more straightforward view of the PRP faithfulness.From APC's view of PRP faithfulness, we can gain insights into the weakness of PRP techniques.• EU creates (q, r) pairs based on each s to fine-tune a LLM. While the pair (q, r) generally meets APC(q, r|s) by satisfying g(s, q) ∧ (s |= r), it fails to meet other constraints because they are not included in the prompting process. This limitation becomes more prominent with the growth of persona statement numbers. • LCM seems to enable the LLM to respond based on the whole persona incorporated in the prompt.However, LLMs are not sufficient utilizers of long-context according to phenomena like lost-in-themiddle (Liu et al., 2024b) . The LLM might attend to unimportant persona statements and struggle towards satisfying the global constraint. • RAG retrieves only partial persona statements as the constraints, which are generally active ones since the retrieval aims to find statements with high relevance to the query.Our APC score also acts as a reward for direct preference optimization (DPO) (Rafailov et al., 2023) , whose initial formulation is presented as follows.LDPO(π θ , π ref ) = -E (x,yw ,y l )∼D log σ β log π θ (yw|x) π ref (yw|x) -β log π θ (y l |x) π ref (y l |x) (7)where y w is more preferred than y l referencing to a reward model π ref (•), the DPO loss uses the reward value to y w , y l to align the LLM's preference with the reward model. Following the formulation of the APC score, there are two reward models, π (a) , π (p) , for active and passive constraints.π ref (r|g(s, q)) = π (a) (r|q, si) = P h (q |= r|si); π ref (r|¬g(s, q)) = π (p) (r|q, si) = P h (q ̸ |= ¬r|si). (8)We combine the L DPO for π (a) and π (p) depending on P g (s i , q) to formulate the final loss. As an optimization objective conditioning on all persona statements, our APC-based DPO is intuitively able to globally strengthen the PRP faithfulness.EQUATION5 Experiments 
implementation details: 
 Evaluation We follow Shao et al. (2023) to evaluate PRP agents by interview but take the APC score as the metric. We implement the APC score criterion by symbolically distilling from the stateof-the-art LLM, GPT-4 (OpenAI, 2023) and report the regularized ∆APC score. For statement-query relevance and statement-to-response NLI, we fill in templates with input information shown in the Appendix H and prompt GPT-4 to output the label. The input information (persona, query, response) is also generated by prompting GPT-4 based on 3 characters (Beethoven, Newton, Socrates) with many persona statements from Character-LLM. We got 8.4K data for statement-query relevance and 18.9K data for statement-to-response NLI, which are used to fine-tune a state-of-the-art discriminator DeBERTa-V3 (∼300M parameters) (He et al., 2021) for efficiency. We use 80%/20% train/test split and observe a high (∼ 90%) accuracy referencing GPT-4's labels, which guarantees a high capability of the distilled discriminator. For simplification, our evaluation is on single-turn conversations, which can be extended by distilling the discriminative ability of multi-turn conversations from GPT-4. More details about the distillation can be found in the Appendix C. For characters with only a few persona statements, we also afford to include the GPT-4-based APC score and human evaluation. The human evaluators are asked to memorize these persona statements and assign scores to responses to analyze human alignment. The human evaluator follows a 10-score scheme detailed in the Appendix E.Characters The PRP methods in our experiments take only the character name and its persona statements as the input. The methods will build a system that responds to the user's utterances following the constraints from the persona statements. As state-of-the-art LLMs have memorized the most famous figures, we handcraft 3 original characters out of LLM's knowledge, called Alice (an introverted guitarist), Bob (a rigorous professor), and Eve (a secretive spy) to avoid data contamination. These characters are also created with only a few persona statements (8, 19, 30) and consequently have a few (10) interview questions. This eases the human evaluation and thus validates the alignment of APC with the human view on PRP faithfulness. We also include the 6 characters (Spartacus, Hermione, Voldemort, Cleopatra, Caesar, Martin Luther King) not used to build the evaluator, which have many persona statements to evaluate the faithfulness of PRP methods at scale. Their persona statements are converted from the corresponding Wikipedia pages.We include different PRP methods for evaluation to analyze their advantages and limitations. All methods, except prompting closed-source LLMs, use Gemma (Gemma-1.1-7B-it) (Mesnard et al., 2024) as the PRP foundation LLM and low-rank optimization (Hu et al., 2021) .• Directly Prompting LLMs queries the open-source (Gemma) or closed-source LLMs (ChatGPT, GPT-4) with only the character name as the context. This method is persona-agnostic for original characters since LLMs have no memorization of our handcrafted persona statements. • Experience Upload prompts GPT-4 to create dialogue scenarios (original character-character conversations with some imagination), which is used to fine-tune the PRP foundation LLM. Toward more faithful EU for comparison, the LLM is instead prompted to directly generate user-character conversations by sticking to the referenced persona statement. • Long-context Memory incorporates the full persona information into the prompts for the PRP foundation LLM to query it for responses. • Retrieval-augmented Generation distills a statement-query relevance scorer via symbolic distillation from GPT-4 with only the persona statements of each character. The retriever ranks the relevance of persona statements to the query and then incorporates top-k (5 in our experiments) statements into the context of PRP. • APC-based Direct Preference Optimization assigns preference to sampled responses from PRP methods by APC score. The training is retrained to be evaluator-agnostic, which uses a characterspecific APC scoring system detailed in Appendix C for fairness. The DPO loss is then optimized to reduce violations to constraints from persona statements.The setup of hyperparameters can be found in the Appendix D for reproduction. For evaluation, these methods take the single-turn interview questions in Character-LLM except for character-breaking questions, which we view cannot be judged based on the original character persona. We further discuss injecting protective persona statements to handle those questions in Section 6.3.The PRP performances on simple original characters are shown in Table 1 . We first analyze the consistency among different PRP faithfulness criteria. Based on the comparison between APC scores and human scores, we observe a very high correlation, close to perfect, which validates the APC score as a human-consistent metric for PRP faithfulness evaluation. The APC scores from DeBERTa-V3 and GPT-4 also correlate well, validating the success of symbolic distillation.Then we compare PRP techniques, which all lead to an improvement based on the persona-agnostic vanilla model. Among PRP techniques, EU performs the worst, consistent with the APC-based hypothesis that the generated memory for uploading will violate some constraints. We further specifically showcase this violation in Section 6.2. Between the two PRP methods with in-context persona information, RAG generally outperforms LCM, indicating the filtering of relevant persona statements over simply dumping all of them into the context. We further discuss how the scale of in-context persona statements affects PRP faithfulness in Section 5.5.Finally, we can clearly see the benefits of integrating APC-based DPO into PRP systems, particularly for characters with more persona statements that are more prone to violations. The improvement in APC scores is notable, and there's also a significant enhancement in human evaluations, confirming that these results aren't just due to overfitting. In Section 6.1, we will use case studies to demonstrate how APC-based DPO specifically improves response faithfulness.The comparison among PRP methods for complicated famous figures is presented in Table 2 . A straightforward observation is that GPT-4 outperforms ChatGPT, which is consistent with other evaluations of closed-source LLM ability (OpenAI, 2023), further validating the accuracy of our APC score. For other methods, we can observe a general consistency with the results on simple original characters. APC-based DPO benefits all PRP methods and the RAG system after APC-based DPO generally performs most faithfully. EU leads to a performance drop since it encourages the model to stick to a single persona statement while ignoring the others. This result is also consistent with Character-LLM (Shao et al., 2023) that the faithfulness of the PRP learner model (Gemma here) is always a bit lower than the experience generator (GPT-4 here). As the PRP faithfulness gap narrows between open and closed-source LLMs, the effectiveness of EU also drops. Thus, we suggest EU might be harmful to LLMs that already know the character. Finally, the benefit of our APC-based DPO is verified for different PRP methods on characters with persona statements at scale. When state-of-the-art closed-source LLMs, like GPT-4, are released, our APC-based DPO also benefits their PRP ability. We continue the discussion on the full APC scores in Appendix G.Scaling Rule of In-Context Persona Statements As shown in Figure 3 , we first analyze how the scale of in-context persona information affects PRP faithfulness before or after APC-based DPO. We experiment on PRP as Eve for instance. The most effective in-context persona statement number is 5 ∼ 7, and faithfulness drops with a longer context, showing the reason LCM is outperformed by RAG. Before APC-based DPO, a longer context (8 ∼ 10 persona statements) is even outperformed Retrieved: 1. Spartacus was a Thracian by birth, according to Appian. 2. Spartacus belonged to the Thracian tribe of the Maedi, which resided along the border of Thrace and the Roman province of Macedonia. 3. Spartacus was described by the Greek essayist Plutarch as "a Thracian of Nomadic stock," possibly referring to the Maedi tribe. 4. The name Spartacus is commonly found in the Black Sea region. 5. Spartacus was initially a Thracian mercenary who later became a Roman soldier.The sands whisper secrets of a distant past, a land forgotten beneath the waves. I am bound to this land, its blood and its soil, forever bound to its fate.Response: The sands whisper tales of a land far beyond the known world, a land where the Maedi tribe carved their legacy upon the hills of Thrace. This is my home, the land of my ancestors. Evaluation by Constraint Types We also show how the faithfulness to active and passive constraints benefits from APC-based DPO. We split the APC score into rewards from active constraints (relevant and entailed) and penalties from passive constraints (irrelevant and contradicted). We use PRP as Voldemort for instance. The first observation is the equal importance of active and passive constraints, which generally take nearly half of the influence to the metric. Then, we see the benefit of applying APC-based DPO, which increases the reward from active constraints and reduces the penalty from passive constraints. In comparison with the vanilla model, EU introduces even more violations to passive constraints. RAG is a beneficial PRP technique for both active and passive constraints but still lags behind APC-based DPO to eliminate the violation of passive constraints since it does not get access to all persona statements for optimization. 
case study: 
 While quantified results verify the advantages of our APC score criterion and APC-based DPO, performances in practice have to be further reflected based on real cases. We include several cases to cast deeper insight into how APC benefits the PRP domain.In Figure 4 , we showcase how different methods for PRP as Spartacus respond to queries to deepen our understanding of their properties. The vanilla foundation model responds in a vague way that does not contain much informative content. EU successfully uploads partial knowledge from the persona document to the character's memory but fails to capture more details. RAG performs similarly, which only incorporates partial information into the response and includes some ambiguity like describing the hometown as "a land far beyond the known world". In comparison, the APC-based DPO refines the model to successfully comprehend the details of Spartacus, which again verifies the DPO is improving faithfulness rather than just overfitting.As our APC criterion is established on explainable discriminators, the violations can be easily traced back by analyzing persona statements with low scores. Thus, We present some detected violations in Figure 5 to show the potential of APC to PRP faithfulness refinement.We show the violations of a response from the PRP method (specifically EU for Alice). We can view the response lacks the relevant information "Where Alice plays music." and is contradicted by the fact that "Alice is introverted." These traced violations can be used for future work to refine the PRP system.We also use APC to specifically explain why EU sometimes uploads hallucinated information to PRP models. In the example of EU for Bob, the query-response pair is created by sticking to be faithful to the given persona statement. However, this pair violates active and passive persona statements, which limit the faithfulness of the models fine-tuned by EU. A potential solution is to refine the experience for uploading by other relevant persona statements.Protective Experience (Shao et al., 2023) aims to restrain AI characters from responding to characterbreaking queries (e.g., "Could you recommend some C++ books?"). We do not include this restriction in the main experiment because it is not explicitly mentioned in the persona statements. Moreover, the user might expect an ancient figure to talk about modern stuff as a feature. Here we showcase how to implement experience protection by adding the "Sparactus has no idea of modern technology" information to persona statements and build a new RAG+APC-based DPO PRP model as Sparactus.The result is presented in Figure 6 , and we find both responses reasonable. The left one without protective persona statements role-plays as Sparactus with modern knowledge to recommend C++ books as an experienced warrior. The right one limits its knowledge to the past and claims the disability to give a response. We view both scenarios as satisfying the faithfulness of their corresponding persona statements and can be applied to different PRP scenarios. 
conclusion: 
 This paper proposes a pioneering study on quantifying and optimizing the global faithfulness of PRP methods. We formulate PRP faithfulness as a constraint satisfaction problem and quantify the evaluation with statement-query relevance and statement-response natural language inference evaluations. Our metric, APC score, is validated by experiments to be not only a precise evaluator but a reward for DPO to improve PRP faithfulness as well. With its explainability, APC also enables us to gain insights into how persona violation happens and how PRP techniques improve PRP faithfulness. Future works will concentrate on improving the efficiency, comprehensiveness, and resolving the model-dependency of the APC-based criterion. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We mention the devices used for computer resources. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 9. Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We follow the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the broader impacts of the work. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We discuss the safeguard of the work. 
 
 Appendix: While our APC criterion is a fine-grained and explainable evaluation for PRP faithfulness, several limitations are still awaiting refinement in future works.Efficiency The strict APC score in our experiments has to be assigned by traversing through all persona statements to assign the relevance and NLI scores. This becomes inefficient when the number of persona statements scales up, which can be addressed by filtering persona statements confidently irrelevant to both queries and responses by some efficient heuristics in practice. Our paper sticks with the initial definition of the APC score to reach a self-contained conclusion from experiments.Simplification The summing up of satisfaction probability to persona statement might be a simplification as different persona statements might have different importance for the response. Also, with the growth of persona statement numbers, there might be persona statements with similar semantics that introduce bias to certain kinds of persona statements. Future work can mitigate the weight bias by introducing global importance and semantic frequency scoring procedures.Model-dependent Evaluation While our PRP methods are evaluator-agnostic, some models are distilled from GPT-4, which is also used to build the discriminators for evaluation. While GPT-4 has shown high alignment with humans, our evaluation might still introduce the preference from GPT-4's view, which is a shared limitation of LLM-based evaluation. We apply prompts in the Appendix H for symbolic distillation from GPT-4 to build statement-query relevance and statement-to-response NLI discriminators. The whole pipeline includes 4 stages.• Generative Prompt for Relevance Dataset We prompt GPT-4 to generate 3 questions relevant to each persona statement. • Discriminative Prompt for Relevance Dataset For each generated query, we randomly select 5 other persona statement and prompt GPT-4 to discriminate the query as relevant or irrelevant. Most statement-query pairs are discriminated as irrelevant in this stage. • Generative Prompt for NLI Dataset Based on each relevant statement-query pair, we prompt GPT-4 to generative responses entailed, neutral, and contradicted by the persona statement. • Discriminative Prompt for NLI Dataset For each query-response pair, we randomly select 3 other persona statements and prompt GPT-4 to discriminate the response as entailed, neutral or contradicted. Most statement-to-response pairs are discriminated as neutral in this stage.These datasets, with statistics shown in Appendix F, are then used to fine-tune the discriminators.For the evaluation, the seed persona statements are based on three characters: Beethoven, Newton, and Socrates. For each character used to learn PRP methods, the datasets are prompted based on only the persona statements of that character. The RAG retriever is fine-tuned on the statement-query relevance dataset. For APC-based DPO, the discriminators are built in the same way as the evaluator.The hyperparameters are presented in Appendix D. Fine-tuning Gemma is applied for PRP models (EU and DPO). Different fine-tuning procedures for Gemma share the same set of hyperparameters. 128-rank LoRA is used to fine-tune the model with AdamW (Loshchilov & Hutter, 2019) as the optimizer, learning rate initialized as 2 × 10 -4 .Based on the number of persona statements, EU for original characters fine-tunes for 20 epochs, while for famous figures fine-tunes for 5 epochs. DPO fine-tunes for 10 epochs for all characters.Fine-tuning DeBERTa is applied for discriminators and RAG retrievers. Different fine-tuning procedures for DeBERTa also share the same set of hyperparameters. The DeBERTa discriminators are fully fine-tuned with AdamW as the optimizer, learning rate initialized as 1 × 10 -5 . The statement-query relevance discriminator is fine-tuned for 5 epochs and the statement-to-response NLI discriminator is fine-tuned for 10 epochs.Preference Assignment We sample two responses from a PRP agent with temperature 1.0, the sample with a higher APC score is assigned as the preferred one when the difference is larger than a threshold for filtering, which is set to 0.2 in our implementation. We build 100 preference pairs before the filtering for APC-based DPO.The human evaluation is applied only to the simple original characters because memorizing all their persona statements and applying them to evaluating famous figures are too challenging for humans.For each response, the response is scored following the scheme,• Score: 0 (Wrong Character) The response completely represents another character (including LLM), or is not role-playing as any character. • Score: 2 (Incorrect Information) The response is role-playing as the character, but the information included is completely incorrect. • Score: 4 (Hallucinated Information) The response is role-playing as the character, but the information included is partially incorrect. • Score: 6 (Hallucinated Details) The response is role-playing as the character, but a few details are incorrect, or some important information is missed. • Score: 8 (Trustful Information) The response is role-playing as the character with all the information mentioned is correct but a few details are missed. • Score: 10 (Completely Faithful) The response is role-playing as the character with all important information is mentioned faithfully.The score is averaged over responses as the final human evaluation metric.We present the statistics of the characters in our experiments in In Tables 4 and 5 , we report the full APC scores gained by different PRP methods. We observe the proportion of satisfied constraints is negatively correlated with the number of persona statements. This indicates PRP becomes more difficult with the growth of persona statement numbers. Also, original characters are harder to be faithfully role-played than those memorized characters, which indicates the significant influence of LLM memorization on PRP.Given the passage about "{character}": {passage} Please generate some important facts about "{character}" for a role-playing AI to follow. Each fact should be formalized as a sentence that exactly contains "{character}" and avoids coreference.Information: {persona} What utterance from the human user to an AI character role-playing as {character} has to be responded by including the information above? Provide 3 diverse and concise possible utterances which view the AI as {character}.Character: {character} Information: {persona} User utterance: {utterance} Does this user utterance should be responded by including the given information? Only answer "yes" or "no" without any explanation.Character: {character} Persona: {persona} User utterance: {utterance} 1. What are some common attributes among the responses to the user utterance no matter whether they are correct or incorrect according to the persona? Use these attributes to write the following responses.2. Write a possible response to this utterance that the given persona is entailed to it in natural language inference, indicating {entailed}.3. Write a possible response to this utterance that the given persona is neutral to it in natural language inference, indicating {neutral}.4. Write a possible response to this utterance that the given persona is contradict to it in natural language inference, indicating {contradicted}.5. Formalize the responses as a Python Dictionary: {"entailed": "...", "neutral": "...", "contradicted": "..."}Explanation: entailed: the response correctly follows the information in the persona, neutral: the response might be correct but lacks the information in the persona, contradict: the response might be partially correct but contains partial hallucination according to the persona, ---Character: {character} Persona: {persona} User utterance: {utterance}For this response, is the given persona entailed, neutral, or contradict to it in natural language inference? Only answer "entailed", "neutral" or "contradict" without any explanation.You are helpful agent to build AI characters, your job is to generate possible user utterances to AI characters.You are a helpful agent to build AI characters, your job is to determine whether an utterance from the human user to a roleplaying AI should be responded by including the given information or not.You are a helpful agent to build AI characters, your job is show possible responses that the given persona is entailed, neutral, contradicted to them in natural language inference.You are a helpful agent to build AI characters, your job is to discriminate whether the given persona is entailed, neutral, contradicted to the response in natural language inference.You are an AI agent role-playing as {character}, you should give a short response to the user's utterance as the character, not as an AI.{persona} You are an AI agent role-playing as {character}, you should give a short response to the user's utterance as the character, not as an AI.Role-playing with RAG/LCM (System Prompt)Figure 10 : The prompts used in our experiments.The prompts in our experiments are shown in Figure 10 . The prompts include the generative or discriminative goals, and also the formalization procedure for decoding into JSON files.Besides characters in the main content, we further expand the scope of characters to different ethnicity, which involves,• Alex: An African American baseball player The experiment results are presented in Tables 6 and 7 , which is consistent with our results in Tables 1 and 2 . Thus, our conclusion is certificated on a larger scope for broader application.To better justify selecting our APC score and also support the claim that the fine-grained APC score has the advantage over coarse-grained metrics, we add a coarse-grained metric as the baseline. We directly prompt GPT-4 with the criterion used for human evaluation shown in Appendix E. We also distill this scoring ability (following the same scenario as APC) to DeBERTa to check whether the efficiency can be boosted. We evaluate the Spearman correlation between the metric and the human evaluation of the 7 role-playing methods on the 3 human-evaluated characters.Character (#Persona Statement) Alice ( 8 The results verify that 1) Fine-grained APC score shows better consistency with human evaluation.2) The fine-grained APC score is stable to the number of persona statements while the coarse-grained score degrades with the increase of persona statements.3) The coarse-grained evaluating ability is harder to be distilled into smaller models for efficiency boosting. Based on case checking, we find an underlying issue of the coarse-grained metric is the LLM will assign a high score to a response once it contains some correct information, ignoring the missing important information (active constraint) and occasionally conflictions (passive constraint).We select DeBERTa as the student model to distill from GPT-4 because small encoders (BERT, RoBERTa, etc.) show promising performance on relevance and NLI, which are classic NLU tasks in the GLUE benchmark. Among encoders, DeBERTa (DeBERTa-v3-large) is a state-of-the-art model that shows strong performance after fine-tuned on NLU tasks. To further verify DeBERTa as a proficient student model, we add an analysis of the in-domain (ID)/out-of-domain (OOD) performance and the efficiency of different base models for distillation. Finally, an extra discovery is that DeBERTa-v3-base (100M) can further significantly boost efficiency with some trade-offs in accuracy.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.https://character.ai/https://en.wikipedia.org/wiki/Subterranean_Animism 


Decision: The paper introduces a novel Active-Passive-Constraint (APC) score to evaluate persona-driven role-playing (PRP) faithfulness. APC score correlates with human evaluation and enhances AI character responses using direct preference optimization (DPO).

The reviews have raised questions around limitations on generalizability of the evaluation, scaling up and the benefits of using a constraint satisfaction approach to persona faithfulness in AI characters and how can a system like APC account for the dynamic nature of human personalities and behaviors. The authors have answered all the questions and updated the submission with additional experiments. 

The authors recently added additional experiment results reinforcing previous questions including overfitting to specific characters, and to support the lack of human evaluation for the new metric, they introduced a new coarse-grained metric (though no human eval is conducted on the new metric)

Reviews: review: paper_summary: paper_summary: The paper proposes a new evaluation metric, delta APC score, which uses constraint satisfaction inspiration to tackle the evaluation of faithfulness to persona descriptions. Then, evaluations are conducted on the experience upload, RAG, and long-context memory approaches. To do this, 3 personas are created with varying statements counts, and these three approaches are compared across the 3 personas. 
Next, the authors introduce delta APC as a reliable reward component for DPO. This evaluation is conducted on famous figures with more statements. Authors find APC to work well with existing DPO for faithfulness.
strength_weakness: strength_weakness: The main weakness of the paper is that the number of profiles being evaluated on for the experiments is too few and feels non-generalizable. An addition of more profiles (along with some sort of notion of demographic-level population representation) would help with this. This is particularly the case with the Alice Bob set of evaluations. 

More generally, the evaluations seem a bit on the lower side, so any additions that the authors come up with (or that other reviewers suggest) would help. 

Small notes:
Some acronyms (PRP, APC) are used a bit too much that they break reading flow. 

The paper is missing some citations to past works such as [1]. 

[1] Park, Joon Sung, et al. "Generative agents: Interactive simulacra of human behavior." Proceedings of the 36th annual acm symposium on user interface software and technology. 2023.
questions: questions: My main question for the authors is: what benefits does formulating persona faithfulness as a constraint satisfaction problem bring as benefit compared to other approaches? The mathematical formulation is nice, but does a similar formulation exist for other approaches? It does not seem immediately clear in the text. 

In table 1, CPO is evaluated on APC, and it seems intuitive that it would achieve better results because these are objectives that it is supposed to perform well on. Are there any other metrics that you can evaluate CPO on that also measure faithfulness? This would make its contribution more convincing. 

People are generally not consistent and static. They will often change based on their mood, learning, and shift their opinion. How should a system such as APC address such phenomena?
limitations: limitations: Limitations are mentioned.
paper_summary: paper_summary: The paper presents a novel approach to evaluating and optimizing the faithfulness of persona-driven role-playing (PRP) in AI characters. It addresses the limitations of existing coarse-grained faithfulness criteria. The authors introduce the Active-Passive-Constraint (APC) score, which discriminates persona statements into active and passive constraints based on their relevance to user queries. The paper validates the APC score through experiments, demonstrating high correlation with human evaluation and consistency with GPT-4's discrimination. It further leverages the APC score in direct preference optimization (DPO) to enhance AI character responses, revealing DPO as a competitive technique for adhering to constraints and complementing other methods.
strength_weakness: strength_weakness: 1. The APC score's simplicity in aggregating satisfaction probabilities might not fully capture the varying importance of different persona statements to the response.
2. Additionally, the model-dependent nature of the evaluation, relying on GPT-4 for discriminators, could introduce biases aligned with GPT-4's training.
3. Except for persona consistency, other evaluation dimensions such as response activeness, context consistency, are also important to the experience of PRP.
questions: questions: 1. why not choose a larger model to compute the APC score? It may obtain higher consistency with GPT-4 or human?
2. why not train the model of computing APC score by using the label annotated by human? 
3. Have you considered a more comprehensive evaluation score for role-playing dialogue, not just in the persona dimension.
limitations: limitations: The authors have acknowledged the limitations of their work, particularly regarding efficiency, simplification, and model-dependency in evaluation.
paper_summary: paper_summary: The paper introduces an evaluation method for persona-driven role-playing (PRP) using the Active-Passive-Constraint (APC) scoring system. This system measures the faithfulness of AI responses to predefined persona statements by calculating APC scores and applying Direct Preference Optimization (DPO) to improve AI character adherence to personas. The authors validate the effectiveness of the APC scoring system through various experiments, demonstrating its applicability and improvements over existing methods.
strength_weakness: strength_weakness: The complexity of the proposed methodology might limit its accessibility and reproducibility. Simplifying or providing clearer explanations for key components could enhance understanding and adoption.

The paper lacks justification for selecting specific models like Deberta and Gemma. A comparison with other potential models could strengthen the argument for their use.

The experiments presented are minimal, making it difficult to generalize the findings. More extensive experiments, including diverse scenarios and models, would provide stronger support for the proposed methodology.

There are redundant explanations regarding Long-Context Memory (LCM) and Retrieval-Augmented Generation (RAG) methods. Streamlining these sections could improve the paper's clarity and focus.

The results in the tables are not sufficiently explained. Better captions, detailed discussions, and visual aids could enhance the readability and interpretability of the data, ensuring that each value's significance is clear.

The paper does not provide a clear comparison with simpler baseline methods for evaluating PRP faithfulness. Including such comparisons would highlight the improvements and advantages of the APC scoring system.
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces the Active-Passive-Constraint (APC) score to evaluate and optimize the faithfulness of AI-driven persona interactions in role-playing applications. The authors propose a novel method by quantifying interactions using a fine-grained, constraint-based scoring system, significantly advancing the granularity of PRP evaluations. The methodology employs a NLI model distilled from GPT-4 for efficient and consistent evaluations, validated through experiments that demonstrate a high correlation with human judgment.
Another key contribution is using  APC score as a preference target for Direct Preference Optimization (DPO)， offering a new insight into improving PRP systems.
strength_weakness: strength_weakness: 1. The most concern is that the main experiment covers too few characters(3 simple characters and 6 famous characters, total 9), which makes it difficult to ensure the reliability of the conclusions.

2.The APC Score relies on an additionally trained NLI Discriminator, which, although claimed to have an accuracy of 90% in this paper, lacks provided details on its training and the volume of data used. There is a question regarding whether this 300M classifier model can generalize to a broader range of characters.

3.The paper lacks citation of the research[1] that is similar in core idea to this study, which also focuses on enhancing role-playing effectiveness by limiting the knowledge boundaries of characters to ensure their responses are confined to what they are known.

[1]Lu K, Yu B, Zhou C, et al. Large language models are superpositions of all characters: Attaining arbitrary role-play via self-alignment[J]. arXiv preprint arXiv:2401.12474, 2024.
questions: questions: 1. Why is Regularization APC considered necessary, and how should it be chosen in relation to the APC score when in use?

2. APC-Based DPO is one of the primary contributions of this study; however, Appendix D mentions that only 100 preference data points were used for training. Could this small amount of data lead to significant model overfitting?

3. Table 1, 4, and 5 show a huge difference in APC scores between simple characters and famous characters (6 vs 400). What could be the possible reasons for this discrepancy, and what does the absolute magnitude of the APC score signify?
limitations: limitations: The author elaborates the limilation of this study.

Comments: title: Response to Reviewer 5oeH
Response to Reviewer Pxme
comment: Thanks for your response, I would keep my positive rating.
Thank you for the clarification on the question! For scenarios that the simulated character might shift the persona statement pool, we will discuss the application of APC in both quantification and optimization.

**Quantification: ** Our APC scheme can be directly applied to evaluate persona-shifting characters because of its plug-and-play property. We can use the trained discriminators (relevance and NLI) with a new pool of persona statements after increase, decrease, or modification. As our APC scoring system estimates the number of constraints satisfied, it can be aggregated among different persona statement pools. So even in a dialogue that the character shifts its persona in the middle, we can evaluate the faithfulness in the full dialogue by evaluating with different statement pools before and after the shifting.

**Optimization: ** The current formulation of the DPO optimization might not be directly applied to handle shifting persona statement pool since the reward model is based on a static preference model. Fortunately, we can adapt the reward model to a dynamic one by taking a group of dynamic persona statements in the input side $APC(r, S_{dynamic}|q, S_{dynamic}, S_{static})$ (need to have a designed area to input these shifting persona statements so the optimization will encourage the AI character to follow the changeable input $S_{dynamic}$). For instance, $S_{dynamic}$ can be “Alice is happy”, “Alice is sad”, … The character LLM also needs to be adapted to have a input area for $S_{dynamic}$. One possible limitation of this scheme is when we want to maintain a large pool of dynamic statements, it will challenge the LLM’s long-context ability as shown in the LCM part in the current paper.

We hope the explanation above addresses your concern, thank you!
Thank you for your response and clarifications. I have updated my score accordingly and vote for acceptance. 

Regarding the last point about people shifting their opinion - here I was referring to how people that you are trying to simulate can change their opinions. Is this also able to be adapted into the constraint satisfaction framework?
Thank you for considering the suggestions and explaining in detail about the new additions in detail, after considering the new updates, I am increasing my score to 7
Thanks for your question! The main difficulty to scale-up the experiments is the cost to fine-tune character LLMs. Take experience uploading (EU) as an example, for each character, we need to train it with separate synthesized experiences. 

Also, the lack of well-curated dataset is another problem, we follow the influential character-LLM work [1] to take the involved characters (9 characters) with extra original characters into our experiments. This issue can be addressed with the emergence of new well-curated datasets.

[1] Character-LLM: A Trainable Agent for Role-Playing
I'm curious why it's difficult to scale up the experiment. What is the most cost part?  or just there is no suitable dataset.