Title: Codifying Character Logic in Role-Playing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Role-playing

Large Language Models

Knowledge Grounding
Abstract: This paper introduces Codified Profiles for role-playing, a novel approach that represents character logic as structured, executable functions for behavioral decision-making. 
Converted by large language model (LLM) from textual profiles, each codified profile defines a set of functions parse_by_scene(scene) that output multiple logic-grounded assertions according to scene, using both explicit control structures (e.g., if-then-else) and flexible check_condition(scene, question) functions where each question is a semantically meaningful prompt about the scene (e.g., "Is the character in danger?") discriminated by the role-playing LLM as true, false, or unknown.
This explicit representation offers three key advantages over traditional prompt-based textual profiles, which append character descriptions directly into text prompts:
(1) Persistence, by enforcing complete and consistent execution of character logic, rather than relying on the model's implicit reasoning;  
(2) Updatability, through systematic inspection and revision of behavioral logic, which is difficult to track or debug in prompt-only approaches;  
(3) Controllable Randomness, by supporting stochastic behavior directly within the logic, enabling fine-grained variability that prompting alone struggles to achieve.  
To validate these advantages, we introduce a new benchmark constructed from 83 characters and 5,141 scenes curated from Fandom, using natural language inference (NLI)-based scoring to compare character responses against ground-truths.  
Our experiments demonstrate the significant benefits of codified profiles in improving persistence, updatability, and behavioral diversity. 
Notably, by offloading a significant portion of reasoning to preprocessing, codified profiles enable even 1B-parameter models to perform high-quality role-playing, providing an efficient, lightweight foundation for local deployment of role-play agents.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method, Codified Profiles, for modeling character logic in LLM-driven role-playing scenarios. The approach converts textual character profiles into executable functions that enable persistence, updatability, and controllable randomness.\nA new benchmark, curated from 83 characters and over 5,000 scenes sourced from Fandom, is used for evaluation. Experiments show codified profiles outperform traditional textual approaches in behavioral consistency, adaptability, and diversity, especially when used with smaller LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. Conceptual Novelty: Introducing executable code as character profiles is a significant and original contribution, effectively combining symbolic logic with LLM capabilities.\n2. Controllable Randomness: Explicit modeling of randomness via code enhances reproducibility and behavioral fidelity.\n3. Clear Advantages Over Baselines: Demonstrates improvements in persistence, updatability, and behavioral diversity compared to previous methods. The use of a comprehensive benchmark and comparisons across multiple conditions (baseline, evolving, stochastic) strengthen the claims.\n\n**Weaknesses:**\n1. Dependence on Proprietary LLMs: GPT-4.1 is used for codification and NLI scoring, raising concerns about reproducibility and transparency. Can the codification pipeline be fully automated using only open-source models, and if so, what trade-offs would arise?\n2. Generalization to Open-ended Scenarios: While effective on benchmarked or scripted contexts, the method's ability to generalize to open-ended, unscripted, or human-character interactive settings remains underexplored. As mentioned in Lines 117-121, the rigidity of code may hinder generalization, which is important for role-playing LLMs.\n\nQUESTIONS:\n1. In line 248, the reference to Table 6 appears to be incorrect; it should likely be Table 2.\n2. Several tables (e.g., Figures 5, 6, and 7) would benefit from clearer labeling of both rows and columns to clarify their meanings. Adding explicit axis labels or captions would improve readability.\n3. Similar to Weaknesses 2, how well do codified profiles generalize to unscripted or human-character interactive scenarios beyond the Fandom benchmark?\n4. Why is LLM-based NLI used as the main scoring method? Prior works [1-3] have introduced various evaluation strategies—some of which offer finer-grained or more interpretable feedback. What advantages justify choosing NLI over those methods?\n\n[1] Character-LLM: A Trainable Agent for Role-Playing, EMNLP 2023.\\\n[2] Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment, ACL 2024.\\\n[3] MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents, ICLR 2025.\n\nLIMITATIONS:\n1. Abstraction vs. Fidelity Trade-off: Codification simplifies logic but may lose narrative nuances present in free-form text.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents codified profiles as a way to use LLMs as in role-playing scenarios with more logic persistence, updatability, and controllable randomness. Codified profiles represents character logic as structured, semi-executable functions for behavioral decision-making. The paper also introduces a new role-playing benchmark and demonstrate the advantage of their approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Good idea, good execution.** The idea of using structured representations to maintain character logic in role-playing settings is interesting, and the results show that it helps produce character responses that are logically consistent with ground-truth responses. The authors run appropriate experiments, evaluation, and analysis. The evaluation dataset is fairly extensive, and seems less prone to data contamination since it is fan fiction. The ablation plots are well-done and show the value of different elements of codified profiles: granularity of segmentation, model size, chain-of-thought length, and structure. \n\nIf this paper is published or otherwise publicly shared, I would share it with other researchers interested in structured representations of human logic. \n\n**NLI vs. human preferences.** From the main body of text, it is not immediately clear that NLI against a ground-truth response is the correct measure for free-form LLM response generation. The authors include validation for using NLI as a quality metric and the human preference results in the appendix. I think that especially given the soft-correctness nature of this setting, these should be moved into the main body of text.\n\nQUESTIONS:\n* Why are eqns (1) and (2) $r = LLM(s | ..)$? Do you mean that the response is sampled from the conditional distribution? $r \\sim LLM(\\cdot | s, ...)$\n* What is the y-axis in Figure 5? Is it still NLI scores? If so, how does different profile representations having similar relative performance in different seasons illustrate \"how evolving profiles maintain alignment with narrative shifts\" (L280)? Wouldn't that require both (a) a ground-truth line plotted on the same graph showing the same relative performance across seasons, and (b) some assurance that relative NLI differences measure narrative shifts?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Codified Profiles, which aim to represent character logic in role-playing scenarios using structured, executable functions. The primary motivation is to enhance consistency, controllability, and efficiency compared to traditional prompt-based approaches. These profiles convert textual character descriptions into executable functions that include explicit control structures (like if-then-else) and conditional checks handled via helper functions. The approach provides persistence, facilitates systematic updates, and supports controlled randomness in character behaviors. To validate the approach, a new benchmark (Fandom Benchmark) sourced from manga, novels, and television series was developed. Experimental results demonstrate that Codified Profiles improve consistency, adaptability, and diversity of behaviors. Furthermore, it was shown that even small language models like 1B models could effectively use these profiles, making role-playing feasible for lightweight deployments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The concept of transforming textual character logic into structured, executable code is innovative and effectively addresses current challenges in consistency and interpretability in LLM-driven role-play.\n- Experiments clearly validate key claims about persistence, controllable randomness, and ease of updating character profiles. The new Fandom Benchmark can be useful for future research in the field of LLM for role-playing.\n\n\n\nWeaknesses\n- The method relies on LLMs (e.g., GPT-4.1) to convert text to code, which may introduce errors in complex logic. \n- NLI scoring (entailment/neutral/contradiction) lacks granularity for nuanced role-playing quality (e.g., emotional depth).\n\nQUESTIONS:\nPlease see weaknesses above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework called \"Codified Profiles\" for LLM-based character role-playing. Instead of appending plain text character descriptions to prompts, this method compiles them into structured, executable Python functions. These functions use explicit control logic (e.g., if-then-else) and a flexible helper function (check_condition) that queries an LLM to interpret scene-specific conditions. The authors argue that this approach offers three main advantages over traditional methods: Persistence (consistent application of character logic), Updatability (easier to debug and evolve character logic over a storyline), and Controllable Randomness (precise simulation of stochastic behaviors). To validate these claims, the paper introduces a new comprehensive benchmark, \"Fandom Benchmark,\" curated from narrative websites. Experiments demonstrate that Codified Profiles improve role-playing consistency, are more adaptable, and allow for greater behavioral diversity, showing particular strength when implemented with smaller language models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.The paper addresses a well-defined and important problem in LLM-based role-playing: the lack of persistence, controllability, and transparency in prompt-based methods. The proposed solution of \"codifying\" character logic into executable functions is highly original and offers a new paradigm for character representation that synergizes symbolic logic with neural models. This is a significant departure from mainstream RAG or fine-tuning approaches.\n\n2.The experimental validation is a major strength of this paper. The authors systematically design experiments to verify each of their three core claims:\n- Persistence: The superiority of codified profiles over textual profiles and RAG baselines in Table 2 and Figure 4 clearly supports this claim.\n- Updatability: The \"Evolving Profile\" experiment (Section 5.3, Figure 5) provides strong evidence that codified logic is more effectively and efficiently updated along a storyline compared to unstructured text.\n- Controllable Randomness: The experiments in Figure 6 convincingly show that codified profiles can simulate specified probabilities accurately, a known weakness of LLM sampling, and that this controlled stochasticity leads to better coverage of plausible actions (Best@K results).\n\n3.By curating 83 characters and over 5,000 scenes from rich, real-world narrative sources, the authors provide a high-quality, behavior-centric benchmark that will be valuable for future research in this area. \n\n4.The paper is exceptionally well-written and easy to follow. The methodology is explained clearly with illustrative examples. The appendices are thorough, providing all necessary prompts, benchmark details, and extended case studies, which strongly supports the reproducibility of the work.\n\n----\n\nWeaknesses:\n\n1.The framework's effectiveness hinges on the ability of a powerful LLM (GPT-4.1 in this case) to accurately translate nuanced text into correct logical code. The performance could degrade substantially if a less capable, open-source model were used for this codification step. While Appendix G explores this, the core experiments rely on a state-of-the-art closed model, which could be a practical barrier.\n\n2.The proposed evolving mechanism is driven by NLI-based comparisons to a ground-truth reference action from the original story. This is a sound approach for evaluation but its applicability to open-ended, interactive role-playing (where no ground-truth exists) is unclear. The paper does not fully explore how character evolution would be triggered or guided in such scenarios.\n\n3.The process of abstraction from rich, descriptive text to structured code may lead to a loss of nuance. A character's personality might contain subtle elements that are not easily captured by discrete if-then logic. The authors acknowledge this in Appendix A and show that combining codified and textual profiles yields slight gains (Table 2), which implicitly supports this concern. This trade-off could be discussed more prominently in the main paper.\n\nQUESTIONS:\n1.How robust is the codification process to the style of the source profile? For example, a Fandom wiki article is often structured and factual. Would the codification LLM struggle more with a profile written in a more literary or implicit style, potentially leading to less accurate or incomplete logic?\n\n2.The check_condition function is a cornerstone of the framework's flexibility. It can return True, False, or None (Unknown). How does the system handle an \"Unknown\" response? Could this lead to situations where no logical branch is triggered, resulting in a generic or null response from the character?\n\n3.The evolving mechanism is supervised by ground-truth actions from the storyline. How do you envision this mechanism adapting to an interactive, open-ended role-playing scenario where no such ground truth exists? Could the framework be integrated with self-reflection mechanisms to enable unsupervised character evolution based on interaction history?\n\n4.The examples in the paper show clear but relatively simple logical structures. In your experiments with complex characters from works like \"A Game of Thrones\", did the codification LLM generate more complex code involving, for instance, state variables that persist across multiple scenes or nested loops? If so, how does the framework manage this statefulness?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}