TIMECHARA: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models (2024)

Abstract: While Large Language Models (LLMs) can serve as agents to simulate human behaviors (i.e., role-playing agents), we emphasize the importance of point-in-time role-playing. This situates characters at specific moments in the narrative progression for three main reasons: (i) enhancing users’ narrative immersion, (ii) avoiding spoilers, and (iii) fostering engagement in fandom role-playing. To accurately represent characters at specific time points, agents must avoid character hallucination, where they display knowledge that contradicts their characters’ identities and historical timelines. We introduce TIMECHARA, a new benchmark designed to evaluate point-in-time character hallucination in role-playing LLMs. Comprising 10,895 instances generated through an automated pipeline, this benchmark reveals significant hallucination issues in current state-ofthe-art LLMs (e.g., GPT-4o). To counter this challenge, we propose NARRATIVE-EXPERTS, a method that decomposes the reasoning steps and utilizes narrative experts to reduce point-intime character hallucinations effectively. Still, our findings with TIMECHARA highlight the ongoing challenges of point-in-time character hallucination, calling for further study.1

heading: 1 Introduction
text: The recent progress in large language models (LLMs) has opened up a new phase of generative agents (Park et al., 2023; Xi et al., 2023; Wang et al., 2024a), where LLMs simulate human-like behaviors, memories, and cognitive processes. A particularly promising area is the development of role-playing LLM agents (Shanahan et al., 2023; Kong et al., 2023; Li et al., 2023b), which simulate the personas of either real individuals or fictional characters and engage with users to provide a more vivid experience. A variety of applications, including Character AI, GPTs, Talkie, Replika, AI
1https://ahnjaewoo.github.io/timechara.
[Harry Potter’s Timeline]
Dungeon, SillyTavern, showcase the growing popularity of these role-playing LLM agents. However, most current approaches of role-playing agents (Han et al., 2022; Li et al., 2023a; Zhou et al., 2023) only simulate characters who are omniscient in timeline; for example, a Harry Potter character who is aware of all events leading up to the end of their respective series.
We suggest the importance of situating characters at a particular moment in the narrative progression. We coin this as point-in-time role-playing, en-
3291
compassing three key rationales: narrative immersion, avoidance of spoilers, and engagement in fandom role-playing. Firstly, while a fully-informed character can interact with users drawing from their entire history, a character in the middle of the story inspires narrative immersion (Ryan, 2003, 2008). It sparks the user’s curiosity about forthcoming events and deepens their emotional bond with the character, who remains unaware of their eventual fate. Secondly, this approach can avoid spoilers. Consider a media franchise such as Harry Potter, where all books are published, but upcoming adaptations (e.g., “Harry Potter TV series”2) are awaited. Users who wish to avoid spoilers before watching the new TV series would prefer interacting with a character from a midpoint in their story, thereby avoiding knowledge of future events. Thirdly, this approach can support recently popular fandom roleplaying3, a blend of fan fiction and traditional roleplaying games. Individuals adopt the personas of their favorite characters at specific points in their stories and craft new narratives or engage with fellow enthusiasts in this creative endeavor.
To accurately represent characters at specific time points, the agents should recognize the character’s knowledge boundary. This includes their unawareness of future events, their ability to recall past events precisely, and their understanding of the individuals involved in those past events. However, current LLM-based role-playing agents are prone to character hallucination (Shao et al., 2023), displaying knowledge that contradicts their character’s identity and historical context (e.g., Julius Caesar talking about his favorite movie). Despite the seriousness, the problem has not been investigated in terms of maintaining character consistency, especially in relation to their historical timelines, and robustness to such hallucinations.
We introduce a new point-in-Time Character hallucination benchmark, TIMECHARA, to rigorously assess role-playing LLMs at specific time points, thereby evaluating the agents’ spatiotemporal self-consistency and their ability to avoid character hallucination. Figure 1 exemplifies the point-in-time character hallucination, where Harry Potter, in his fifth year at Hogwarts, inappropriately mentions a future fact about his wife, Ginny Weasley. We select 14 fictional characters from four popular novel series and develop a pipeline to
2https://www.theguardian.com/film/2023/apr/12/ harry-potter-tv-series-hbo-max-jk-rowling.
3https://fanlore.org/wiki/Fandom_RPG.
generate interview questions tailored to each character at a specific point in their story, along with spatiotemporal labels to determine the spatiotemporal consistency of their responses. Table 1 outlines comparison with existing related benchmarks.
Our empirical experiments reveal a significant issue of point-in-time character hallucination in state-of-the-art LLMs, including GPT-4o (OpenAI, 2024) and GPT-4 (OpenAI, 2023). This suggests that despite LLMs memorizing extensive knowledge from books (Chang et al., 2023), they still struggle with maintaining spatiotemporal consistency during role-playing scenarios. To mitigate this, we propose a decomposed reasoning method, NARRATIVE-EXPERTS, which partitions reasoning tasks among narrative experts specialized in temporal (i.e., identifying between past and future events) and spatial domains (i.e., discerning whether a character was present or absent in specific past events). Experiments show that NARRATIVE-EXPERTS significantly reduces point-in-time character hallucination and enhances spatiotemporal consistency. Still, our TIMECHARA underscores the ongoing challenge of point-in-time character hallucination and highlights the potential for future improvements.
Our main contributions are as follows:
1. We introduce TIMECHARA, a novel benchmark for evaluating character hallucination in point-in-time role-playing agents. We also develop an automated pipeline to construct the dataset, comprising 10,895 instances in total.
2. Through TIMECHARA, we identify significant hallucination issues within state-of-theart role-playing LLMs including GPT-4o.
3. We propose NARRATIVE-EXPERTS, a simple but effective method to mitigate point-in-time hallucination by decomposing reasoning with each step led by the narrative expert.

heading: 2 Related Work
text: We include a more thorough literature review in Appendix A. In this section, we only discuss the most relevant works.
Role-playing LLM agents. Prior research on conversational AI has focused on developing dialogue agents with self-consistent personas (Zhang et al., 2018; Kim et al., 2020; Ahn et al., 2023). Furthermore, LLMs are increasingly being used to simulate human behavior (Park et al., 2023). Many
of these efforts involve using LLMs to role-play specific characters, such as Harry Potter, Socrates, and others (Shanahan et al., 2023; Wang et al., 2023a,b). In this context, Shao et al. (2023) introduced character hallucination, a scenario where a role-playing agent inappropriately exhibits knowledge that is inconsistent with the character’s identity and historical background. On the other hand, Chen et al. (2023) proposed a point-in-time “Harry Potter” role-playing dialogue dataset. They focused on assessing whether a role-playing agent responds naturally to the character within a specific point in a storyline. Rather than directly stress-testing role-playing LLMs at specific time points by asking confusing questions, their relevance metrics are designed to gauge overall character alignment given natural scene and dialogue context. However, existing studies on role-playing agents have not extensively examined how well these agents are robust to point-in-time character hallucination. We aim to stress-test role-playing LLMs at specific time points by assessing their spatiotemporal self-consistency and robustness against pointin-time character hallucination, as detailed in Table 1. Additionally, we compare concurrent work to TIMECHARA in Table 6 in Appendix A.
LLM’s temporal reasoning capability. Understanding the concept of time is crucial for LLMs, as the information they acquire is often timesensitive (Chen et al., 2021; Zhang and Choi, 2021; Dhingra et al., 2022; Chu et al., 2023). To assess LLMs’ temporal reasoning capabilities, several studies have set benchmarks. Jang et al. (2022) examined how well LLMs adapt to frequentlyupdated knowledge corpus. Feng et al. (2023) focused on whether LLMs can interpret the impact of subtle contextual changes on relevant temporal relationships. Tan et al. (2023b) developed TimelineQA, a dataset for querying the lifelogs of imaginary people. While these benchmarks evaluate the temporal reasoning capabilities of LLMs, we extend them to point-in-time role-playing scenarios by evaluating whether role-playing LLMs maintain the character’s spatiotemporal consistency.

heading: 3 The TIMECHARA Benchmark
text: To create TIMECHARA, we select four renowned novel series: Harry Potter, The Lord of the Rings, Twilight, and The Hunger Games. This choice is based on two main reasons: (i) the ease of gathering raw text content (i.e., transcripts) and personality
information for each character, useful for dataset construction, and (ii) the fact that recent state-ofthe-art LLMs store knowledge of these series well in their parametric memories (Chang et al., 2023), facilitating tests for point-in-time character hallucination. Note that TIMECHARA is not exclusively limited to these series; it is easily extendable to other narratives, provided that raw text content and personality information for the characters can be obtained. Then, we identify 14 main characters across the four novel series, detailed in Appendix F. We pinpoint a particular moment in each character’s timeline (e.g., Hermione on Christmas during her first year at Hogwarts) rather than assuming they are aware of all events up to the end of the series as in previous studies (Tan et al., 2023b; Wang et al., 2023b).
We organize our dataset in an interview format where an interviewer poses questions and the character responds. Specifically, we differentiate between fact-based and fake-based interviews.

heading: 3.1 Fact-Based Interview
text: To evaluate point-in-time character hallucination, we categorize the data into four types as follows:
The unawareness of the future (i.e., future type): The character at the chosen time point should not know about future events (e.g., “Who is your wife?” to first-year Harry).
The memorization of the past: The character should accurately recall past events. Since episodic events occur at specific locations or scenes, the questions are further categorized as follows. The awareness of the absence (i.e., past-absence type): The character recognizes they are not in an event (e.g., “Did you see the moment when Harry received the Invisibility Cloak on Christmas?” to first-year Hermione on Christmas). The awareness of the past (i.e., past-presence type): The character acknowledges they are in an event (e.g., “Did you see the moment when Harry received the Invisibility Cloak on Christmas?” to first-year Ron on Christmas). The awareness of the past, irrelevant of participation (i.e., past-only type): Questions in this type focus on gauging the character’s overall knowledge of past events, including relationships between characters or the significance of magical items (e.g., “Who is Dobby?” to second-year Harry on Halloween). The term “only” suggests that these questions primarily assess the character’s understanding and memory of past information, not exclusively tied to their direct
experiences or observations.
Table 2 shows an example of a fact-based interview. TIMECHARA assesses point-in-time character hallucination using questions derived from
the same narrative. This contrasts with Shao et al. (2023), who (1) use questions that span different time or narratives and (2) do not support point-in-time role-playing (e.g., “Can you write
Python codes?” to Beethoven), as marked in Table 1. Hence, our interviews demand detailed narrative understanding, making hallucination detection more challenging.

heading: 3.2 Fake-Based Interview
text: In addition to the fact-based interview, which tests whether questions about real events are answered correctly, we introduce the fake-based interview. It is designed to evaluate if role-playing agents can identify and rectify the errors in interview questions by partially altering fact-based questions. Fake-based interviews are concentrated on pastonly type questions (e.g., “How did you become Slytherin?” to first-year Harry on September 1st: The correct answer is that he became Gryffindor). We exclude future type questions since correcting misinformation about unknown future events is not possible. Similarly, we exclude past-presence and past-absence type questions because verifying or refuting a character’s event participation in nonexistent past events is ambiguous.

heading: 3.3 Evaluation on TIMECHARA
text: Since it is not scalable to manually evaluate the roleplaying LLMs’ responses to interview questions, as done in Shao et al. (2023); Zheng et al. (2023), we adopt the LLM-as-judges approach to assess along two dimensions:
Spatiotemporal consistency for assessing pointin-time character hallucination: The model should accurately recall the character’s past experiences. This includes the character’s unawareness of future events and awareness of presence or absence in past events, as described in § 3.1. This metric is time-dependent; the model should only exhibit the knowledge that the character possesses up to the specific time point.
Personality consistency: The model should emulate the character’s personality, including their manner of thinking, speaking styles, tones, emotional responses, and reactions. This encompasses the character’s preferences, values, and convictions. This metric is time-independent; the response should consistently reflect the character’s enduring personal traits.
Step-by-step evaluation with spatiotemporal labels. Following Wei et al. (2022), we instruct the GPT-4 Turbo (gpt-4-1106-preview) model (OpenAI, 2023) to step-by-step score the performance in each dimension. For specific examples of prompts

heading: Past-presence

heading: GPT-4 Label GPT-3.5 Label GPT-4 No Label GPT-3.5 No Label
text: used in this process, refer to Appendix B. Unlike Shao et al. (2023), our evaluation of point-in-time character hallucination (or spatiotemporal consistency) provides judges with precise spatiotemporal labels, which encompass the character’s experiences with people, events, and objects. As shown in Figure 2, the labels enable a much more accurate evaluation of response consistency with the character’s known history (e.g., During his first year on Christmas, Harry can respond based on the moment but should not wrongly recall it:{moment description}. Please refer to Table 10 in Appendix C for details of this evaluation of past memorization). The responses with contradiction or inconsistency regarding the spatiotemporal labels are scored as 0; otherwise, those in alignment are rated as 1. We will describe the details of how to construct these spatiotemporal labels in § 3.4.
For evaluating personality consistency, we adopt a methodology similar to Shao et al. (2023) but enhance it by sourcing more detailed personality traits from the Fandom page4. We then rate these traits on a 1-7 Likert scale to measure how closely a response aligns with a character’s personality, where 1 signifies a weak reflection, and 7 indicates an exact match.

heading: 3.4 Dataset Construction
text: To create the TIMECHARA benchmark, we propose a new automated pipeline that easily scales up the
4https://www.fandom.com/.
dataset while reducing the need for manual human annotation, as depicted in Figure 3.
Extract scenes, event summaries, and participants lists from books. The first step is to extract specific scenes from literary works using GPT-4 Turbo. We instruct it to extract N distinct scenes containing multi-turn dialogues among characters, as detailed in Appendix D.1 (see Table 13). For every extracted scene, we instruct GPT-4 to generate (1) a concise, single-sentence summary (i.e., event summary) of scene information and (2) a list of the participants involved in that scene, as shown in Appendix D.1 (see Table 14).
Generate questions from event summary. Initially, we generate fact-based questions based on two different methods. We begin by creating factbased structured questions about characters’ involvement in events by combining question templates with the event summary. To this end, we curate 18 different question templates like "Tell me your feelings when {event summary}." Appendix D.2 shows all 18 question templates. Following this, we create fact-based free-form questions that assess a character’s understanding of the event, regardless of their direct participation, as detailed in Appendix D.2. Subsequently, we proceed to formulate fake-based questions, employing a methodology similar to the creation of fact-based free-form questions, with further details available in Appendix D.2.
Assign spatiotemporal labels to each character. Given a question for a specific scene and event, the goal is to create a combination of {scene, event summary, question, character with their time point}. By choosing the character and its time point, the data type is automatically classified into one of
four types: future, past-absence, past-presence, past-only. These data types serve as spatiotemporal labels for the {scene, event summary, question, character with their time point} combination. Appendix D.3 details how to select the character and assign its time point for each type.
Add detailed descriptions to the spatiotemporal labels. Based on the {scene, event summary, question, character with their time point} combination and the predefined data type (i.e., spatiotemporal label), we add detailed descriptions to the spatiotemporal labels to serve as a basis for evaluating role-playing agent responses. Refer to Appendix D.4 for details.
Generate gold responses and manually filter data instances. To generate a gold response for each data instance, we prompt GPT-4 with the com-
bination of {question, character with their time point, spatiotemporal label}. At last, we manually filter by the authors, whose criteria and results are shown in Appendix D.5.

heading: 3.5 Dataset Analyses
text: Statistics. The total number of event summaries is 1,643: 914 for Harry Potter, 261 for The Lord of the Rings, 245 for Twilight, and 223 for The Hunger Games. As a result, the dataset contains 10,895 instances, and Table 3 provides detailed statistics. The average lengths of (questions, gold responses, spatiotemporal labels) are (29.2, 117.6, 543.2) words, respectively.
In addition, we manually reviewed the test datasets from three different benchmarks (i.e., HPD, Character-LLM, and TIMECHARA) and classified them based on the spatiotemporal label of the given question or previous utterances in the case of multi-turn dialogue, without considering the agent’s response for simplicity, as detailed in Table 4. While TIMECHARA consists of data instances evenly distributed over spatiotemporal labels, examples in HPD and Character-LLM are mainly classified as Past-only (Fact) or None types. This result demonstrates that TIMECHARA focuses on stress-testing the spatiotemporal consistency of role-playing LLMs, while the others focus on assessing the fact-based question-answering task or plain conversation between characters.
Furthermore, we provide the lexical diversity of free-form questions and compare it to the structured questions in Appendix E.1. Finally, we present a detailed human evaluation process to ensure the quality of TIMECHARA, as described in Appendix E.2.

heading: 4 Decomposed Reasoning
text: We find that existing LLMs struggle with spatiotemporal consistency as in Table 5, despite their extensive knowledge from books (Chang et al., 2023). To overcome this issue, we propose a reasoning method named NARRATIVE-EXPERTS, which decomposes reasoning steps into specialized tasks, employing narrative experts on either temporal (i.e., distinguishing past from future events) or spatial (i.e., identifying characters’ presence in past events) aspects while utilizing the same backbone LLM.
Temporal Expert: This expert pinpoints the scene’s book and chapter from a question, assigning a future or past label. If deemed future, it bypasses the Spatial Expert and advises the roleplaying agent with a specific hint (i.e., “Note that the period of the question is in the future relative to {character}’s time point. Therefore, you should not answer the question or mention any facts that occurred after {character}’s time point.”).
Spatial Expert: It assesses whether a character is involved in the scene, indicating a past-absent label if applicable. A tailored hint is then provided to the role-playing agent if the scene is past-absent (i.e., “Note that {character} had not participated in the scene described in the question. Therefore, you should not imply that {character} was present in the scene.”).
Finally, the role-playing LLM incorporates hints from these experts into the prompt and generates a response. Appendix G.5.1 offers details of the algorithm and the prompts designed for experts.

heading: 5 Experiments on TIMECHARA

heading: 5.1 Dataset Sampling for Evaluation
text: Due to the high computational cost of employing GPT-4 judges, fully evaluating the 11K instance dataset is challenging. Instead, we randomly sample 600 data instances to assess the point-intime character hallucination of role-playing agents. First, we sample 300 instances with fact-based structured questions, evenly distributed across three data types: future (100 instances), pastpresence (100 instances), and past-absence (100 instances). Then, we pick 200 instances with factbased free-form questions, with an equal split of 100 instances each from future and past-only types. Lastly, we choose 100 instances with fake-based free-form questions, all from the past-only type.
In addition, we provide experimental results for the entire 11K dataset in Appendix G.6.

heading: 5.2 Baseline Methods
text: We focus on inference-based agents as opposed to training-based agents (Shao et al., 2023), due to the impracticality of training agents to simulate characters across diverse time points; notably, our dataset includes 219 time points. We utilize four different state-of-the-art LLMs as a backbone model for role-playing agents to respond to our dataset: GPT-4o (i.e., gpt-4o-2024-05-13), GPT-4 (i.e., gpt-4-1106-preview), GPT-3.5 (i.e., gpt-3.5-turbo-1106), and Mistral 7B Instruct (i.e., mistral-7b-instruct-v0.2) (Jiang et al., 2023). To test their various reasoning capabilities, we employ several baselines as follows.
Zero-shot prompt. This is to directly prompt an agent to generate a response based on the system instruction and a question as follows:
Zero-Shot Prompt Template
System Instruction: I want you to act like {character} from {author}’s {series_name} novel series. I want you to respond and answer like {character}, using the tone, manner, and vocabulary {character} would use. Assume that you are on {time_point} in {book_name} and interviewing with the interviewer. You should not answer the question and mention any fact that is future to the period. If he (or she) was not present at the location where the question was raised, he (or she) is likely unaware of the information or knowledge related to that question.
User Prompt: {question}
Note that we instruct the agent to be unaware of future events and to acknowledge the absence when responding to past-absence type questions.
Zero-shot-CoT prompt. (Kojima et al., 2022). This method exploits a zero-shot prompt by adding the phrase “Let’s think step by step” at the end of the question. This addition aims to improve the step-by-step reasoning capability of LLMs.
Few-shot prompt (in-context learning). This approach provides LLMs with four instances (4- shot), with details on how examples were selected available in Appendix G.2.
Iterative self-correction. Recent studies (Pan et al., 2023; Shinn et al., 2023) found that LLMs have the capability for self-correction, iteratively refining their initial responses based on the given criteria. Among various methods, we choose the selfrefine (Madaan et al., 2023), since it is adaptable to dialogue domains with multiple evaluation criteria. Further details can be found in Appendix G.3.
Retrieval-augmented generation (RAG). In some prior research, retrieval-augmented generation (Lewis et al., 2020) can mitigate hallucinations (Shuster et al., 2021). We develop a retrieval module that employs OpenAI’s embedding (i.e., text-embedding-ada-002) to provide contexts to LLMs. In addition, we add a variant named RAGcutoff, which is designed to limit its retrieval exclusively to the events prior to a defined character period. Thanks to this constraint, agents can avoid access to future contexts. Further details are available in Appendix G.4.

heading: 5.3 Decomposed Reasoning with RAG
text: Beyond NARRATIVE-EXPERTS, we also explore NARRATIVE-EXPERTS-RAG-CUTOFF, which integrates NARRATIVE-EXPERTS with the RAG-cutoff method. We provide a complete algorithm and prompts used for experts in Appendix G.5.2.

heading: 5.4 Experimental Results
text: See Appendix G.1 for implementation details. Table 5 finds even GPT-4o and GPT-4, state-of-the-art LLMs, still struggle with point-in-time character hallucinations.
Future type. All baselines exhibit confusion with future type questions with accuracies at 51% or below. It highlights a prevailing issue of role-playing agents that inadvertently disclose future events. The naive RAG scores the lowest among baselines, showing that indiscriminately providing contexts harms the performance. Our NARRATIVE-EXPERTS and NARRATIVEEXPERTS-RAG-CUTOFF significantly enhance performance, thanks to the temporal expert.
Past-absence and past-only types. Both naive RAG and RAG-cutoff can potentially mitigate hallucinations for these question types by leveraging context from their retrieval modules. However, their performance still lags behind that observed in past-presence questions, with gaps of 10% points and 13% points in past-absence types, and 19% points and 20% points in past-only types, respectively. Conversely, our methods enhance outcomes in both past-absence and past-only types, thanks to the support of both temporal and spatial experts.
Past-presence type instances: All baselines, except for Mistral, perform admirably, showcasing the role-playing LLMs’ proficiency in memorizing narratives from novel series. Our methods slightly lag in this type due to narrative experts’ occasional
mispredictions, yet this shortfall is minor compared to significant enhancement in the other three types.
Personality consistency: All methods generally maintain a consistent character portrayal, scoring above 5.6 in personality consistency. However, our methods receive lower scores from the GPT-4 judge due to their tendency to respond with unawareness regarding future events or character absences, which sometimes falls short of fully conveying the expected character’s personality. In contrast, the GPT-4 judge appears to favor responses from roleplaying agents that indiscriminately disclose information, regardless of its relevance to the character’s knowledge boundary at a specific time point.
AlignScore evaluation: In addition, we utilize AlignScore (Zha et al., 2023), a metric based on a post-trained RoBERTa-large model, to assess the spatiotemporal consistency of the role-playing agent without relying on GPT-4 judges. Refer to Appendix G.6 for details of the AlignScore. The results in Table 5 show that AlignScore is in agreement with evaluations from GPT-4 judges, and our methods achieve the highest AlignScores across the three different backbone LLMs.
Furthermore, we provide further analyses be-
yond the main experiments, including human evaluation results that closely align with those of the LLM judges, as detailed in Appendix H.

heading: 6 Conclusion
text: We highlighted the importance of point-in-time role-playing agents for enhancing narrative engagement, preventing spoilers, and facilitating fandom role-play activities. To maintain a character’s spatiotemporal consistency and avoid hallucinations, we introduced the TIMECHARA benchmark and developed an automated pipeline, resulting in 10,895 instances. Using TIMECHARA, we identified significant hallucination issues in state-of-the-art roleplaying LLMs. To address these, we proposed NARRATIVE-EXPERTS, an effective method to reduce character hallucinations by breaking down the reasoning process and guiding it with narrative experts. Despite these efforts, our findings indicate ongoing challenges with point-in-time character hallucinations, suggesting the need for further improvements.

heading: Limitations
text: Despite the advancements presented in this study, there are some limitations as follows. (1) Sourced only from English books: Since TIMECHARA consists primarily of texts written in English and sourced from English-speaking countries, it may reflect cultural biases inherent to these regions. One solution is to incorporate multilingual and multicultural books, as new data can be added to TIMECHARA automatically. (2) High costs of GPT4 judges: The financial expenses of extensive GPT4 evaluations can be prohibitive, restricting the feasibility of conducting large-scale assessments. An alternative would be using open-source LLMs for evaluation, such as Kim et al. (2024). (3) Latency and cost issues with NARRATIVE-EXPERTS: The narrative expert requires generating multiple hints and responses per question. This introduces increased latency and computational costs. Future research on efficiently generating responses while reducing point-in-time character hallucinations will be anticipated.

heading: Ethics Statement
text: To mitigate any potential issue arising from the use of the four novel series, we address concerns about copyright issues as follows:
1. Source attribution: The dataset utilizes raw text from each novel series, and we acknowledge the copyrights held by the authors and publishers.
2. Fair use justification: We believe our use of the copyrighted text qualifies as “fair use” under U.S. law, whose criteria include:
• Purpose of use: The dataset is used exclusively for non-commercial, educational, and research purposes.
• Nature of the copyrighted work: The work is used in a research context to evaluate point-in-time character hallucination of role-playing LLMs, an inherently academic pursuit.
• Lack of market harm: Our dataset does not substitute for the original works nor harm their market.
3. Content of dataset: TIMECHARA comprises only a fraction of the content necessary for dataset construction, indicating that our
dataset includes approximately 40% of the original text from the sources.
4. Accessibility and reproducibility: We will publish all dataset scripts and the dataset itself, restricting access to those who agree to use it only for research.
Besides, the generated dataset may inadvertently include harmful content intended to mislead characters. In adherence to the NLP ethics community’s guidelines on ‘toxic text’ (Gehman et al., 2020; Askell et al., 2021), We manually reviewed all 11K data instances and filtered out those containing provocative scenes. Specifically, we removed fewer than ten cases associated with severe trauma and explicit violence that could negatively impact users who will read questions and the agent’s responses.
By implementing these measures, we ensure that our research respects both the legal rights of the original content creators and the ethical standards of the research community.

heading: Acknowledgements
text: We thank Jamin Shin, Hyunwoo Kim, Euihyun Tae, and the anonymous reviewers for their valuable comments. This work was supported by SNU-NAVER Hyperscale AI Center, the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2019II191082, SW StarLab; No. RS-2022-II220156, Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D metaverse transformation; No. RS-2021II211343, Artificial Intelligence Graduate School Program (Seoul National University)), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2023R1A2C2005573). Gunhee Kim is the corresponding author.

heading: A Related Work (Full ver.)
text: Concurrent role-play benchmarks. We compare TIMECHARA to concurrent role-playing datasets or benchmarks (Shen et al., 2023; Xiao et al., 2023; Tao et al., 2024; Tu et al., 2024; Lu et al., 2024; Tang et al., 2024; Wang et al., 2024b; Chen et al., 2024) released in or after December 2023, as detailed in Table 6. Note that RoleEval (Shen et al., 2023) assesses whether LLMs possess specific role knowledge by having them answer multiple-choice questions, rather than evaluating role-playing LLM agents. While some concurrent benchmarks partially address the goals of TIMECHARA intended to do (e.g., evaluation of event participant reasoning in RoleEval (Shen et al., 2023), evaluation of fake event awareness in MORTISE (Tang et al., 2024), use of temporal labels for LLM judges in WikiRole (Lu et al., 2024)), TIMECHARA remains the most comprehensive benchmark for evaluating point-in-time character hallucination.
LLM hallucinations. Since LLMs are prone to hallucinations (Ji et al., 2023; Zhang et al., 2023), many studies have evaluated hallucinations in generated texts (Zha et al., 2023; Min et al., 2023; Mishra et al., 2024; Hong et al., 2024). To reduce hallucination, some works have focused on retrieval-augmented generation (Lewis et al., 2020; Shuster et al., 2021; Mialon et al., 2023), while others have incorporated honesty alignment (i.e., refusing to answer questions when LLMs lack knowledge) in text generation (Yang et al., 2023; Cheng et al., 2024).
Unlike prior works that address general hallucination issues, we specifically focus on pointin-time character hallucination, which includes unique spatiotemporal consistency problems such as future unawareness, absence awareness, and fake event awareness. Furthermore, we propose a new method, NARRATIVE-EXPERTS, to mitigate these hallucinations.
Neural theory-of-mind in narrative understanding. Neural theory-of-mind (ToM) examines whether LLMs possess the ability to understand the mental stages (e.g., thoughts, beliefs, and intentions) of others (Sap et al., 2022; Kim et al., 2023). In narrative understanding, ToM capabilities are essential for role-playing agents to comprehend both the narratives and the characters’ minds (Sang et al., 2022; Yu et al., 2022; Zhao et al., 2024).
Although it is well known that LLMs memorize extensive knowledge from books (Chang et al., 2023) and can precisely answer questions about narrative (See Table 28 in Appendix H.5), we found that they cannot maintain spatiotemporal consistency while acting as point-in-time role-playing agents, as shown in Table 5. The results demonstrate that LLMs are not yet capable of ToM in point-in-time role-playing scenarios.

heading: B Prompt Demonstration
text: Prompts for GPT-4 Turbo judges. We present the prompts used for evaluating the two dimensions as follows:
1. Spatiotemporal consistency: refer to Table 7.
2. Personality consistency: refer to Table 8.

heading: C Examples of TIMECHARA
text: We show examples of our dataset across four different data types: future, past-absence, past-presence, and past-only, each illustrated as follows:
1. future: An instance of future data is presented in Table 2, characterized by its fact-based structured question.
2. past-absence: Table 9 illustrates the pastabsence type with a similarly fact-based structured question.
3. past-presence: Table 10 shows an example of past-presence type data, whose question is also a fact-based structured question.
4. past-only: Demonstrations of past-only type data are provided in Table 11 for a fact-based free-form question and in Table 12 for a fakebased free-form question.

heading: D Details of Dataset Construction
text: D.1 Extract scenes, event summaries, and participant lists from book
For the Harry Potter series, we utilize the scene dataset from Chen et al. (2023), which comprises manually selected 1,037 unique scenes from the books. For the other three series, we extract 300 distinct scenes containing multi-turn dialogues among characters, as detailed in Table 13. Note that we include multi-turn dialogues among characters to
ensure that each scene features interaction among participants, with at least one character involved. In addition, Table 14 shows an example of a prompt for generating an event summary and participant lists.
D.2 Generate questions from event summary For fact-based structured questions, we use a total of 18 different question templates as follows.
1. Tell me your feelings when {event summary}.
2. Tell me your genuine feelings when {event summary}.
3. Describe your feelings when {event summary}.
4. Describe your honest feelings when {event summary}.
5. Can you describe your experience when {event summary}?
6. Can you describe your true experience when {event summary}?
7. Did you see the moment when {event summary}.
8. Did you truly see the moment when {event summary}?
9. What did you see as {event summary}?
10. What did you actually see as {event summary}?
11. What did you hear when {event summary}?
12. What did you precisely hear when {event summary}?

heading: Prompt for Spatiotemporal Consistency Evaluation
text: You will be given responses written by an AI assistant mimicking the character {agent_name}. Your task is to rate the performance of {agent_name} using the specific criterion by following the evaluation steps. Below is the data:
*** [Interactions] Interviewer: {question} {agent_name}: {response} *** [Fact] {spatiotemporal_label}
[Evaluation Criterion] Spatiotemporal Consistency (0 or 1): Is the response consistent with the character’s spatiotemporal knowledge?
[Evaluation Steps] 1. Read through the [Fact] and identify the knowledge scope of the character. 2. Read through the interactions and responses of the AI assistant to find the evidence of knowledge used in the response. 3. Compare the evidence to the [Fact]. Check if the response is consistent with the character’s knowledge scope. 4. If some knowledge contradicts or contains inconsistencies about the [Fact], given a 0 score. Otherwise, assign a 1 score. ***
First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then, print the score on its own line corresponding to the correct answer. At the end, repeat just the selected score again by itself on a new line.
Table 7: Prompt for GPT-4 Turbo judges to evaluate spatiotemporal consistency.

heading: Prompt for Personality Consistency Evaluation
text: You will be given responses written by an AI assistant mimicking the character {agent_name}. Your task is to rate the performance of {agent_name} using the specific criterion by following the evaluation steps. Below is the data:
*** [Interactions] Interviewer: {question} {agent_name}: {response} *** [Personality] {personality_label}
[Evaluation Criterion] Personality Consistency (1-7): Is the response consistent with the character’s personality?
[Evaluation Steps] 1. Read through the [Personality] and write the personalities, including preferences, values, and convictions of the real character. 2. Read through the interactions and identify the personalities, including preferences, values, and convictions of the AI assistant. 3. After having a clear understanding of the interactions, compare the response to the [Personality]. Look for any consistencies or inconsistencies. Do the responses reflect the character’s personalities, including preferences, values, and convictions? 4. Use the given scale from 1-7 to rate how well the response reflects the personalities, including preferences, values, and convictions of the character. 1 being not at all reflective of the character’s personalities, and 7 being perfectly reflective of the character’s personalities. ***
First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then, print the score on its own line corresponding to the correct answer. At the end, repeat just the selected score again by itself on a new line.
Table 8: Prompt for GPT-4 Turbo judges to evaluate personality consistency.
13. Were you at the moment when {event summary}?
14. Were you really at the moment when {event summary}?
15. Were you present as {event summary}?
16. Were you indeed present as {event summary}?
17. Is it true that you were at the moment when {event summary}?
18. Is it right that you were at the moment when {event summary}?
Note that the templates were randomly chosen to ask if a character experienced a specific event at a location. We found that using more than 18 templates didn’t enhance the diversity of expression. In addition, The inclusion of ‘feeling’ questions aimed to enrich the narrative by encouraging a vivid description of the character’s personal emotions during an event, providing depth to their direct experiences.
For fact-based free-form questions, we prompt GPT-4 Turbo to generate questions using the 5W1H approach (i.e., what, who, where, when, why, and

heading: Prompt for Scene and Speakers List Extraction
text: First, read chapter 1, part 1 of the book “The Lord of the Rings - The Fellowship of the Ring”. Then, extract 5 parts from the raw text with dialogues that can be considered as one scene. Each part should meet the following requirements. Start by analyzing the text that I gave you.
1. Each scene should be unique throughout the entire series: The Lord of The Ring. 2. Each scene shouldn’t be ambiguous, which means that the summary of each scene should be talking or related to a specific event, item, or person. 3. Scenes shouldn’t be everyday conversation such as the summary of the scene being: “Frodo Baggins talked to Sam about his breakfast”, which could be an everyday conversation. 4. Each scene should contain at least 5 dialogues. The extracted raw text should be between 15 to 35 sentences long to sufficiently form the scene and contain sufficient information about the scene.
how) based on the event summary, as shown in Table 15. For fake-based free-form question, we first generate a fake event summary based on a true event summary by utilizing GPT-4 Turbo, as outlined in Table 16. Note that we use six different strategies for generating fake event summaries: (1) changing the character, (2) changing the key object, (3) altering the location, (4) switching the action, (5) introducing a nonexistent character or object, and (6) changing the character’s knowledge. Subsequently, we generate a fake question by prompting GPT-4 Turbo to generate 5W1H question based on the fake event summary, as detailed in Table 17. The generation of all free-form questions was conducted in a single trial. The choice of 5W1H questions was random, with the distribution as follows: what—48.2%, who—19.0%, why—17.4%, how—9.8%, when—4.6%, and where—1.0%. The prevalence of ‘what’ questions reflects the foundational approach of our free-form question generation process, which draws upon event summaries that focus on entities (characters, objects, events) and their interrelationships. Specifically, the versatility of ‘what’ questions in covering a wide array of topics related to these entities (e.g., “What’s the meaning of the spell?”, “What event occurred?”, “What action was taken?”, “what do you think of him/her?”, etc) likely led to their predominance. D.3 Assign spatiotemporal labels for each character in their time point. We first identify distinct time points in each character’s timeline and annotate these moments with the relevant book and chapter for timeline comparison with the questions. We enlist all character time points in Appendix F. For constructing (1) future type data instances, we employ both structured and free-form questions. Given a question that originates from a specific scene and event, we randomly select a character with the condition that the character must be from the novel series relevant to the event’s source. We then set their time point to be earlier than that of the event, based on a comparison of the book and

heading: Prompt for Fact-based Free-from Question Generation

heading: Prompt for Event Summary and Participant List Generation

heading: Prompt for Fake Event Summary Generation
text: First, read the event summary from the Harry Potter series. Generate the fake event summary that converts the true event summary to confuse readers using one of the six methods as follows.

heading: Prompt for Fake-based Free-from Question Generation
text: chapter numbers between the character’s and the event’s time points. Note that the character’s time point is chosen randomly from among those that are earlier than the event’s time point. For generating (2) past-only type data instances, we only use free-form questions (i.e., both fact-based and fake-based questions). This is because free-form questions are aimed at assessing the character’s overall knowledge of past events (which is crucial for past-only instances), whereas structured questions are designed to assess their direct experiences or observations. In this case, the character’s period is randomly chosen from among those that occur after the event’s time point. For creating data instances of (3) past-presence and past-absence types, we only use structured questions. This is because, similar to the past-only instances, free-form questions are aimed at assessing the character’s general knowledge of past events, whereas structured questions are more focused on their direct experiences or observations. Here, we select a character who was present/absent at the event, utilizing the previously generated participants list. Their time point is then adjusted to be after that of the event. To scale up the dataset for past-presence and pastabsence types, we apply this process to all main characters to the same question instead of selecting only a single character per question, resulting in K data instances per question (e.g., K = 3 in the context of the Harry Potter series).
D.4 Add detailed descriptions to the spatiotemporal labels.
All {scene, event summary, question, character with time point} combinations are categorized into four types based on their spatiotemporal relevance. Based on each type, we enrich the spatiotemporal labels as follows: (1) Future type: we include annotations like “During {character time point}, {character} should not be aware of or contain any expression that reveals the moment when {event summary} since the moment is the future.” in the spatiotemporal label (see Table 2). (2) Past-only type: we include statements like "During {character time point}, {character} can respond based on the moment but should not wrongly recall it. (- Moment: {scene}).” in the spatiotemporal label (see Table 11). (3) Past-absence and past-presence type: we add “During {character time point}, {character} should not say that he/she was present/absent when {event summary}” to the spatiotemporal label, based on the previously generated list of partic-
ipants while adding the same statements as for pastonly type (see Table 9 and Table 10). In addition, we add a personality label for all questions, tailored to the character, by summarizing personality traits from their Fandom page personality section.
D.5 Generate gold responses and manually filter data instances.
We apply a manual filtering process that involves a series of assessments to ensure that each data instance meets all criteria:
1. Event summary quality: we evaluate the quality of each generated event summary, ensuring a survival rate of 83.96% (1,643 out of 1,957 scenes). The criteria for exclusion are as follows. (1) Lack of uniqueness: instances describing common events, such as “Harry struggled with casting spells using a blackthorn wand.” are excluded due to their repetitiveness throughout the story. (2) Ambiguity: we exclude instances where the time point or participants of the event are unclear. For example, instances are excluded if there is no specific event or object that indicates the unique time point, such as “at a class”, or if it is unclear who the participants are, as in “revealed to the school”.
2. Participants list accuracy: we assess the accuracy of the list of participants in each scene. 97.45% of them are correct (1,907 out of 1,957 scenes).
3. Fact-based free-form question quality: we scrutinize the quality of the fact-based freeform questions, with 86.46% (1,379 out of 1,595 questions). The criteria are as follows. (1) Lack of uniqueness: we exclude questions if they can be answered in multiple ways. For example, the question “Who informed Harry Potter about the threat of Sirius Black, and how did Harry respond?” is not unique since multiple characters cautioned Harry about Sirius at different times. (2) Ambiguity: we exclude questions when the entities in the question are unclear. For instance, a question like “What triggered the disturbance at the Dursleys’ breakfast table that involved Harry Potter and resulted in chaos?” is considered ambiguous due to its vague references. (3) Incorrect questions: we exclude questions that are incorrectly formulated or answer themselves. For
example, a question that embeds its own answer is “Who did Rubeus Hagrid introduce to Harry Potter and Hermione Granger as his half-brother, Grawp the giant, and request they care for in the event of his dismissal from Hogwarts?” (4) Duplication: we exclude questions when generated questions are nearly identical due to closely related event summaries.
4. Fake-based free-form question quality: we evaluate the fake-based free-form questions, achieving a survival rate of 78.24% (1,248 among 1,595 questions). The criteria are as follows. (1) Clarity: we exclude questions that remain fact-based instead of being appropriately converted to fake questions. (2) Incorrect questions: we remove questions that are irrelevant or incorrectly related to the given event’s context.
5. Gold response quality: our review of the gold responses for 10,895 {question, character with timepoint} pairs results in a survival rate of 98.04% (10,682 among 10,895). We do not exclude responses that directly contradict or are inconsistent with the assigned spatiotemporal label. Instead, we continue to regenerate the response until it meets the assigned spatiotemporal label.

heading: E Details of Dataset Analyses

heading: E.1 Statistics
text: We present a nested pie chart of free-form questions in Figure 4, and another nested pie chart of structured questions in Figure 5. Since structured questions are derived from only 18 different templates, they lack the diversity of free-form questions, which are generated using 5W1H (i.e., who, what, when, where, why, how) approach. From another perspective, the diverse verb-noun structures in free-form questions indicate their aim to probe a character’s overall knowledge of past events, whereas verb-noun structures in structured questions are designed to elicit direct experiences or observations. Nevertheless, structured questions play a crucial role in exploring the limits of a character’s knowledge boundary. That is, they include concepts of past-absence and past-presence, which are absent in free-form questions.

heading: E.2 Dataset Quality
text: In order to ensure the quality dataset, we additionally performed a human evaluation conducted via Amazon Mechanical Turk.
First, we provided annotators with comprehensive instructions as shown in Figure 6, including several examples for each criterion. The criteria are as follows: (1) the quality of the generated event summary (0 if bad, 1 if good), (2) the accuracy of participant lists (0 or 1), (3) the quality of questions generated from event summaries (0 or 1), (4) spatiotemporal consistency of the gold response (0 if inconsistent, 1 if consistent), and (5) personality consistency of the gold response (1 to 7). Note that the last criterion, ‘personality consistency’, is not our main contribution; it revisits a concept introduced by Shao et al. (2023). Additionally, we ensured that annotators were provided with ‘extracted scene’ information, in Figure 7, to enhance their contextual understanding.
Secondly, annotators were required to pass the 1st qualification Human Intelligence Task (HIT), aimed at ensuring their comprehension of the provided instructions. The compensation for each HIT was $0.75. We restricted our selection to workers from English-speaking countries (namely AU, CA, NZ, US, and GB) with a HIT approval rate of over 98%, with greater than 5000 HITs approved. Each annotator was assigned a single example, randomly chosen from four novels. It was mandatory for
them to not only select the appropriate labels but also to write their rationales in complete sentences. Based on their choices and provided rationales, we identified and qualified annotators who were both accurate in their selection and provided sound rationales. This process resulted in 42 qualified workers out of an initial pool of 86.
Thirdly, annotators were tasked with completing the 2nd qualification HITs, designed to expose them to a variety of examples before attempting the main HITs. We presented 200 examples, each evaluated by two annotators. We also committed to offering our annotators a fair compensation of approximately $16/hour (with each HIT paying $1.0), which surpasses the minimum wage in the countries from which we recruited. This stage involved 38 annotators, yielding average scores for (1) summary quality: 0.95, (2) participants list accuracy: 0.96, (3) question quality: 0.94, (4) spatiotemporal consistency of the gold response: 0.75, and (5) personality consistency of the gold response: 4.82. We use Gwet’s AC1 scores to measure inter-annotator agreement, with scores of (1) 0.9, (2) 0.91, (3) 0.87, (4) 0.5, and (5) 0.17. Note that we chose Gwet’s AC1 score for its robust performance in situations of class imbalance (Gwet, 2008; Wongpakaran et al., 2013; Wong et al., 2021). Upon reviewing their selections and rationales, we observed that some annotators still struggled with the concept of spatiotemporal consistency. As a result, we excluded 9 workers, ultimately assembling a
team of 29 annotators fully prepared to undertake the main HITs.
Finally, we assigned workers to complete the main HITs. We presented 100 examples, each evaluated by two annotators. We compensated the workers at a rate of $1.00 per HIT, and a total of 19 workers completed their assignments. The final results of the main HITs were favorable, with average scores of (1) summary quality: 0.88, (2) participants list accuracy: 0.92, (3) question quality: 0.89, (4) spatiotemporal consistency of the gold response: 0.95, and (5) personality consistency of the gold response: 5.91, with substantial Gwet’s AC1 scores of (1) 0.74, (2) 0.85, (3) 0.81, (4) 0.90, and (5) 0.19, respectively. We also highlight the agreement ratios between annotators choosing the same versus different labels: 0.78 vs. 0.22 for the quality of the summary, 0.86 vs. 0.14 for the accuracy of the participants’ list, 0.84 vs. 0.16 for the quality of the questions, 0.90 vs. 0.10 for the spatiotemporal consistency of the responses, and 0.32 vs. 0.68 for the personality consistency of the responses. Unlike the evaluation of personality consistency, which is inherently subjective, assigning binary labels to the other criteria (e.g., spatiotemporal consistency) is more straightforward due to their objective nature and the presence of clear, definitive answers, as demonstrated by the results.

heading: F Character Time Points
text: We select 14 different characters across four different novel series, assigning a total of 219 unique {character, time point} pairs.
For the Harry Potter series, time points correspond to the Hogwarts school year for each character. We have identified four key periods within this timeline: the start of the school year on September 1st, Halloween, Christmas, and the end of the scene. Note that these time points (A total of 25 time points) are consistently applied across three main characters—Harry Potter, Ronald Weasley, and Hermione Granger. To facilitate a comparison of time points with those in the questions, we manually annotate each character’s time point using the format {book_number - chapter_number}, categorized as follows:
• September 1st: [1-6, 2-5, 3-5, 4-11, 5-10, 6-7, 7-12]
• Halloween: [1-10, 2-8, 3-8, 4-16]
• Christmas: [1-12, 2-12, 3-11, 4-23, 5-23, 6- 16, 7-19]
• End of the scene: [1-17, 2-18, 3-22, 4-37, 5-38, 6-30, 7-36]
For the Lord of the Rings series, we select five main characters: Frodo Baggins, Samwise Gamgee, Gandalf, Aragorn, and Legolas. Differing from the approach taken with the Harry Potter series, we assign specific time points individually for each character due to the lack of recurring events throughout the books. In addition, we annotate the character’s time point using the format {volume_number - book_number - chapter_number}, as follows:
• Frodo Baggins (Total 12 time points): at Bilbo
Baggins’s Farewell party (1-1-1), at the moment when Frodo was stabbed by one of the Ringwraiths (1-1-11), at the moment when The Fellowship loses Gandalf in Moria (1- 2-5), end of Volume 1 (1-2-10), at the moment when encountering Gollum and decides to spare his life (2-4-1), at the moment of encountering the Black Gate of Mordor (2-4- 3), at the moment when Frodo captured by Faramir (2-4-5), at Shelob’s lair (2-4-9), end of Volume 2 (2-4-10), at the event when captured by Orcs at the Tower of Cirith Ungol (3-6-1), at the moment when rescued from Mount Doom by the eagles (3-6-4), end of Volume 3 (3-6-9).
• Samwise Gamgee (Total 12 time points): at Bilbo Baggins’s Farewell party (1-1-1), at the moment when Frodo was stabbed by one of the Ringwraiths (1-1-11), at the moment when The Fellowship loses Gandalf in Moria (1-2- 5), end of Volume 1 (1-2-10), at the moment when encountering Gollum and decides to spare his life (2-4-1), at the moment of encountering the Black Gate of Mordor (2-4-3), at the moment when captured by Faramir (2- 4-5), at Shelob’s lair (2-4-9), end of Volume 2 (2-4-10), at the event when captured by Orcs at the Tower of Cirith Ungol (3-6-1), at the moment when rescued from Mount Doom by the eagles (3-6-4), end of Volume 3 (3-6-9).
• Gandalf (Total 12 time points): at Bilbo Bag-
gins’s Farewell party (1-1-1), at the moment when the Fellowship was formed at the council of Elrond (1-2-3), at the moment when The Fellowship loses Gandalf in Moria (1-2- 5), end of Volume 1 (1-2-10), at the moment when Gandalf the white met Aragorn, Legolas, and Gimli at Fangorn Forest after the fall at Moria (2-3-5), at the moment when arriving at Isengard after the battle of Helm’s Deep (2-3- 8), at the moment when Gandalf met Saruman at Isengard (2-3-10), end of Volume 2 (2-4- 10), at the moment when Gandalf arrived at Minas Tirith with Pippin (3-5-1), during the Battle of the Pelennor Fields (3-5-6), at the coronation of King Elessar(Aragorn) (3-6-5), end of Volume 3 (3-6-9).
• Aragorn (Total 12 time points): at the moment when Aragorn first met Frodo and his companions (1-1-9), at the moment when the Fellowship was formed at the council of Elrond (1-2-3), at the moment when The Fellowship loses Gandalf in Moria (1-2-5), end of Volume 1 (1-2-10), at the moment when Gandalf the white met Aragorn, Legolas, and Gimli at Fangorn Forest after the fall at Moria (2-3-5), at the moment when Aragorn arrived at Isengard after the battle of Helm’s Deep (2-3-8), at the moment when Aragorn met Saruman at Isengard (2-3-10), end of Volume 2 (2-4-10), at the moment when Aragorn arrived at the paths of the dead (3-5-2), at the Battle of the Pelennor Fields (3-5-6), at the coronation of King Elessar(Aragorn) (3-6-5), end of Volume 3 (3-6-9).
• Legolas (Total 12 time points): at the moment when the Fellowship was formed at the council of Elrond (1-2-3), at the moment when The Fellowship loses Gandalf in Moria (1-2-5), at the moment of leaving Lothlórien (1-2-8), end of Volume 1 (1-2-10), at the moment when Leoglas met Gandalf the white at Fangorn Forest after Gandalf’s fall at Moria (2-3-5), at the moment when Legolas arrived at Isengard after the battle of Helm’s Deep (2-3-8), at the moment when Legolas met Saruman at Isengard (2-3-10), end of Volume 2 (2-4-10), at the moment when Legolas arrived at the paths of the dead with Aragorn (3-5-2), at the Battle of the Pelennor Fields (3-5-6), at the coronation of King Elessar(Aragorn) (3-6-5), end of Volume 3 (3-6-9).
For the Twilight series, we select three main characters: Bella Swan, Edward Cullen, and Jacob Black. We annotate each character’s time point using the format {book_number - chapter_number}, as follows:
• Bella Swan (Total 16 time points): at the moment when Bella moved from Phoenix to Forks (1-1), at the moment when Bella first confirmed Edward’s true nature as a vampire (1-9), at the moment when Bella first visited the Cullens (1-15), end of book 1 (1-25), on Bella’s 18th birthday (2-1), at the moment when Bella jumps off the cliff into the ocean (2-15), at Volterra (2-20), end of book 2 (2- 25), at the moment when Bella was grounded
by her father (3-1), at the moment when Bella learns about the history of the Quileute tribe and the Cullens (3-11), at the moment when Bella receives an engagement ring from Edward (3-20), end of book 3 (3-27), at Bella and Edward’s Wedding (4-3), at the moment when Renesmee was born (4-18), at the moment when Bella forges passports and IDs for Renesmee and Jacob from J. Jenks (4-33), end of book 4 (4-39).
• Edward Cullen (Total 16 time points): at the moment when Edward saves Bella from a Van (1-3), at the moment when Bella first confirm Edward’s true nature as a vampire (1-9), at the moment when Bella first visited the Cullens (1-15), end of book 1 (1-25), on Bella’s 18th birthday (2-1), at the moment when Edward tells Bella that he and the Cullens are leaving Forks (2-3), at Volterra (2-20), end of book 2 (2-25), at the moment when Edward rewarded Alice for watching Bella by giving her the canary yellow Porsche from Italy (3-6), at Bella’s graduation ceremony (3-16), at the moment when Jacob crawled into the sleeping bag beside Bella at the campsite, chosen for Bella’s hiding place (3-22), end of book 3 (3- 27), at Bella and Edward’s Wedding (4-3), at the moment when Renesmee was born (4-18), at the moment when Esme has renovated a cottage on the property for Bella, Edward, and now Renesmee (4-24),end of book 4 (4-39).
• Jacob Black (Total 16 time points): at the beach at La Push, when Jacob met Bella and her friends (1-6), on March 10, 2005 (1-11), on March 13, 2005 when Bella found Jacob before watching a baseball game with Edward (1-17), end of book 1 (1-25), at the moment when Jacob and Bella worked together on repairing two old motorcycles (2-6), at the moment when Bella first discovers Jacob’s werewolf identity (2-10), at the moment when Jacob pulled out Bella from drowning (2-16), end of book 2 (2-25), on May 31, 2006 when Bella found Jacob on his motorcycle at the school (3-16), at Bella’s graduation ceremony (3-7), at the moment when Jacob crawled into the sleeping bag beside Bella at the campsite, chosen for Bella’s hiding place (3-22), end of book 3 (3-27), at Bella and Edward’s Wedding (4-3), at the moment when Renesmee
was born (4-18), at Christmas, 2006 (4-34), end of book 4 (4-39).
For the Hunger Games series, we select three main characters: Katniss Everdeen, Peeta Mellark, and Gale Hawthorne. We annotate each character’s time point using the format {book_number - chapter_number}, as follows:
• Katniss Everdeen (Total 12 time points): at the moment when Katniss volunteered to take her sister’s place as the female tribute (1-2), at the start of the 74th Hunger Games (1-11), at the moment when Katniss found wounded Peeta hidden under a layer of mud (1-19), end of book 1 (1-27), at the moment when they arrived at District 11 for the first stop of the Victory Tour (2-4), at the announcement of the Quarter Quell (2-12), at the moment when Katniss first witnessed a heavy fog during the Quarter Quell (2-20), end of book 2 (2-27), at the first conversation about the bombing of district 12 with Gale (3-1), at the moment when Peeta suddenly warned of an impending attack on District 13 (3-9), at the moment when the squad 451 was attacked by the mutts in the tunnels (3-22), end of book 3 (3-28).
• Peeta Mellark (Total 12 time points): at the moment when Katniss volunteered to take her sister’s place as the female tribute (1-2), at the start of the 74th Hunger Games (1-11), at the moment when Katniss found wounded Peeta hidden under a layer of mud (1-19), end of book 1 (1-27), at the moment when they arrived at District 11 for the first stop of the Victory Tour (2-4), at the announcement of the Quarter Quell (2-12), at the moment when Katniss first witnessed a heavy fog during the Quarter Quell (2-20), end of book 2 (2-27), at the moment when Peeta suddenly warned of an impending attack on District 13 (3-9), at the moment when Peeta was sent as new member of squad 451 by president Coin (3- 18), at the moment when the squad 451 was attacked by the mutts in the tunnels (3-22), end of book 3 (3-28).
• Gale Hawthorne (Total 12 time points): at the moment when Katniss volunteered to take her sister’s place as the female tribute (1-2), at the start of the 74th Hunger Games (1-11), at the moment when Katniss found wounded Peeta
hidden under a layer of mud (1-19), end of book 1 (1-27), at the moment when Katniss delivered the animals she caught before the Victory Tour (2-1), at the announcement of the Quarter Quell (2-12), at the moment when Katniss first witnessed a heavy fog during the Quarter Quell (2-20), end of book 2 (2-27), at the first conversation about the bombing of district 12 with Katniss (3-1), at the moment when Peeta suddenly warned of an impending attack on District 13 (3-9), at the moment when the squad 451 was attacked by the mutts in the tunnels (3-22), end of book 3 (3-28).

heading: G Details of Experiments on
text: TIMECHARA
G.1 Implementation Details
For generating responses, we apply nucleus sampling with p = 1 and temperature τ = 0.2 across all role-playing LLMs. We cap the maximum token length at 2048 tokens. For GPT-4 as judges, we set p = 0.95, a temperature of τ = 0.0, and a maximum token length of 1024 tokens. We use a single NVIDIA RTX A6000 GPU to generate responses with Mistral. For AlignScore evaluation, we use a single NVIDIA Quadro RTX 6000 GPU.

heading: G.2 Few-Shot Method
text: The examples are carefully selected to represent a range of question types: future, past-presence, past-absence, and past-only, each paired with a corresponding response. The responses are generated by GPT-4 and are then manually checked for spatiotemporal and personality consistency. We then append these four examples, complete with their questions and correct responses, before the question.

heading: G.3 Self-Refine Method
text: We incorporate a self-feedback prompt to facilitate this process, as outlined in Table 18. The model iteratively refines its response for a maximum of three iterations. It considers both spatiotemporal and personality scores, each with a maximum of 3 points. The response is finalized when a combined score of 5 or more is reached.

heading: Prompt for Self-Feedback of Self-Refine Method
text: We want to iteratively improve the provided responses, mimicking the character {agent_name}. To help improve, scores for each response on desired traits are provided: 1) Spatiotemporal Consistency and 2) Personality Consistency.
*** [Interactions] Interviewer: {question} {agent_name}: {response}
[Evaluation Criterion] - Spatiotemporal Consistency (0 or 3): Is the response consistent with the character’s spatiotemporal knowledge? If the response includes information that the character couldn’t have known (either because it pertains to a future event or a past event they were not present for), assign a score of 0. If the response accurately reflects only the knowledge and events the character has experienced or been aware of, give a score of 3. - Personality Consistency (1 to 3): Is the response consistent with the character’s personality? Use the given scale from 1-3 to rate how well the response reflects the personalities, including preferences, values, and convictions of the character. 1 being not at all reflective of the character’s personalities, and 3 being perfectly reflective of the character’s personalities. ***
1. Read through the [Interactions] and evaluate the spatiotemporal consistency: print the single-sentence rationale with the score on its own line corresponding to the correct answer. 2. Read through the [Interactions] and evaluate the personality consistency: print the single-sentence rationale with the score on its own line corresponding to the correct answer. 3. Print the total score.
Table 18: Prompt for generating self-feedback during response generation with self-refine method.

heading: G.4 Retrieval-Augmented Generation (RAG) Method
text: We employ the LangChain framework5 to implement the RAG method. We retrieve up to six paragraphs from a raw text source based on the given question. These retrieved paragraphs are then attached to the end of the question, serving as additional context for generating the response.

heading: G.5 Decomposed Reasoning via
text: NARRATIVE-EXPERTS

heading: G.5.1 NARRATIVE-EXPERTS
text: We detail the prompts used for two narrative experts: a temporal expert (refer to Table 19) and a spatial expert (refer to Table 20). For the temporal expert, we compare the outputs numerically to the character’s time point, presented in the ‘book number - chapter number’ format, to determine if the question pertains to the future relative to the character’s timeline. In addition, we outline the complete algorithm that describes how the role-playing LLM generates a response based on a question and the hints provided by these two experts, as presented in Algorithm 1.

heading: G.5.2 NARRATIVE-EXPERTS-RAG-CUTOFF
text: It can be effective to combine NARRATIVEEXPERTS with RAG-cutoff methods, because the RAG-cutoff addresses past-only type questions, especially those that are fake-based, and NARRATIVE-EXPERTS is compatible with any baselines. For NARRATIVE-EXPERTS-RAGCUTOFF, we incorporate a retrieval module to aug-
5https://github.com/langchain-ai/langchain
ment the capabilities of both the temporal and spatial experts. Specifically, we retrieve up to six paragraphs from a raw text source based on the input question. These paragraphs are subsequently incorporated into the prompts for the two narrative experts, as detailed in Table 21 and Table 22. The role-playing LLM then generates an answer using the question, hints from both experts and the paragraphs corresponding to events before the character’s defined period. Note that we utilize all six paragraphs to assist the narrative experts (adopting a naive RAG approach rather than RAG-cutoff), but we limit the paragraphs when producing the final response. Furthermore, we exclude paragraphs in the final response if the temporal expert predicts ‘future’ events since it slightly enhances spatiotemporal consistency performance. The algorithm for NARRATIVE-EXPERTS-RAG-CUTOFF is detailed in Algorithm 2.

heading: G.6 Experimental Results on the 11K Dataset
text: While Table 5 presents experimental results from a randomly sampled set of 600 data instances, we additionally conduct experiments on the entire dataset comprising 10,895 instances.
Baseline methods. We utilize two different open-source LLMs as the backbone models for our role-playing agents: Mistral 7B Instruct (i.e., mistral-7b-instruct-v0.2) (Jiang et al., 2023) and Llama-2 Chat 13B (Touvron et al., 2023). Note that we exclude GPT-4 Turbo and GPT-3.5 Turbo from our backbone models due to their high costs. Following § 5.2, we implement zero-shot prompting and RAG-cutoff as our baseline methods.

heading: Prompt for Book and Chapter Identification of NARRATIVE-EXPERTS Method
text: You will be given a question from {series_name} series at a specific time. Your task is to identify the exact {book_chapter_name} of the scene of the question. Below is the data:
*** [Question] {question} *** [Identification Criterion] What is the exact {book_chapter_name} of the scene of the question?
1. Read through the [Question], recall the scene from the question, and describe it using the six Ws (Who, What, When, Where, Why, and How). 2. Identify the exact {book_chapter_name} of the scene of the question, in ’{book_chapter_format}’ format.
First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then, print the output on its own line corresponding to the correct answer. At the end, repeat just the selected output again by itself on a new line.
Table 19: Prompt for identifying the exact book and chapter of the scene in the question.

heading: Prompt for Event Participants Identification of NARRATIVE-EXPERTS Method
text: You will be given a question and a character from {series_name} series. Your task is to classify whether the character is a participant (i.e., present or absent) in the scene of the question. Below is the data:
*** [Question] {question} [Character] {character} *** [Classification Criterion] Is the character a participant in the scene of the question?
[Classification Steps] 1. Read through the [Question], recall the scene from the question, and describe it using the six Ws (Who, What, When, Where, Why, and How). 2. Identify the exact {book_chapter_name} of the scene of the question. 3. Write a list of every character involved in the scene described in the question, including those not explicitly mentioned in the question but who were present in the scene. 4. Compare the list of participants to the character. Check if the list of participants contains the character. 5. If the list contains the character, classify it as ’present’. Otherwise, classify it as ’absent’. ***
First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then, print the output on its own line corresponding to the correct answer. At the end, repeat just the selected output again by itself on a new line.
Table 20: Prompt for identifying whether the given character is a participant in the scene of the question.

heading: Prompt for Book and Chapter Identification of NARRATIVE-EXPERTS-RAG-CUTOFF Method
text: You will be given a question and contexts from {series_name} series at a specific time. Your task is to identify the exact {book_chapter_name} of the scene of the question. Below is the data:
*** [Question] {question} *** [Contexts] {contexts} *** [Identification Criterion] What is the exact {book_chapter_name} of the scene of the question?
1. Read through the [Question] and [Contexts], recall the scene from the question, and describe it using the six Ws (Who, What, When, Where, Why, and How). 2. Identify the exact {book_chapter_name} of the scene of the question, in ’{book_chapter_format}’ format.
First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then, print the output on its own line corresponding to the correct answer. At the end, repeat just the selected output again by itself on a new line.
Table 21: Prompt for identifying the exact book and chapter of the scene in the question while using a retrieval module.

heading: Prompt for Event Participants Identification of NARRATIVE-EXPERTS-RAG-CUTOFF Method
text: You will be given a question, a character, and contexts from {series_name} series. Your task is to classify whether the character is a participant (i.e., present or absent) in the scene of the question. Below is the data:
*** [Question] {question} [Character] {character} *** [Contexts] {context} *** [Classification Criterion] Is the character a participant in the scene of the question?
[Classification Steps] 1. Read through the [Question] and [Contexts], recall the scene from the question, and describe it using the six Ws (Who, What, When, Where, Why, and How). 2. Identify the exact {book_chapter_name} of the scene of the question. 3. Write a list of every character involved in the scene described in the question, including those not explicitly mentioned in the question but who were present in the scene. 4. Compare the list of participants to the character. Check if the list of participants contains the character. 5. If the list contains the character, classify it as ’present’. Otherwise, classify it as ’absent’. ***
First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then, print the output on its own line corresponding to the correct answer. At the end, repeat just the selected output again by itself on a new line.
Table 22: Prompt for identifying whether the given character is a participant in the scene of the question while using a retrieval module.
Evaluation metrics. Evaluating spatiotemporal or personality consistency with GPT-4 judges incurs an extremely high cost. As a result, we utilize AlignScore (Zha et al., 2023) for assessing factual inconsistencies across scenarios such as natural language inference and fact verification tasks. Specifically, we employ the RoBERTa-large model equipped with a 3-way classification head. Given the gold response and the predicted response from the role-playing agent, the RoBERTa model produces an AlignScore ranging from 0 to 1, which we then average.
Experimental results. Experimental results are presented in Table 23. As expected, methods based on Mistral achieve higher AlignScores than those based on Llama 2, despite the smaller model size. NARRATIVE-EXPERTS and NARRATIVE-EXPERTS-RAG-CUTOFF outperform baseline methods across both Llama 2 and Mistral models. Note that the performance trend of the Mistral-based method mirrors that of the main experiment in § 5 (Table 5), with scores of 19.68, 17.94, 21.85, and 22.34 for zero-shot, ragcutoff, NARRATIVE-EXPERTS, and NARRATIVEEXPERTS-RAG-CUTOFF, respectively, in Table 23, compared to 18.50, 17.82, 20.57, 22.20 in Table 5. Although the AlignScore may not be entirely interpretable or reliable, our results indicate that employing our methods enhances spatiotemporal consistency across the entire dataset.

heading: H Further Analyses

heading: H.1 Human Evaluation on Assessing Spatiotemporal Consistency
text: We supplement our findings with results from manual evaluations by human judges via Amazon Mechanical Turk on a subset of the dataset used in Table 5. To specify, we randomly selected 50 instances out of 600, ensuring an even distribution across the four data types (i.e., future, pastabsence, past-presence, and past-only types). We then applied four methods (i.e., zero-shot, RAGcutoff, narrative-experts, narrative-experts-RAGcutoff) based on GPT-4 to generate four different responses for each instance. Subsequently, we tasked annotators with evaluating the spatiotemporal consistency of each response using a binary scale: 0 for inconsistency and 1 for consistency. Two annotators evaluated each response, and we calculated the average score by dividing the sum of the scores by 2. Specifically, we opted to assemble a new pool of annotators rather than rely on those who had previously worked on Appendix E.2. This is because we observed that evaluating ‘predicted’ responses posed more challenges than assessing gold responses. To be similar to Appendix E.2, we collected qualified human annotators via two-step qualification steps, ultimately forming a team of 27 annotators ready to proceed with the main HITs.
The results in Table 24 reveal a closely aligned trend between the two evaluation approaches (i.e.,
Algorithm 1: Decomposed Reasoning Method for Role-Playing LLM
Input: A question regarding a scene, a character with their time point Output: A response Function TemporalExpert(question)
Identify the book and chapter of the scene of the question; if scene is in the future then return “future”, “Note that the period of
the question is in the future relative to {character}’s time point. Therefore, you should not answer the question or mention any facts that occurred after {character}’s time point.”;
end return “past”, “”;
end Function SpatialExpert(question, character)
Determine if character participates in the scene of the question; if character is past-absent then return “Note that {character} had not
participated in the scene described in the question. Therefore, you should not imply that {character} was present in the scene.”;
end return “”;
end Function RolePlayingLLM(question, hints)
Append hints to the prompt; Generate response based on enhanced prompt; return response;
end Procedure MainProcedure(question, character)
// Invoke Temporal Expert temporalStatus, temporalHint := TemporalExpert(question); spatialHint := “”; if temporalStatus == “past” then
// Invoke Spatial Expert only if past spatialHint := SpatialExpert(question, character);
end // Prepare hints hints := temporalHint + spatialHint; // Invoke Role-Playing LLM with hints response := RolePlayingLLM(question, hints); return response;
end
GPT-4 evaluation and human evaluation) across methods and data types, with a marginal gap. In addition, we calculated Gwet’s AC1 scores between two annotators, obtaining a value of 0.91. We also highlight the agreement ratios between annotators who chose the same versus different labels: 0.94 vs. 0.06 for GPT-4 zero-shot, 0.94 vs. 0.06 for GPT-4 RAG-cutoff, 0.96 vs. 0.04 for GPT-4 narrativeexperts, 0.94 vs. 0.06 for GPT-4, and narrativeexperts-RAG-cutoff. This further step emphasizes our commitment to ensuring the reliability and validity of both the automatic evaluation process and our experimental findings.

heading: H.2 Fine-grained Hallucination Detection
text: While the spatiotemporal consistency in Table 5 presents only accuracy, GPT-4 judges also provide rationales for their decision. To thoroughly analyze point-in-time hallucinations in the role-playing LLM, we first use four types of hallucination type from Mishra et al. (2024) and additionally define four types.
1. Entity: Incorrect facts about a person, place, or thing.
2. Relation: Wrong connections between characters or objects.
3. Contradictory: A statement that goes against known facts.
4. Invented: Making up facts or details that don’t exist.
5. Temporal error (new): Getting the timing of events wrong, applicable to future type instances.
6. Spatial error (new): Incorrect event participants reasoning, relevant for past-absence and past-presence type instances.
7. Unclarified (new): Not correcting a wrong question, specifically not addressing false information in fake-based questions.
8. Incomplete (new): Omission of parts of an answer, especially when responding to freeform questions.
Then, we collect and manually annotate 199 instances where GPT-4 Turbo’s responses are incorrect.
Algorithm 2: Decomposed Reasoning with RAG-cutoff Input: A question regarding a scene, a character Output: A response Function RetrieveParagraphs(question)
Retrieve up to six paragraphs based on the question from raw text source; return paragraphs;
end Function LimitParagraphsBeforeCharacterPeriod(paragraphs, character)
Filter paragraphs to only include those corresponding to events before the character’s period; return filteredParagraphs;
end Function TemporalExpertWithRAG(question, character, paragraphs)
Incorporate paragraphs into the prompt to identify the book and chapter of the scene of the question; if scene is in the future then
return “future”, “Note that the period of the question is in the future relative to {character}’s time point. Therefore, you should not answer the question or mention any facts that occurred after {character}’s time point.”;
end return “past”, “”;
end Function SpatialExpertWithRAG(question, character, paragraphs)
Incorporate paragraphs into the prompt to determine if character participates in the scene of the question; if character is past-absent then
return “Note that {character} had not participated in the scene described in the question. Therefore, you should not imply that {character} was present in the scene.”;
end return “”;
end Function RolePlayingLLMWithRAGCutoff(question, hints, paragraphs, character, temporalStatus)
filteredParagraphs := ; if temporalStatus == “future” then
filteredParagraphs := “”; else
filteredParagraphs := LimitParagraphsBeforeCharacterPeriod(paragraphs, character); end Append hints and filtered paragraphs to the prompt; Generate response based on the enhanced prompt; return response;
end Procedure MainProcedureWithRAGCutoff(question, character)
// Retrieve relevant paragraphs paragraphs := RetrieveParagraphs(question); // Invoke Temporal Expert with RAG temporalStatus, temporalHint := TemporalExpertWithRAG(question, character, paragraphs); spatialHint := “”; if temporalStatus == “past” then
// Invoke Spatial Expert with RAG only if past spatialHint := SpatialExpertWithRAG(question, character, paragraphs);
end // Prepare hints and paragraphs hints := temporalHint + spatialHint; // Invoke Role-Playing LLM with RAG-cutoff response := RolePlayingLLMWithRAGCutoff(question, hints, paragraphs, character, temporalStatus); return response;
end
Table 25 shows a fine-grained hallucination-type distribution of 199 responses. As anticipated, ‘Temporal Error’ was the most common type, primarily because all the incorrect examples from the future type data instances fell under Temporal Error. On the other hand, most errors in the past-absence and past-presence were classified as ‘Spatial Errors’. The distribution for past-only was the most balanced, with a notable number of cases falling into the ‘Entity’ category. This trend is largely due to instances where role-playing agents failed to detect swapped entities in fake-based questions.

heading: H.3 Experimental Results per Novel Series
text: Table 26 shows the spatiotemporal consistency of the GPT-4 Turbo zero-shot method across different novel series, highlighting distinct performance trends. Specifically, the model tends to produce incorrect responses more often for the future and pastabsence types within the Harry Potter series. In contrast, it exhibits lower performance for the pastpresence type but performs better in other types within the Lord of the Rings series. Across all series, a common challenge is the model’s struggle to accurately respond to instances of both future and past-only (fake) types.

heading: H.4 Performance for Fact-Based vs. Fake-Based Questions
text: As shown in Table 26 and further supported by additional experimental results in Table 27, LLMs indeed do better at fact-based questions than fakebased ones, as expected. Although fake-based questions might not seem natural within a role-playing
scenario, they are crucial from a fact-checking perspective to evaluate the agent’s capability of avoiding point-in-time character hallucinations. Therefore, the inability of the agent to appropriately respond to fake-based questions remains a significant concern.

heading: H.5 Accuracy of Narrative Experts
text: While we show the effectiveness of NARRATIVEEXPERTS and NARRATIVE-EXPERTS-RAGCUTOFF, we further analyze the performance of each expert. To evaluate the temporal expert, we use all 600 data instances used in Table 5, since all data instances are classified as either future or past. On the other hand, we only use 200 fact-based structured data instances from past-absence and past-presence types to evaluate the spatial expert.
Table 28 demonstrates the accuracy of narrative experts using three backbone LLMs.
NARRATIVE-EXPERTS vs. NARRATIVEEXPERTS-RAG-CUTOFF. Despite lacking access to external knowledge, narrative experts significantly outperform random selection in classifying past/future and presence/absence scenarios. This indicates that LLMs rely on their parametric memories for spatiotemporal reasoning. Moreover, NARRATIVE-EXPERTS-RAG-CUTOFF outperforms NARRATIVE-EXPERTS on average, highlighting the benefits of incorporating retrieval modules.
Comparison of backbone models. As expected, narrative experts using GPT-4 Turbo achieve the highest accuracy on average. While the accuracy of temporal experts with both GPT-4 Turbo and GPT-3.5 Turbo in predicting future events exceeds 75%, the temporal expert with Mistral faces challenges in future predictions. Conversely, Mistral surpasses GPT-3.5 Turbo in predicting past events and absences.
Instruction following capability. The temporal expert of GPT-3.5 Turbo NARRATIVE-EXPERTSRAG-CUTOFF shows an impressive 89.5% accuracy in detecting future events. However, its performance on the future type in Table 5 reveals that the GPT-3.5 Turbo model, when acting as a roleplaying agent, often fails to follow provided instructions. Furthermore, we demonstrate that the underperformance of NARRATIVE-EXPERTS and NARRATIVE-EXPERTS-RAG-CUTOFF in the pastpresence type, as shown in Table 5, is often due to the temporal and spatial experts’ occasional failures to predict past events and presences accurately.
Consequently, this leads to the role-playing LLMs following ‘incorrect’ instructions.

heading: H.6 Evaluating Role-Playing LLM Agents on Time Points More Distant in the Future
text: While we presented Figure 1 as a motivational example of TIMECHARA, we observed that the challenge of detecting character hallucination, particularly with future type questions, diminishes as the temporal gap (i.e., the gap between the character’s time point and the question’s time point) widens. Conversely, it remains challenging for LLMs to identify hallucinations when the temporal difference narrows, as illustrated in Table 5. To empirically validate this observation, we designed an experiment as follows: We selected 15 questions from the latter part of “Harry Potter and the Deathly Hallows” (book 7), allocating them between Harry Potter and Hermione Granger - 8 for Harry and 7 for Hermione. We then positioned the characters at seven distinct time points (i.e., the start of each school year from year 1 to year 7), resulting in a total of 15 ∗ 7 = 105 unique future type instances. Note that this scalable approach enabled us to generate examples that assess models at time points distant in the future.
The Table 29 presents the spatiotemporal consistency across 105 instances, illustrating the effect of the temporal gap between the character’s time point and the question’s time point on model performance. According to the results, LLMs (with zero-
shot prompt) exhibit an increase in spatiotemporal consistency as the temporal distance between the character’s time point and the question’s time point widens. Specifically, ‘GPT-3.5 zero-shot’ showed a maximum of 46.7% spatiotemporal consistency, indicating persistent hallucinations, while ‘GPT-4 zero-shot’ achieved 100%. This suggests that roleplaying LLMs, especially GPT-4, with a higher memorization capability, are able to recognize this extent of temporal difference as the temporal gap between the character and the question expands in future type instances. However, it remains challenging for LLMs to maintain spatiotemporal consistency when the temporal distance between the character’s time point and the question’s time point decreases (e.g., GPT-3.5: 6.7%, GPT-4: 46.7%), as also depicted in Table 5.

heading: Future Work
text: While we presented Figure 1 as a motivational example of TIMECHARA, we observed that the challenge of detecting character hallucination, particularly with future type questions, diminishes as the temporal gap (i. e. , the gap between the character’s time point and the question’s time point) widens. Conversely, it remains challenging for LLMs to identify hallucinations when the temporal difference narrows, as illustrated in Table 5. To empirically validate this observation, we designed an experiment as follows: We selected 15 questions from the latter part of “Harry Potter and the Deathly Hallows” (book 7), allocating them between Harry Potter and Hermione Granger - 8 for Harry and 7 for Hermione. We then positioned the characters at seven distinct time points (i. e. , the start of each school year from year 1 to year 7), resulting in a total of 15 ∗ 7 = 105 unique future type instances. Note that this scalable approach enabled us to generate examples that assess models at time points distant in the future. The Table 29 presents the spatiotemporal consistency across 105 instances, illustrating the effect of the temporal gap between the character’s time point and the question’s time point on model performance. According to the results, LLMs (with zero-
shot prompt) exhibit an increase in spatiotemporal consistency as the temporal distance between the character’s time point and the question’s time point widens. Specifically, ‘GPT-3. 5 zero-shot’ showed a maximum of 46. 7% spatiotemporal consistency, indicating persistent hallucinations, while ‘GPT-4 zero-shot’ achieved 100%. This suggests that roleplaying LLMs, especially GPT-4, with a higher memorization capability, are able to recognize this extent of temporal difference as the temporal gap between the character and the question expands in future type instances. However, it remains challenging for LLMs to maintain spatiotemporal consistency when the temporal distance between the character’s time point and the question’s time point decreases (e. g. , GPT-3. 5: 6. 7%, GPT-4: 46. 7%), as also depicted in Table 5