Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection
Authors: Kabir Ahuja
Melanie Sclar
Yulia Tsvetkov
Venue: COLM 2025 (2025)
Keywords: narrative understanding
reasoning
synthetic data generation
test time scaling
evaluation
natural language generation
TL;DR: We propose an automatic method to generate stories with logical inconsistencies, which we then use to curate a benchmark to evaluate capabilities of LLMs towards reasoning for plot holes in stories.

Abstract: Stories are a fundamental aspect of human experience. Engaging deeply with stories and spotting plot holes—inconsistencies in a storyline that break the internal logic or rules of a story’s world—requires nuanced reasoning skills, including tracking entities and events and their interplay, abstract thinking, pragmatic narrative understanding, commonsense and social reasoning, and theory of mind. As Large Language Models (LLMs) increasingly generate, interpret, and modify text, rigorously assessing their narrative consistency and deeper language understanding becomes critical. However, existing benchmarks focus mainly on surface-level comprehension. In this work, we propose plot hole detection in stories as a proxy to evaluate language understanding and reasoning in LLMs. We introduce FlawedFictionsMaker, a novel algorithm to controllably and carefully synthesize plot holes in human-written stories. Using this algorithm, we construct a benchmark to evaluate LLMs’ plot hole detection abilities in stories —FlawedFictions—robust to contamination, with human filtering ensuring high quality. We find that state-of-the-art LLMs struggle in accurately solving FlawedFictions regardless of the reasoning effort allowed, with performance significantly degrading as story length increases. Finally, we show that LLM-based story summarization and story generation are prone to introducing plot holes, with 50%+ and 100%+ increases in plot hole detection rates with respect to human-written originals.

Published as a conference paper at COLM 2025
Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection
Kabir Ahuja
Melanie Sclar Yulia Tsvetkov
Paul G. Allen Center for Computer Science & Engineering
University of Washington Seattle, USA
(kahuja,msclar, msclar, yuliats) jecs. washington. edu
Abstract
Stories are a fundamental aspect of human experience. Engaging deeply with stories and spotting plot inles--inconsistencies the internal logic or rules of a story's s world-require, nuanced reasoning skills, including tracking entities and events and their interplay, abstract thinking, pragmatic narrative understanding, commonsense and social reasoning, and theory of mind. As Large Language Models (LLMs) in- creasingly generate, interpret, and modify text, rigorously assessing their narrative consistency and deeper language understanding becomes critical. However, existing benchmarks focus mainly on surface-level comprehen- sion. In this work, we propose plot hole delection in stories as proxy to evaluate language understanding and reasoning in LLMs. We introduce SLAWEDFICTIONSMM...
storyline that break
novel algorithm to controllably and carefully synthesize plot holes human-written stories. Using this algorithm, we construct benchmark to evaluate LLMs' plot hole detection abilities PLAWEDFICTIONS- robust to contamination, with human filtering ensur- ing high quality. We find that state-of-the of-te-of-the-art LLMs struggle in accurately solving FLAWEDFICTIONS regardless of the reasoning effort allowed, with performance significantly degrading story length increases. Finally, show that LLM-based story summarization and story generation are prone to introducing plot holes, with 50%+ and 100%+ increases in plot hole detection rates with respect to human-written originals.
https: //gpt////gthub..... www.gtthu...ee htt.:h/w.w.cookkc....aiiaa/aaaae
1 Introduction
Narratives form a fundamental mode of human cognition and weeaing-mmkingg acting as primary way people organize, experience, and construct reality (Bruner, 1991). When we engage with stories, we typically go beyond a literal understanding of what happened, instead performing complex and nuanced reasoning that involves mental representation of a story's ss world and its characters (Gerrig, 1993; Mar & Oatley, 2008; Zunshine, 2006; Kidd & Castano, 2013). Ultimately, narrative understanding cognitive capacities for language comprehension and reasoning (Kintsch, 1998).
reflection of broader human
In this work, we propose quantify narrative understanding in LLMs as a novel test bed of general language understanding and reasoning abilities. While different language understanding benchmarks are widespread in existing literature(Wang et al., 2018; 2019; Zellers et al., 2019; Hendrycks et al., 2020; Jaradeh et al., 2023), they often fail to capture the full spectrum of abilities present in narrative understanding. For example, the popular MMLU benchmark (Hendrycks et al., 2020) evaluates advanced multi-hop knowledge, but lacks assessment of pragmatics and implicit social dynamics inherent in narratives. Existing datasets studying such capabilities (Mostafazadeh et al., 2016; Sap et al., 2019; Sprague et al., 2024; Kim et al., 2023), on the other hand, are not suited for benchmarking LLMs at scale, they focus on very short fully synthetic stories that lack core elements of

---

Published as a conference paper at COLM 2025
[..]] His left orm sigred. He holds
B. Extract Story Facts
[-.] knea
[-]] Hia arm
continuly emor?
Adc
(Binary task)
(4:)
91
stff and unnaatural
Biker Street de Watson a
His holds it stif and unneatural (..]
CA2
File holds in stit and unisetural
yea
manner [..]
(..]
Ast
I..3 "I made mp
famark, howiiver
ALL
L..] 11 made no remark, howeverr viuunde knea (...]
AGE
[..] "I made no
but however. wounded kndel [..]
yes, spectyy what's the and locallee the laxt lines [Localization task)
(4.
wounded arm L-.]
CA
And he stretshed his long
Azt
Strotched his long
di What Wastsnn hid a war wound on left knee instaad??
streched his long
200
Wacastt's la snsueed changed locastonn
whoo hand up for
wha up
hand up for [...]
44g
[-.]
(4.7
A. Partition Original Story in Three Acts
Select and Build
Generate
Rebuild Story:
Contradicting Fact
Counterfactual Story
Creating Plot Hole
Evaluate on rebuilt story
Figure 1: Example of FLAWEDFICTIONSMAN (without the filtering step) in action that can be used to introduce plot holes in a plot hole-free story.
narrative structure. As consequence, it remains difficult to holistically assess overall progress in language understanding and reasoning, despite recent advances in improving LLM reasoning capabilities through advanced prompting (Wei et al., 2022; Yao et al., 2024; Wang et al., 2023) or inference time scaling (Lambert et al., 2024; Guo et al., 2025).
How do we quantify such "deeper narrative understanding"" We propose a novel task of plot hole detection as a proxy to assess deep narrative understanding and reasoning in LLMs. Plot holes are inconsistencies in a story that go against the logic flow established by the story plot (Ryan, 2009), with significant discourse dedicated to both locating' and preventing them during screen writing (McKee, 1997; MasterClass, 2021). Plot hole detection requires nuanced reasoning about the implications of established facts and elements, how they interplay, and their plausibility. Specifically, robust state tracking is needed to follow entities and rules established by the story over a long context; commonsense and pragmatic reasoning are needed for interpreting implicit world knowledge and beliefs; and theory of mind is required for reasoning over beliefs, motivations, and desires of characters. Beyond acting as test bed for complex reasoning, models that can accurately assess plot holes in stories can be useful to improve consistency in writing, be it human- or machine-generated
We propose FLAWEDFICTIONSMARE AKFR, an automatic method for introducing plot holes in existing stories Our algorithm functions by extracting relevant facts from the first act of a story and negating them in subsequent acts to introduce an inconsistency (Figure 1). to curate the first high-quality benchmark for plot
We then use FLLWEDDICTIONSSMARE hole detection-FLAWWDEE - consisting of short stories labeled with their inherent inconsistencies lack thereof. We opt for partial synthetic data approach to construct this benchmark to make it dynamically extensible and avoid data contamination (i.e., memorization of the existing stories with plot holes during LLM training). Data generated through our algorithm is then manually verified to ensure quality. consists of two tasks: binary classification task where the LLM must determine whether there is a plot hole in the story, and a localization task where the model determines both the text span introducing the plot hole and the one with the information being contradicted. The first task is naturally reduced version of the second.
FICTIONSS
We find that large majority of frontier LLM and reasoning models like GPT-4o, 03-mini, and 1amma-3.3-70B struggle in FLAWEDFICTIONS, with story length having significant negative effect on LLM's plot hole detection capabilities. FLAWEDFICTIONNI..NN our benchmark containing longer stories in the 1, 1,200-4,000 word range, proves particularly difficult, with almost all models obtaining close to random level performance on the classification task. Plot hole detection also proves to be difficult irrespective of the reasoning budget allowed: and sometimes worsened performance with increased reasoning budget.
an extension of
reasoning models, such as ol and o3-mini, show a stable
Finally, we conduct a case study to explore the use of plot hole detection for evaluating consistency of LLM generated stories. Considering the tasks of story summarization and
This is especially true in the context of films, with dedicated subreddits like r/plotholes and r/MovieMistaker or Goofs section dedicated to each film page on IMDB.
2

---

Published as a conference paper at COLM 2025
contemporary adaptation of classical short stories, we find that LLM-generated outputs trig- ger significantly more plot-holes-ove over 50% more in summarization and 100% more in contemporary adaptation-usee
our best performing model on FLAWEDFICTIONS.
Overall, our work introduces a novel evaluation task-plot deeper language understanding and reasoning in LLMs, along with controllable synthetic data generation algorithm FLAWEDFICTIONSMAKE FLAWEDFICTIONS, enabling systematic and holistic comparison of state-of-the-art models, uncovering critical gaps in their narrative comprehension, and providing a powerful frame- work for evaluating the quality of LLM-generated stories. We will make our dataset and code publicly available at the time of publication.
hole detection- for assessing
and an accompanying benchmark
2 Defining Plot Holes: Continuity Errors
Plot holes are commonly categorized into multiple categories (Shattuck, 2024) including: continuity errors (contradictions of established facts), out of character behavior (actions incon- sistent with established motivations), factual errors (historical anachronisms or real-world inaccuracies), impossible events (violations of science or logic), and unresolved storylines (in- complete plot threads). See Table 2 in Appendix for examples. We focus on continuity errors as they encompass the most general form of plot hole: both out of character behavior and impossible events can be framed as breaks in continuity, character traits or story settings. While Ryan (2009) distinguishes between harmless plot holes (serving symbolic functions rather than causal functions) and truly unbridgeable ones (affecting plot integrity), our approach treats both types as under the same umbrella.
they contradict established
Formally, consider fictional story containing set of propositions F ( pn) that are true in the fictional world of f (e.g., "Sherlock Holmes lived on Baker Street" is a statement that is true in the fictional world of Sherlock Holmes). We make use of the possible worlds theory from Lewis (1978), defining the notation isTrue(f, to de- note that the proposition & true in the fictional world of f and define the shorthand isTrueeff,F F) == isTrue f, P1) A... A isTrue pn). We can then define continuity error:
Definition 2.1 (Continuity Error) A proposition &e in story is associated with continuity error if the following inference rule holds:
isTruelf (p+3) = isTruelf,-ph
(1)
In other words, if using all the propositions in F except &ee we can conclude that the negation of &e is true in f. that means de is logically inconsistent with the rest of the story.
While the above definition formalizes many types of continuity errors, it assumes the contradictions are derived using the propositions explicitly stated in the story. However, reasoning for contradictions in stories often requires implicit knowledge such as one's world understanding and beliefs. We expand our definition to incorporate such implicit knowledge in Appendix $A.1, but informally, an expanded version of Definition 2.1 can be expressed as: If using all the propositions in F except d., along with a set of reader's 's belief statements (or community of readers') that are also non-vacuously true in f, one can derive that the negation of &e is true in f, then &e considered logically inconsistent with the resl of the story. We highlight this difference to emphasize that reasoning for plot holes in stories is not simply about checking for contradictions using rules and statements explicitly stated in text, but necessarily incorporates common sense and world knowledge.
3 Automatically Generating Plot Holes in Stories
Conceptually, FLAWEDFICTIONSMAKE inconsistency by selecting one of the propositions stated earlier in the story and negating it in the later parts. Our method, summarized in Figure 1, consists of 5-staged pipeline:
is story-editing approach that introduces
1. Three Act Structure Extraction. We start by dividing the story into the traditional three act structure Aristotle (1902), consisting of Act One (A1), where the main characters and
3

---

Published as a conference paper at COLM 2025
setting of the story are introduced, Act Two (A2), where the main conflict is developed, and Act Three (As), which builds to the climax and resolves the main conflict. This division aids to control where the original proposition is established in the story and when it gets contra- dicted in the later parts of our pipeline. We perform the three-act extraction of an original story/ through LLM prompting, and denote it (A1, A3} + ThreeActExtract()) (f). Note that f is the concatenation = A A2 A3 of the resulting three acts A2,
2. Proposition Extraction and Scoring. Next, we retrieve the set of proposi- tions that are stated in the first act A₁ of the the story through LLM prompting: PropExtract(Ai). Specifically, these propositions contain the informa- tion established about the characters (foreground) and the setting (background) of the story?. These propositions help us to control the specific continuity error that we wish to introduce. We also include a proposition scoring step, which determines how relevant is proposition & to the plot in the second and third acts using 4-point Likert scale: 69 <- PropScore(P: A,, A2, Az). We only retain the propositions that are moderately im- portant (sg E (2.3)) to avoid negating statements that lead to no change in the story, or changing a fundamental aspect which would render the final story completely nonsensical.
( 42.
3. Counterfactual Story Generation. We rewrite the story while negating an original propo-
Counterfact (1), A1, A2, Ag). Note that negating & does not just negate that single statement in the story, but may also lead to modifying other existing propositions maintain coherence and plausibility (e.g., when changing a character's 's nationality, their name might need to be changed).
sition & with LLM prompting (Qin el., 2019), A, 4,7.A, AA2
9,, AAg
4. Re-building Story ("Patching"). Now, given the original story f A A2 Ag and its counterfactual f+ = A, A, & Ag we create story with potential continuity error by concatenating A from the original story and the subsequent acts from the counterfactual: fpatch := A A² AA.. A.333
5. Filtering. As final step, we ensure that the patched story results an inherent story inconsistency. This includes removing obvious LLM prompting issues, such as cases where
A2 = A," or A3 = Ag or preemptively removing cases where there are too many changes (> 5) in the counterfactual, since an increasing number of LLM edits increases the probability of making counterfactual reasoning errors. We additionally run an extremely aided version of the task as quality filter: we prompt an LLM with fpalch, , specifying the modified lines in A2 and Ag and use the LLM as a judge of whether these lines introduce a continuity error. This much simpler problem* aids us in eliminating cases with errors during Step 3, where the newly introduced propositions might still be consistent with the original fact Φ. To improve reliability of filtering, we use self-consistency (Wang et al., 2023), only retaining the cases where the model predicts a continuity error in at least 4 out of the 5 completions. At the filtering step we also prompt the model to provide an explanation if it predicts that the modified lines introduce a continuity error, which is shown later to humans to verify if the stories actually have continuity error.
We use GPT-4o for all steps, except for counterfactual story generation where we quali- tatively found GPT-4turbo to perform significantly better. All the prompts used for our pipeline are provided in Appendix & A.10.1. While four out of five steps in our pipeline make use of LLMs, we do not claim that LLMs to be perfect at these tasks. Step 3, which requires counterfactual reasoning can in particular be difficult for LLMs with evidence in prior work (Huang et al., 2024). Hence, we follow our automatic generation process with human verification to curate a high quality benchmark.
We choose to extract the propositions only from the first act because we want to consider informa- tion that is established earlier in the story but later contradicted. Doing this help us controllably create plot holes in the later acts.
We select this patching method for simplicity. Note that other choices such as A A, A3 or
A p A2 Ag might also have been appropriate.
"This is much simpler problem because the model only needs to check the lines marked for contradiction, as opposed all the possible combinations of them.
4

---

Published as a conference paper at COLM 2025
6. Human Verification. Annotators are provided with stories and the proposed continuity errors from FLAWEDFICTIONSMAKER and are asked to rate if the continuity error is legiti- mate or not, with at least annotators per instance. Note that the annotators receive the final outputs after the Filtering step for verification. An example is considered legitimate only when the majority agrees about its legitimacy."
4 FLAWEDFICTIONS: Tasks, Metrics, and Dataset Statistics
We now discuss how the data generated by FLAWEDFICTIONSMAKE are used to create benchnark-FL.wWWDFFccC FTLAFIFFIS for reasoning about plot holes in stories across two tasks.
Classification Task. This represents a simpler version of the plot hole detection problem where the model is tasked to predict whether a continuity error exists in a story- - binary classification task. The positive examples (with continuity errors) come from data gen- erated using our method, while the negative examples use original unmodified stories All synthesized positive examples are verified by humans before being included in our benchmark.
Two-Way Localization Task. While the classification task provides some signal for the correctness in a model's assessment for continuity errors, we are ultimately interested in evaluating the specific continuity error predicted rather than merely its presence or absence. Given that evaluating open-ended natural language explanations remains challenging even when ground truths are available, we propose a two-way localization task as a proxy for continuity error explanation. In this task, the model must predict two sets of sentences in the story: SError, containing the sentences in the story that contain the error (i.e., that imply p where & is the original proposition), and SContr, containing sentences that entail Φ. We compare these predicted sets with the ground truth from WWWWEDFICIIONSMAKKKK KER to evaluate the validity of the model's predicted continuity error. Specifically, we define the Continuity Error Evaluation Full metric (CEEval-Full), which operates in two steps: first checking if the model correctly identifies whether an error exists, and if so, verifying if the predicted sentence sets contain at least one sentence from the ground truth 7, If the model incorrectly determines the existence of continuity error, it receives a score of
Dataset Composition and Statistics. To construct our benchmark's positive and negative examples, we scraped short story collections from Project Gutenberg using keywords such as fairytales and short stories. We retained only stories under 1200 words to reduce cognitive load on human annotators. From approximately 300 stories edited with FLAWEDFICTIONS. MAKER and verified by humans, we selected 207 stories (70% acceptance rate) positive examples. We then included an equal number original unmodified stories as negative examples, resulting in total of 414 examples in FLAWEDFICTIONS The final dataset has average length of 731 words and includes classical fairy tales, myths, legends, and historical fiction. See detailed statistics in Table 3, with dataset examples in SA.7.
FLAWEDFICTIONSLONG. Our preliminary experiments showed LLMs struggle with as- sessing plot holes story length increased (see $A.5.2 in Appendix). Consequently, we curated an extension of PLAWEDFICTIONS: FI FIAAEEDFICOONNSL
consisting of stories 1,200-4,000 words long: we selected stories from Fairy TaleQA (Xu et al., 2022) meeting this length criterion and processed them through PLAWEDFICTIONSMAKE
to generate positive examples. Due to increased cognitive load and annotation costs, only one-third of these longer stories were annotated by Prolific users, with the remainder annotated by this paper's lead author. Post-verification, we selected 97 stories as positive examples and 103 original stories as negative examples, totaling 200 examples in FLAWEDFICTIONSLLNG Unlike FLAWED FICTIONN,,, FLAWEDFICTIONSL.ON consists entirely of fairy tales and has an average length of 2703 words per story.
5 Annotators were hired via Prolific. Details about the annotation process are in Appendix $A.2. We discuss alternative approaches for negative examples in $A.6 in Appendix. We use this less strict metric because our primary concern is whether the model recognizes the error correctly, rather than whether it identifies all instances of the error (or contradicted proposition) in the story.

---

Published as a conference paper at COLM 2025
5 How Well do Frontier LLMs Perform on FLAWEDFICTIONS?
Experimental Setup. We evaluate different proprietary LLMs from OpenAl and Anthropic as well as open weights models Llama-3 (Van Der Maaten et al., 2024), Deepseek-RI Distilled (Guo et al., 2025), and Qwen-2.5 (Yang et al., 2024) series, which represent the most recent iterations available at the time of publication. For ol and 03-mini, we experiment with the three values of reasoning efforts parameter provided in the API low, medium, and high, which controls the amount of intermediate reasoning tokens generated before the final completion. Similarly, Anthropic API provides extended thinking mode for Claude 3.7 Sonnet model, which uses intermediate tokens to "think" before answering. We also consider another inference time scaling strategy, where we augment the plot hole detection model i.e. generator with verifier model (Cobbe et al., 2021) that validates the legitimacy of the plot hole detected by the generator. Our verifier is a Claude 3.5 Sonnet model prompted to perform the verification task. For more details on the experimental setup, prompts that we use, and other prompting methods that we evaluate such as few-shot and tothuhh (Co'T), refer to Appendix $A.4.
Baselines. To highlight the contextual nature of our problem, we use an entailment model that examines all ordered sentence pairs in story to detect contradictions, If no contra- dictory pairs are found, the baseline predicts the story lacks continuity errors; otherwise, the pair with highest contradiction confidence determines the error location. We employ DeBERTa-v3-large (He et al., 2021) fine-tuned on MNLI (Williams et al., 2018) (achieving 91% on MNLI dev) as our entailment model. We also consider random baseline and baseline that always predicts No continuity error found, with the latter achieving 50% on CEEval-Full due to our balanced dataset.
Benchmarking Human Performance. To establish a meaningful baseline against which to compare performance of various LLMs on FLAWEDFICTIONS, we estimated human performance by recruiting 9 undergraduate English majors who evaluated 50 samples from FLAWEDFICTIONS with three responses per sample. Further details about the study are provided in Appendix SA.2. It is important to recognize that this task is non-trivial for humans as requires a high amount of cognitive load due to the limited working memory, which has been shown to affect reading comprehension abilities in adults and children (Barreyro et al., 2025; Cain et al., 2004).
5.1 Results
Performance of different LLMs on FLAWEDFICTIONS is provided in Table 1a. On the classification task, we observe all open weights models like Llamm-3.1-708 and DeepSeek- R1-Qwen-32B to perform comparable to the random baseline. Similar trends were also observed for GPT--o-mini, GPT-4-turbo, and Claude 3.5 Haiku models. While other models like GPT-4o, 03-mini, ol demonstrate superior performance compared to the aforementioned models, it is only Claude 3.5 Sonnet, which matches human performance.
For the localization task, we again notice Claude 3.5 Sonnet to demonstrate superior perfor- mance CEEval-Full score of 0.67 (the ideal score is 1), and with verifier it matches human performance. Other than Claude 3.5 Sonnet, Claude 3.7 Sonnet with extended thinking, and ol, other models only show marginal improvements over the baseline that always outputs no error. The entailment baseline gets negligible score on CEEval-Full. This underscores the complex contextual nature of our task, which cannot be solved by merely finding two contradictory statements in the story. When viewed in isolation, two statements which in the broader context of the story are consistent with each other might appear to contradict each other. Consequently, the entailment baseline tends to trigger false positives and incorrectly localizes SEmrr and S
Results on PLAWEDFICTIONSLONG. We also conducted evaluations on FLAWEDFICTIONS- LONG, which contains stories approximately four times the length of those in FLAWEDFIC- TIONS on average. Table 1b shows that there is sharp drop in performance on FLAWEDFIC- TIONsLONG, with the best-performing model ie. ol obtaining CEEval-Full score of 0.53, only marginally outperforming the Always No Error baseline. Although FLAWEDFICTIONS- LONG has longer stories than FLAWEDFICTIONS, it still comprises stories with fewer than 4,000 words. This presents significant limitation, in realistic scenarios, plot holes are

---

Published as a conference paper at COLM 2025
Model
Accuracy
CEEval-Full
Model
Accuracy Task
CEEval-Full
Random Baseline
0.50
0.00
Always No Errur Baseline
0.50
0.50
Random Baseline
0.50
0.00
Entailment Baseline
Always Errur Baseline
0.51
Entailment Baseline
0.48
0.00
L.ama-3.3-70B Llama-3.1-88
0.57
0.38
0.50
0.10
11ama-3.3-70B
0.53
0.16
DeepSeek-Rl-Qwen--2
0.56
0.35
Llama-3.1-8B
0.48
0.02
Qwen2.5-328
PerpSee--Rl-ween-ce
0.52
Qwen2,5-32B
0.51
0.23
GPT-4o (with CoT)
0.64
0.58
GPT-4o-mini (with CoT)
0.53
0.32
GPT-4o
0.57
0.35
GPT-4-turbo (with CoT)
0.57
0.55
(with CoT)
0.56
0.42
off (Low)
GFT-4o-mini
0.51
(Medium)
0.70
0.65
(with CoT)
0.43
0.20
(High)
0.69
0.64
GPT-4-turbo
0.52
0.52
03-mini (Low)
0.55
0.52
(with CoT) olf (Medium)
0.54
0.53
(Medium)
0.62
0.53
0.61
0.53
(High)
0.63
0.47
03-minit (Low) (Medium) (High)
0.53
0.56
0.42
Claude 3.5 Haiku (with CoT)
0.57
0.46
0.45
0.07
Claude 3.5 Sonnet
0.76
0.67
(with Verifier)
0.74
0.68
Claude 3.5 Haiku Sonnet
0.48
0.37
Claude Sonnet
0.55
(with Extended Thinking)
0.73
0.66
(with Verifier)
0.60
0.50
Claude 3.7 Sonnet
0.49
0.29
Human Performance
0.76
0.68
(with Extended Thinking)*
0.54
0.37
(a) Performance comparison of different models the FLAWEDFICTIONS.
(b) Performance comparison of different models
FLAWEDFICTIONSLONG
Table 1: Performance comparison of different models on FLAWEDFICTIONS and AWWEEDFIC- TIONSLONG. Models trained to test-time compute for reasoning i.e. reasoning models are marked with +
more common for long-form stories like feature films or series of books and films, which typically contain substantially more than 4,000 words. Therefore, findings suggest that there exist substantial gaps in the capabilities of contemporary LLMs to reliably detect and evaluate consistency issues in long-form narratives.
Extra Test Time Compute Provides Minimal Gains. Interestingly, we found that extra test time compute would in most cases result in minimal improvement towards accurately detecting continuity errors. Table 1a shows that increasing the reasoning effort from low to high results in a drop in CEEval-Full score for both oll and o3-mini. For o3-mini this represents an increase from less than 1000 reasoning tokens on average to over 5000 tokens (roughly 5 times the number of tokens in the stories) for reasoning, yet results degraded performance. Similarly, the DeepSeek-R1 distilled models, which are also trained to utilize test time compute for reasoning, demonstrate suboptimal performance on the task, with only marginal improvements observed for Claude 3.7 Sonnet, where enabling extended thinking results in substantial improvements. Nevertheless, Claude 3.5 Sonnet, which utilizes additional test time compute for reasoning and generates approximately one-tenth the tokens of Claude 3.7 Sonnet with extended thinking, achieves marginally superior performance. Figure in the Appendix illustrates the relationship. These findings raise important questions regarding whether the absence of datasets similar to FLAWEDFICTIONS while training reasoning models explains the limited improvements observed, or whether inference time scaling is not adequate for solving problems like plot hole detection? frequently observed limitation of reasoning models is their tendency to persist on wrong hypothesis for potential plot hole during the reasoning process and continue with that chain of thought resulting an incorrect judgment. Since the space of possible hypotheses in our problem is at least quadratic in the number of sentences in the story, iterating through each of the hypothesis through intermediate generation becomes computationally prohibitive for extended narratives. We defer a more comprehensive investigation these questions for the future work.
the base 2.ve22.5-32B model. The sole exception is
What types of mistakes do LLMs make in assessing plot holes? We qualitativelyy analyzed the types of reasoning errors LLMs L.MSs-specfiicllly GPT-4o, Claude 3.5 Sonnet, and Claude 3.5 Sonnet with Verifier-make make on FLAWEDFICTIONS, We find that models often misinterpret
7

---

Published as a conference paper at COLM 2025
characters' motivations or behavior, e.g. character being deceptively nice bluffing is not necessarily a continuity enror. Another commonly observed mistake is models wrongly tracking and interpreting entities' states, e.g. miscounting the number of alive characters, or incorrectly assessing the passage of time, and interpreting these as plot holes. We also find that sometimes models fail to understand genre conventions, misinterpreting fantastical elements in fairy tales as logical inconsistencies. Finally, it is also common for models to misinterpret or overinterpret established rules or plot points in a story. For example, Claude 3.5 Sonnet incorrectly identifies a contradiction when a character tries multiple suits after stating they "will not try any suit more than once". We provide many examples for these errors in Appendix $A.8. In contrast, such reasoning errors were rare among humans, whose mistakes usually stem from overlooking details that may be attributed to humans' limited working memory. This is also evidenced by humans showing higher precision but lower recall than the best models on FLAWEDFICTIONS (see Table 5 in Appendix).
Measuring Logical Consistency in LLM Generated Narratives
A study by Mirowski et al. (2023) examining LLMs as screenplay co-writers identified that L.L.-generated narratives exhibited issues with maintaining consistency in plot's logic or characters' behaviors. While these observations were made based on participants' interviews, we propose quantitative evaluation framework for the phenomenon, Our setup consists of generating short stories using LLMs, which are subsequently evaluated for the existence of plot holes using our best model on FLAWEDFICTIONS i.e. Claude 3.5 Sonnet with Verifier. We define continuity error delection rate as the fraction of the generated stories for which the detection model identifies a continuity error.
Rather than employing unconditional and fully open-ended generations from the mod- els, we focus summarization and con- temporary adaptation tasks. In contempo- rary adaptation, the model is instructed to generate a modern retelling of a classical fairy tale i.e. transporting the setting of the story to modern times, while preserving similar themes, central conflict, and char- acters from the original story. We opted for conditional generation as they facilitate uti- lization of original human-authored stories as controls while checking for continuity er- rors. For summarization, we utilized 200 fairy tale stories from FairyTale QA dataset and prompt the models to write concise summaries of roughly 1000 words. For the contemporary adaptation task, we utilize the original stories (total of 207) included in FLAWEDFICTIONS We provide the prompts used for generation for both tasks in the Ap- pendix 10.10.3. Our focus on short stories for generations (i.e. less than 1200 words), stems from the suboptimal performance of even the nighest-pprformingg models on long stories.
Summarization
Contemporary Adaplation
Original Stories
MH
0.31
CPT-lo-mini
105
0.99
GPT-4o
Claude 3.5 Haiku
0.56
Claude 3.5 Sonnet
0.0
0.2
0.4
0.6
Delection
Figure 2: Continuity Error Detection Rate for stories generated using different LLMs for summarization and contemporary adaptation tasks.
Results. The continuity error rates for the two tasks are provided in Figure 2. We observe that generations from different LLMs demonstrate significant error rates relative to the original stories for both tasks. In case of summarization, lowest error rate was observed with GPT-4o, while still representing a 50% increase (0.31 to 0.45) in detected continuity errors when compared with original un-summarized stories. For contemporary adaptation the increase in error rates was even higher, with an almost 100% increase (0.14 to 0.27) in the best case for Claude 3.5 Haiku and a 278% (0.14 to 0.53) in the worst for GPT-4o-mini. For summarization, we identified that the models frequently omitted critical information in the summary that would render future events inconsistent with the rest of the narrative. E.g. in a story with sequence of events The dragon is on an year long sleep He is awakened by his brothers + He chases the prince, the summary from event where the dragon was awakened, and the sequence of events becomes: The dragon is
3.5 Haiku omitted the second
8

---

Published as a conference paper at COLM 2025
on an year long sleep + He chases the prince, creating clear contradiction. For contemporary adaptation, we identified issues where the models would fail to account for believability of certain plot elements in different settings. For instance, if the original fairytale had a horse talking to its owner, having the event play out identically in a modern setting without any reaction from any of the characters creates an inconsistency with the established setting of the story (impossible event). Additional examples are presented in Appendix $A.9.
7 Related Work
Narrative Understanding and Reasoning Tasks. Narrative understanding tasks can be categorized descriptive or interpretive. Descriptive tasks, which involve understanding explicitly stated plot elements, include question answering benchmarks (Narrative (Kočiský et al., 2018), FairyTaleQA (Xu al., 2022), and BookQA (Angelidis et al., 2019)), narrative summarization (Ouyang et al., 2017; Papalampidi et al., 2020; Kryscinski et al., 2022), and claim verification (Karpinska et al., 2024). Interpretive tasks require forming men- tal representation of story's worlds and utilzing those to infer their logical implications, such as selecting correct endings (Mostafazadeh et al., 2016), assessing causality (Roemmele et al., 2011), or generating counterfactuals (Qin et al., 2019). However, unlike PLAWEDFICTIONS, these datasets focus on very short stories that are roughly 4 to 5 sentences long. While, MuSR (Sprague et al., 2024) introduced multi-step reasoning over narratives involving tasks like solving murder mysteries, it uses synthetic stories with specific templates, whereas FLAWEDFICTIONS comprises edited versions of human-written stories with diverse nar- rative structures. For a review about narrative theory for narrative understanding, please refer to Piper et al. (2021).
Evaluating Quality of LLM Generated Stories. Studies show GPT--3-generated stories score highly on fluency and coherence compared to specifically tuned models and competitively with humans (Xie et al., 2023). However, human-written stories have been shown to exhibit more diverse narrative structures than the largely homogeneous LLM-generated stories (Tian et al., 2024). While GPT-4 stories surpass human-writter ones on the Psychological Depth Scale (Harel-Canada al., 2024), which quantifies the emotion, empathy, and engage- ment in stories, they score lower the Creativity Index (Lu et al., 2025), which measures linguistic creativity by searching for verbatim matches against web documents. None of these measure the logical and motivational consistency of narratives and there is evidence (Mirowski et al., 2023) that LLM authored stories can lack plot and character consistency.
Plot Holes and Impossible Worlds. Plot holes are inadvertent inconsistencies in story's logical and motivational texture (Ryan, 2009). Lewis (1978) defines such stories where the plot contradicts itself as impossible fictions, citing the example of contradicting locations of Watson's 's old war wound in Sherlock Holmes. Lewis (1978) proposes resolutions of truth in such fictions by considering revisions that remain close the original. Badura & Berto (2019) extends this theory with "impossible worlds" that can contain logical contradictions without rendering everything vacuously true to make sense of stories that deliberately defy logic (Priest, 1997). Plot holes have also been discussed in mathematics education contexts (Miežys, 2023).
Automatic Detection of Plot Holes. Davids (2022) introduced a symbolic approach using epistemic logic to identify plot holes, though the approach requires structured story events and is not flexible to operate on any story. Chadalapaka et al. (2023) generate synthetic data for plot hole detection by negating a randomly sampled statement in the story. However, this approach may not consistently generate plot holes, and to the best of our knowledge the authors do not perform human verification for their generated data.
8 Conclusion
In this work, we introduced FLAWEDFICTIONSMAKER an algorithm for automatically gener- ating continuity errors in stories, which we utilized to curate a benchmark FLAWEDFICTIONS for evaluating LLMs' capabilities to reason about plot holes in stories. Our experiments reveal that frontier L.LSs struggle to accurately solve the task and inference time scaling pro-

---

Published as a conference paper at COLM 2025
vides minimal performance improvements. Finally, employing the best-performing model on FLAWEDFICTIONS, we analyzed LLM generated stories and summaries, and found them to contain significantly higher continuity error rates compared to human authored stories. Overall, our work demonstrates that despite significant progress in reasoning capabilities of LLMs, substantial gaps remain in their deeper narrative understanding capabilities.
offers general approach for generating continuity errors, future work could explore methods providing finer control over the types and complexity of introduced plot holes. Additional research might focus on designing new post-training strategies that can enhance model performance on FLAWEDFICTIONS. Another promising direction would be to investigate whether using PLAWEDFICTIONSMAKE to generate large amounts of synthetic training data could enhance LLMs' reasoning capabilities more broadly. Future work can also consider plot deficiencies other than plot holes, like plot conveniences or coincidences (termed cheap plot tricks Ryan (2009)) or apply similar approaches to non- fictional contexts like fact-checking, misinformation detection, and education.
While FLAWEDFICTIONSMAKK
Acknowledgments
We thank Maria Antoniak for her feedback on the initial project idea. We would also like to thank Alexander Spangher for his detailed and helpful comments our draft. Finally, special thanks to all the Prolific annotators and UW undergraduates who participated in our annotation and evaluation studies, and whose hard work made the FLAWEDFICTIONS benchmark possible.
References
Jan Alber. Logical Contradictions, Possible Worlds Theory, and the Embodied Mind, pp- 157-176. University of Nebraska Press, 2019. ISBN 9780803294998.. URL http::/www.jstttt org/ stable/j.cttvxxgge.. 11.
Stefanos Angelidis, Lea Frermann, Diego Marcheggiani, Roi Blanco, and Lluis Mårquez. Book QA: Stories of challenges and opportunities. In Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen (eds.), Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 78-85, Hong Kong, China, November URL
2019. Association for Computational Linguistics. doi: (0.18633/v1/D19-55 https://aclantholoogy org//19-8///11911 httpss//ccaanttoooooyyr/////1-1/
Aristotle. Poetics. Macmillan, New York, 1902.
Christopher Badura and Francesco Berto. Truth in fiction, impossible worlds, and belief revision. Australasian Journal of Philosophy, 17(1) 2019. doi: 10.1080 2018.1435698. URL httpp:://dol 0r0/ 00000.0008000004000 2018. 1435698. 14388843
Juan Barreyro, Sofía S. Ortíz, and Jesica Formoso. The role monitoring, prior knowledge, and working memory in the comprehension of expository texts in university students. Psicología Educativa, 31(1):45-5, 2025. doi: 10.5093 00.5093/pee22025.. psed2025a6.
Jerome Bruner. The narrative construction of reality. Critical Inquiry, 18(1):1-21, 1991. doi: 10.1086/ 4486199.
Kate Cain, Jane Oakhill, and Peter Bryant. Children's reading comprehension ability: Concurrent prediction by working memory, verbal ability, and component skills. Journal of Educational Psychology, 96(1):31-42, 2004. ISSN 0022-0663. doi: 10.1037 10.1037/0002 /0022-0663.96. 1.31.
Viswanath Chadalapaka, Derek Nguyen, JoonWon Choi, Shaunak Joshi, and Mohammad Rostami. Low-shot learmingg for fictional claim verification. arXiv preprint arXiv:2304.00766, 2023.
10

---

Published as a conference paper at COLM 2025
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv,, abs/2110.14168, 2021.
Aron Davids. Identifying plot holes narrative stories by simulating events, July 2022.
URL http:: http://eesay..uunttte.eeee www.wtw..we.........w....... hwwpeeetteen/9999... n1/91967/.
Richard J. Gerrig. Experiencing Narrative Worlds: On the Psychological Activities of Reading.
Yale University Press, New Haven, 1993.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. ariiv preprini ArXiv:2501. 110.001..99900 12948, 2025.
Fabrice Y Harel-Canada, Hanyu Zhou, Sreya Muppalla, Zeynep Senahan Yildiz, Miryung Kim, Amit Sahai, and Nanyun Peng. Measuring psychological depth in language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17162-17196, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ org/ ennne- 9531.
w1/2024.emnlp-main..9 953. URL https: /ettps://aclle http///aaaaahhhlll/oo
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-
enhanced bert with disentangled attention. In International Conference on Learning Repre- sentations, 2021. URL htttps::/opperrre (tt.:tttp...//gpeeeeeeee/ee/e/et wtt.://wewtttooommmmm///aaaaauue
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Yinya Huang, Ruixin Hong, Hongming Zhang, Wei Shao, Zhicheng Yang, Dong Yu, Chang- shui Zhang, Xiaodan Liang, and Linqi Song. CLOMO: Counterfactual logical modifi- cation with large language models. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), PP. 11012-11034, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 /11 acl- 5933. URL acl https://aclanttologg.orr//////// aclooong..l11nnc55m 593/.
Mohamad Yaser Jaradeh, Markus Stocker, and Sören Auer. The sciqa scientific question answering benchmark for scholarly knowledge. Scientific Reports, 13(1):7336, 2023.
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit lyyer. One thousand and one pairs: A "novel" challenge for long-context language models. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17048-17085, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. D0.18633/v//000 emnlp-main.945 URL htt.t: httpss://aclantthlles/// wrw./024....... emnlp-main.ge 998//
David Comer Kidd and Emanuele Castano. Reading literary fiction improves theory of mind. Science, 342(6156)) 2013. doi: 00.1126//ciencee12303
Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. FANToM: A benchmark for stress-testing machine theory of mind in interactions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14397-14413, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/ org/2023 emnlp-main. 890/.
2023.emnlp-main.c 8900. URL https: http://///laaaaahlaaaaao
Walter Kintsch. Comprehension: A Paradigm for Cognition. Cambridge University Press, 1998.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018. doi: 10.1162/ tacl 00023. URL httts:://aclantholo.g wwwpoloooyyoog/////0 arg/Q18-1102)
11

---

Published as a conference paper at COLM 2025
Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. BOOKSUM: A collection of datasets for long-form narrative summarization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp.
Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/ (6633v1l///// /2022. findings-emnlp.455 .888. URL https:://cllantoology org/202.. Findings-emnlp 488/.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T ulu Pushing frontiers in open language model post-training. ar arv preprint ar dfXo::2411.1512, 2024.
David Lewis. Truth in fiction. American Philosophiccl Quarterly, 15(1):37-45 1978. ISSN 00030481. URL http::/ww.. jstor. http://ssoorooossstleeee/eeee/ee wtgg/stablee288096999.
Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Chandu, Nouha Dziri, and Yejin Choi. Al as humanity's s salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text against web text. In The Thirteenth International Conference on Learning Representations, 2025. URL htttps://opereevieeee
net/ wtwpeewwwwwwwwweeee//e/ htte///oummmmi1100m0aaa
Raymond A. Mar and Keith Oatley. The function of fiction is the abstraction and simulation of social experience. Perspectives on Psychological Science, 3(3):173-192, 2008. doi: 10.1111/j. 1745-6994422088.0077...
MasterClass. How fix plot holes your story, 2021. URL https://www www.:.asteerlass...o com/
Last updated: Dec 7, 2021.
Robert McKee. Story:
Substance, Structure, Style and the Principles of Screenwriting. Regan- Books, New York, 1st edition, 1997. ISBN 0-06-039168-5.
Vytautas Miežys. Cheap plot tricks and plot holes in mathematical stories. Educational Studies in Mathematics, 113(2))271-285,, Jun 2023. ISSN 0013-1954.
Piotr Mirowski, Kory W Mathewson, Jaylen Pittman, and Richard Evans. Co-writing screen- plays and theatre scripts with language models: Evaluation by industry professionals. In Proceedings of the 2023 CHI conference on human factors in computing systems,, pp. 1-34, 2023.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839- 849, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/ 11/11111111118 URL https: http://ss/aaaaaooooooooggg////// org/N16-18988/
Jessica Ouyang, Serina Chang, and Kathy McKeown. Crowd-sourced iterative annotation for narrative summarization corpora. In Mirella Lapata, Phil Blunsom, and Alexander Koller (eds.), Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 46-51, Valencia, Spain, April 2017. Association for Computational Linguistics. URL httpt:t//aclaaaaa http://aaaaannnoo....ooo -00
Pinelopi Papalampidi, Frank Keller, Lea Frermann, and Mirella Lapata. Screenplay summa- rization using latent narrative structure. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Com- pulational Linguistics, pp. 1920-1933, Online, July 2020. Association for Computational URL https://aclanthhoogg. arg/2020
Linguistics. doi: 10.18653/ v1 22020.acl-main acl-main. 174/.
Andrew Piper, Richard Jean So, and David Bamman. Narrative theory for computational narrative understanding. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural
12

---

Published as a conference paper at COLM 2025
Language Processing, Pp. 298-311, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: http//111222.emmppppp URL https:://allatttollalaa///
org/2021. eemnlp-maing 26/.
Graham Priest. Sylvan's box: A short story and ten morals. Notre Dame Journal of Formal Logic, 3844)5373-882 1997.
Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. Counterfactual story reasoning and generation. In Conference on Empirical Methods in Natural Language Processing, 2019. URL https://api. www.oiissmanicccoo...... .rr//
C0rPuSID:2025422444
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In AAAI Spring Sym- postum on Logical Formalizations of Commonsenss Reasoming, Stanford University, March 2011. URL http://ict. wsc. http:////ubbccccccccc/////0/2/00 bttp:///aaa..ataauooooooeeeeetea eexeaAl http:///wh...ttooooooooooooooona
pdf.
Marie-Laure Ryan. Cheap plot tricks, plot holes, and narrative design. Narrative, 17(1):56-75, 2009.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4463-4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/ 11/53//11/1911455 URL https: http::///aclathholooy// MDD119-1454/
Catia Shattuck. 6 types of plot holes and how to catch them, 08 2024, URL https:// mybookcave.com com/ wwtharpostto- syypes-of-plothhooee -od-how-
to- to-chhhtttttchttttheeee them/.
Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the limits of chain-of-thouge with multistep soft reasoning. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpp::/ /twps//eveeeeeew..eommomeo net/forummid jenyYQzue1.
Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, and Nanyun Peng. Are large language models capable of generating human-level narratives? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17659-17681, Miami, Florida, USA, November 2024. Association for Computational 978. URL https::/aclaatthlloo org/
Linguistics. doi: 2024. 11c44emmlp-minn 1
Laurens Van Der Maaten et al. The llama 3 herd of models. Xiv preprint arXiv:2407.21783 Jul 2024. v3, last revised 23 Nov 2024.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bow- man. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackhoxNLPP Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: D018653/v1/W18.. https://aclanthologyy wrrp://tthooooooo.o////1-/-- 5446.
URL
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. In Advances in Neural Information Processing Systems, 2019.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL htR.https:
net wwt.:e/wwww.oooom/dd/Pa. 1NIMMM..
13

---

Published as a conference paper at COLM 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Adoances in neural information processing systemss, 35:24824-24837, 2022.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), PP. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. URL https: http://acaaanhoooo.ooooo org/N18-1101/
doi:
Zhuohan Xie, Trevor Cohn, and Jey Han Lau. The next chapter: A study of large lan- guage models in storytelling. C. Maria Keet, Hung-Yi Lee, and Sina Zarrieß (eds.), Proceedings of the 16th International Natural Language Generation Conference, pp. 323-351, Prague, Czechia, September 2023. Association for Computational Linguistics. doi: 10.18653/v11 00/553/v12000/ //t////2132n/-mmaaiiii mai.anni..mm.......... URL https: / hacas///ollaaaoooogooooooo org/ org./023. anlg-main.225 23/.
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer. Fantastic questions and where to find them: FairytaleQA an authentic dataset for narrative comprehension. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 447-460, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v /v1 w112022.acl-lom///
URL /tttps://aclantthttt
2ttg/2222..000 acl- 1ong..oo..... 34/.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. Xiv preprint ar 2r11:22422.15111 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Lisa Zunshine. Why We Read Fiction:
Theory of Mind and the Novel. Theory and Interpretation of Narrative. Ohio State University Press, Columbus, 2006. ISBN 978-0-8142-1000
14

---

Published as a conference paper at COLM 2025
A Appendix
Table of Contents
Introduction
2 Defining Plot Holes: Continuity Errors
3 Automatically Generating Plot Holes in Stories
FLAWEDFICTIONS: Tasks, Metrics, and Dataset Statistics
5 How Well do Frontier LLMs Perform FLAWEDFICTIONS?
5.1
Results
Measuring Logical Consistency in LLM Generated Narratives
7 Related Work
8 Conclusion
Appendix
A.1 A More Formal Treatment of Continuity Errors
16
A.2 Human Annotation and Benchmarking
18
A.3
Patasst
19
A.4 More Details on Experimental Setup
19
A.5 Additional Results.
20
A.5.1
Detailed Results FLAWEDFICTIONS and FLAWEDFICTIONSLLONG
20
A.5.2
Factors Effecting Performance on FLAWEDFICTIONS
24
A.5.3
Task Subjectivity.
24
A.6
Other Considerations for Negative Examples.
24
A.7
FLAWEDFICTIONS Examples.. Example........
25
A.8
Examples of Reasoning Errors on FLAWEDFICTIONS
30
A.9
Examples of Continuity Errors in LLM Generations
39
A.9.1 Summarization
39
A.9.2
Contemporary Adaptation
43
.............
A.10 Prompts ................................ ................................
48
A.10.1 FLAWEDPICTIOSSMAKRR Prompts
48
A.10.2 Evaluation Prompts
53
A.10.3 Generation Prompts
53
A.11 Human Benchmark Study Document..
59
15

---

Published as a conference paper at COLM 2025
A.1 A More Formal Treatment of Continuity Errors
We discussed in $2 that the Definition 2.1 fails to account for implicit knowledge such as our world understanding and beliefs that are often essential to reason about contradictions in stories. We utilize the Possible Worlds theory from Lewis (1978) to extend our definition. The core contribution of Lewis's theory is to assess truthfulness of the statements that are never stated in the text of the narrative. E.g. can we say that Sherlock lived closer to Paddington Station than Waterloo Station? While using a map of real world London one can check Baker Street being closer to I'addington Station, story's s text never explicitly states this. However, we can still assign truth to this statement since we do not have any special reason to believe that geography of London in Sherlock Holmes is remarkably different from the real world. To decide a proposition p, which is true in the belief world of the reader (or community of readers) is also true in story f f-isTrue(ff, p)-, without explicitly being stated in f. Lewis (1978) uses the notion of counterfactuals. Specifically, proposition is non-vacuously true in f, when some world where f is told as fact and p is true, is closer to the belief world of the reader W3, than any world where f is told as fact and is not true. Hence, while we can consider a world where Sherlock Holmes is told as fact and London is arranged very different from the real world such that Baker Street is closer to the Waterloo Station than Paddington Station, that world will be further away from the belief world of the reader compared to a world that preserves the geography of London.
We now utilize Lewis's theory to extend our definition of continuity errors incorporate implicit world knowledge and beliefs. We first define the operator, TF P( + P))) where for any F - Φ, TF(F) = {p E B sim(WF,p W/) < the belief world of the reader and W5,p represent any closest world to W,, where both F and are true. Here, c denotes the set of all possible propositions, P((b) is its power set, B C & is the set of true propositions in the belief world, and sim is similarity measure between possible worlds. In other words, TF (FFF operator returns the set of propositions form the belief world of the reader that can also be established to be non-vacuously in true in story f with propositions F. Using this we can rework our definition of a continuity error:
W,]} where W, is
Definition A.1 (Continuity Error with Beliefs Incorporated) A proposition &e in story is associated with a continuity error when:
isTrue (de3) A (f, \. (fe)) = isTrruef, -Re)
(2)
In other words, if using all the propositions F except Pe, as well as the propositions from the belief world that are non-vace non-vacuuusly true in fs, we can conclude that the negation of &e is true, that means de represents a continuity error in f.
According the possible worlds theory, stories with such logical contradictions lead to impossible fictions, where there exists no possible world where the story is told as fact, ie. W, () In principle, for such impossible story, any statement p is vacuously true. However, such treatment can be too harsh especially when the logical contradictions are accidental and not blatantly renders the plot useless (e.g. we can still make sense of story even if wound placement on a character has changed without notice). There are formalizations to non-vacuously evaluate truth statements in impossible worlds in Lewis (1978) and follow-up work Alber (2019); Badura & Berto (2019), however that falls out of the scope of this work. Our primary concern here is understanding if LLMs can reason when story represents worlds that are impossible.
Below we provide detailed example to further clarify the definition:
Consider this story:
Jack is drinking hot tea in a cup
Jack gives his tea to Jill
Jill drinks the cold tea from Jack's cup
Here, ff is a story fl where de never stated.
16

---

Published as a conference paper at COLM 2025
From the story extract propositions: 01 = "Jack drinks tea", d2 ="tea is hot", 43 "Jack gives tea to Jill", 41 = "fill drinks tea", and Φ = "tea is cold". Note that removing d5 from proposition set does not allow to conclude -05 ("tea is not cold") from the remaining propositions alone, since "hot" does not logically entail "not cold" without additional assumptions. This precisely why we need the extended definition with reader beliefs. A reader's s commonsense knowledge includes the belief that "if tea is hot, then tea is not cold" (f2 + -s)) When we include this reader belief in our reasoning, the contradiction: 44 + -45 42) F -bs (by modus ponens) This shows that Øs is logically inconsistent with the rest of the story when combined with reasonable reader beliefs, making it a continuity error according to our extended definition. The key insight is that continuity errors in fiction cannot be detected purely from explicit textual propositions they require the integration of implicit world knowledge that readers bring to their understanding of the story.
can derive
Type of Plot Hole
Film / Story
Plot Hole Description
Harmless
Source
Notes
or Unbridge- able
Continuity Error
Sherlock Holmes by Arthur Conan Doyle
When are first introduced Watson A study in pink, he is described as having injury in his left arm, but the very next story sign of Four contradicts this where his war wound is on his knee. In the film Kane dies alone, but group of reporters trying to discover meaning his dyning words. he died alone who heard the words Rosebud?
Harmless
Lewis (1978)
Citizen Kane (1941)
Harmless
Ryan (2009)
Example of incorpo- rating real world beliefs to
about plot
holes
"when
people die
alone that
means no
one could
hear their
last words"
propo-
sition we
know to
be true
from
common-
sense and
not something stated in the story
Continued on next page...
17

---

Published as a conference paper at COLM 2025
Table continued from previous page
Type of Plot Hole
Film Story
Plot Hole Description
Harmless
Source
Notes
or Unbridge- able
Out of Character Behavior
Little Red Riding Hood by Brothers Grimm
A mother tells her daughter, Little Red Riding Hood, to go through the forest and to bring some food to her ailing grandmother. She warns the little girl not to talk to strangers. On her way, Little Red Riding Hood meets a hungry wolf and tells him about her mission. The wolf runs to the grandmother's house, eats her, and takes her place in bed. When Little Red Riding Hood arrives she mistakes the wolf for the grandmother. After conversation during which he pretends to be the grandmother, the wolf jumps out of the bed and eats Little Red Riding Hood. Why did he not just eat her when they met for the first time?
Unbridgeable Ryan
(2009)
Factual Error
Titanic (1997)
In Titanic, Jack mentions fishing at Lake Wissota which man-made lake created 1917 five years later when titanic sank In The Dark Knight Rises (2012), a full team of police members was trapped underground for months, yet they all walk out cleanshaven and well-dressed.
Harmless
Impossible Event
Dark Knight Rises (2012)
Harmless
Davids (2022)
Unresolved Storylines
Game of Thrones (2011-2019)
Many plot lines in the tv show were never resolved like the mysterious character of Quaithe who makes multiple prophecies that never end up playing out in the story.
Harmless
Table 2: Examples of different types of Plot Holes
A.2 Human Annotation and Benchmarking
The annotators were hired from the Prolific platform with screening conditions that the candidates have English as their primary language, are residents of UK, US, or Canada, have at least an undergraduate degree, and face no literary difficulties. We also conducted screening test where candidates were given small set of examples from the task for which the ground truths were already verified by the authors and selected candidates for the actual study who performed well on this screening test. The selected examples had 50% samples that were incorrectly assessed by ChatGPT and we made use of this to find candidates who were
Verifying stories from FLAWEDFICTIONSMAKCE
18

---

Published as a conference paper at COLM 2025
potententially using LLMs for annotations. We also checked the average amount of time it took for participants to complete the pilot study, and didn't 't consider those who solved the task too quickly, with the risk of them potentially using LLMs. We finally ended up recruiting 19 annotators, who were paid $12 per hour for their work with extra 20-30% bonuses each time they annotated more than 10 stories. Estimated time per annotation for each example was 5 minutes and we ended up paying total of $6500 the annotators. We got roughly 350 stories annotated, and got least 3 annotations for each story. An example of our annotation framework built using Argilla is provided in Figure 3.
hightrt!
UnERgE accoNNE pecome MANE COMESSEEEETE Tr
Thy
It Revaaeene vo sostra medecte cado catioieer
valeer de nnd in mone to nas
Fms unseaal moder
Figure 3: An example of our human annotation interface for verifying outputs of FLAWED- FICTIONSMAKER.
Benchmarking Human Performance. We recruited undergraduates with English major and present them with the same task of plot hole detection and the same specifications and instructions as we do for different LLMs. We sampled 50 examples from our dataset and obtained 3 responses for each instance. The estimated time for solving each task was 15 minutes (approximated by the first author) and participants were compensated $5 for providing response for each story, thereby providing $20 per hour for their work. To encourage participants to give their best efforts towards solving the task, we provide a 30% bonus for solving the task with higher accuracy (> 70% accuracy on the classification task). We paid total of $944.60 to the participants. An example of the interface has been provided in Figure 4. The complete study document shared with the participants is included the end of this paper $A.11.
For both of the annotation exercises, we got an Institutional Review Board (IRB) exemption from the Human Subjects Division which determined our studies as human subjects research that qualifies for exempt status (Category 101) and the IRB IDs were STUDY0002192 and STUDY00022395 for the verification and benchmarking studies respectively.
A.3 Dataset Statistics.
Descriptive statisties of lengths of the stories included in FLAWEDFICTIONS and FLAWEDFICTIONS- LONG are provided in Tables 3 and 4 respectively.
A.4 More Details Experimental Setup
For all experiments, we use temperature of 0.5 and specify maximum of 4096 tokens for all models except the reasoning models ol, 03-mini, and Claude 3.7 Sonnet with extended thinking, for which we use a maximum of 8192 tokens. All experiments with open weights models were run on single A40 and 1,0 instances. We experiment with three types of prompting strategies, the vanilla case where we describe the task and output format the model and ask generate the answer, few-shot case where we provide everything from the vanilla case plus two examples (one positive and one negative)
10 hthttps://gite httpss////ggtuuuuu.o/////ggl//// www.arooo//ocoeccc lariaarrriiliiiooiilrllllll
19

---

Published as a conference paper at COLM 2025
66. Did you findl contenity in tnwwarey
carice Sinel
Plys foord vvees non pow of ever
Own emrutier, ber the
reaide mng Doattameet conpact aut suneer
Detambiee
inimes
they
tor wieent aus ladken von somaur hor shoudde secent
ontegee weltiter dng
shee tet teni aed
mults sentrces seperts them seriictees
cnttore
pa my Loaul timsle ooee care wiy aocu
who ohe undprameed
thew
found avanss asse pos serticar contaciinn
remaiic alerc Anganid lid by unbtrr owarment ant and cocok, the dog hasddie Sinca thacchal sayy onn betoer cas
thew you pinte the
iimted to the hord vad breiigg caicf anthomen anding.
Obeore
Game
Ealent
Figure 4: An example of the interface used for benchmarking human performance on FLAWEDFICTIONS.
Statistic
Value
Count
414
Mean
731.81
Standard Deviation
225.51
Minimum
132
25th Percentile
569.25
Median
754
75th Percentile
923.50
Maximum
1236
Table 3: Descriptive statistics story lengths (in words) in our FLAWEDFICTIONS.
of the task, and finally chain-of-though prompting which builds upon the vanilla case by asking model to first create scratchpad analyzing the story. The prompts that we use for evaluation are provided in SA.10.2.
Verifier We augment the plot hole detection model iee generator with a verifier model (Cobbe et al., 2021) that validates the plot hole detected by the generator is legitimate. if is deemed illegitimate, we sample from the generator again, till either the verifier agrees generator answers by saying No continuity error detected. The maximum number of samples from the generator capped 5. For the verifier
Claude 3.5 Sonnet model prompted to test the validity of proposed plot hole. Due to increased cost with using verifier we only report results when Claude 3.5 Sonnet generator is augmented with the verifier.
A.5
Additional Results.
A.5.1
Detailed Results on FLAWEDFICTIONS and FLAWEDFICTIONSLONW
We provide expanded versions of the results the main paper (Tables la, 1b) containing multiple evaluation metrics and prompting methods in Tables and 6. CEEval-Pos metric is defined by only considering positive examples ie. the ones with continuity error during the localization task. Figure 5 plots performance of different models vs the average number of completion tokens generated by the model solve the task, which we use as proxy for inference time compute.
20

---

Published as a conference paper at COLM 2025
Statistic
Value
Count
200
Mean
2703.09
Standard Deviation
805.16
Minimum
1246
25th Percentile
1965
Median
2575
75th Percentile
3350
Maximum
3999
Table 4: Descriptive statistics of story lengths (in words) in our FLAWEDFICTIOSSLON.
Effect of different prompting methods. We find few-shot prompting often leads worse performance compared to vanilla prompting and tfain-of-thought, with the exceptions on Claude 3.5 Haiku and GPT-4-turbo, where it helps slightly. Chain-of-thoi thothhtght is effective for CPT-4o and GPT-4o-mini, but offers little to no improvements for other models.
Remoeing Mairis
GPT- Mockk
Cludk
Rewasing Middi
07
SiraaaalCeeOveee Dir
300
2000
4.0 Cumplaation
Completion
(a) CEEval-Full average number of com- pletion tokens on FLAWEDFICTIONS.
(b) CEEval-Full vs average number of
pletion tokens on FLAWEDFICTIONSLONG
Media
Modil
Besssiing
022
045
Quverit
1000
7000
Number of Complltion
Completion
(c) Accuracy score average number of comple- tion tokens on FLAWEDFICTIONS.
(d) Acurracy score vs average number of comple- tion tokens on FLAWEDFICTIONSLONG
Figure 5: Effect of inference time compute represented using the average number of comple- tion tokens on the performance on FLAWEDFICTIONS and FLAWEDFICTIONSLONNG.
21

---

Published as a conference paper at COLM 2025
Classification Task Recall
Localization Task
Model
Accuracy
Precision
F1-score
CEEval-Pos
CEEval-Full
Random Baseline
0.50
0.50
0.50
0.50
0.00
0.00
Always No Error Baseline
0.50
0.0
0.0
0.0
0.0
0.50
Entailment Baseline
0.53
0.52
0.68
0.02
0.04
Llama-3.3-700 3.3-70B
0.57
0.56
0.73
0.63
0.34
0.38
1lama-3.1-70B
0.56
0.54
0.76
0.63
0.26
0.31
Llama-3.1-8B
0.50
0.50
0.99
0.66
0.18
0.10
DeepSeek-R1
0.56
0.54
0.69
0.61
0.28
0.35
0.58
0.57
0.65
0.61
0,15
0.33
Qwen2.5-32B
0.53
0.53
0.50
0.51
0.08
0.31
GPT-4o
0.60
0.62
0.51
0.56
0.34
0.51
(with Few-Shot)
0.57
0.55
0.80
0.65
0.43
0.38
(with CoT)
0.64
0.72
0.45
0.56
0.33
0.58
GPT-4o-mini
0.48
0.48
0.62
0.54
0.09
0.21
(with Few-Shot)
0.50
0.50
0.64
0.13
0.11
(with CoT)
0.53
0.53
0.52
0.52
0.10
0.32
GPT-4-turbo
0.55
0.86
0.12
0.21
0.08
0.53
(with Few-Shot)
0.60
0.78
0.27
0.40
0.18
0.55
(with CoT)
0.57
0.90
0.17
0.28
0.13
0.55
o1 (Low) (Medium)
0.71
0.93
0.44
0.60
0.34
0.65
0.70
0.96
0.42
0.58
0.32
0.65
(High)
0.69
0.94
0.40
0.56
0.31
0.64
03-minif (Low)
0.55
0.71
0.12
0.52
(Medium)
0.62
0.75
0.37
0.50
0.19
0.53
(High)
0.63
0.65
0.57
0.61
0.25
0.47
Claude 3.5 Haiku (with Few-Shot)
0.55
0.59
0.30
0.40
0.12
0.46
0.57
0.72
0.23
0.35
0.11
0.51
(with CoT)
0.57
0.64
0.35
0.45
0.13
0.46
Claude 3.5 Sonnet
0.76
0.73
0.83
0.78
0.64
0.67
(with Few-Shot)
0.58
0.54
0.96
0.69
0.66
0.42
(with CoT)
0.71
0.66
0.64
0.59
(with Verifier)
0.74
0.81
0.63
0.71
0.51
0.68
Claude 3.7 Sonnet
0.66
0.61
0.88
0.72
0.67
0.55
(with Extended Thinking)
0.73
0.68
0.87
0.76
0.72
0.66
Human Performance
0.76
0.84
0.64
0.73
0.48
0.68
Table 5: Performance comparison of different models the FLAWEDFICTIONS. Models
trained to test-time compute for reasoning i.e. reasoning models are marked with 1,
22

---

Published as a conference paper at COLM 2025
Classification Task Recall
Localization Task
Model
Accuracy
Precision
Fl-score
CEEval-Pos
CEEval-Full
Random Baseline
0.50
0.50
0.50
0.50
0.00
0.00
Always No Error Baseline
0.51
0.0
0.0
0.0
0.0
0.51
Entailment Baseline
0.48
0.48
1.00
0.65
0.00
0.00
Llama-3.3-70B
0.53
0.50
0.88
0.64
0.13
0.16
Llama-3.1-700
0.53
0.51
0.88
0.64
0.06
0.13
Llama-3.1-8B
0.48
0.48
0.99
0.65
0.04
0.02
DeepSSeek-Rl-Qwen
0.52
0.51
0,56
0.53
0.03
0.27
0.50
0.48
0.42
0.45
0.0
0.3
Qwen2.5-32B
0.51
0.49
0.62
0.55
0.03
0.23
GPT-4o
0.57
0.54
0.72
0.62
0.27
0.35
(with CoT)
0.56
0.55
0.48
0.51
0.21
0.42
GPT-4o-mini
0.51
0.50
0.93
0.65
0.03
0.08
(with CoT)
0.43
0.43
0.51
0.46
0.05
0.20
GPT-4-turbo
0.52
1.00
0.01
0.02
0.00
0.52
(with CoT)
0.54
1.00
0.06
0.12
0.03
0.53
ol (Medium)
0.61
0.76
0.29
0.42
0.12
0.53
03-mini (Low)
0.53
0.55
0.16
0.25
0.02
0.46
(Medium)
0.56
0.57
0.37
0.45
0.08
0.42
(High)
0.45
0.46
0.59
0.06
0.07
Claude 3.5 Haiku
0.48
0.44
0.25
0.32
0.02
0.37
Claude 3.5 Sonnet (with Verifier)
0.56
0.53
0.77
0.63
0.33
0.35
0.60
0.60
0.49
0.54
0.30
0.50
Claude 3.7 Sonnet (with Extended Thinking)
0.49
0.49
0.63
0.47
0.29
0.54
0.52
0.81
0.63
0.46
0.37
Table 6: Performance comparison of different models on FLAWEDFICTIONSLONG.
23

---

Published as a conference paper at COLM 2025
A.5.2 Factors Effecting Performance FLAWEDFICTIONS
We investigate if length of story has effect on how accurately do different LLMs detect continuity errors in them by measuring correlation 11 between a story's length (measured by counting number of words) and the CEEval-Full score that story. We find negative correlation coefficients for all the models that we test and while the correlation values are low -0.1 to -0.2, for 13 out of 14 models the correlation observed statistically significant (p-value 0.05). Refer to the Table 7 for the exact values.
Model
Correlation
p-value
-lama-3.1-8B-nstruue
-0.134*
6.21 x 10 10
Llama-331-70B-nnttrutt
-0.154*
1.64 x 10
-amma-3.-770--nsstruut
-0.14**
2.57 10
DeepSeek-RI-Qwen--148
-0.192*
7.77 x 10
DeppSek-RI--Wwen-221
-0.116*
1.75 x 10
Qwen-2.5-14B
-0.127*
9.39 x 10-3
GPT-4o-mini
-0.029
0.551
GPT-4o
-0196**
5.70 x 10
Claude-3.5-Sonnet
-01722*
4.24 x 10 4
Claude-3.5-Sonnet with verifier
-0.163*
8.42 x 10
Claude-3.5-Haiku
-0.156*
1.40 10-3
Claude-3.7-Sonnet
-0.122*
4.36 x 10
o1
-0.104*
2.48 X 10
o3-mini
-0.174*
5.82 X 10-10
Table 7: Point-Biserial Correlation between number of words in a story and the correspond- ing CEEval-Full scores by different LLMs.
Model
Total Annotated
Total Accepted
Acceptance Rate
-o---o-mini
54
2
0.04
GPT-4o
37
3
0.08
Claude 3.5 Sonnet
37
8
0.22
03-mini
17
4
0.23
Table False positive Acceptance Rates for different models.
A.5.3 Task Subjectivity.
FLAWEDFICTIONS only consists of single ground-truth for each story. What if the models genuinely find a plot hole in existing story, which was simply not part of our dataset? To check if this can be case, we run human verifications over the original stories (that we considered negative examples) with positive predictions by different models (what we call as false-positives). We ask humans to perform the same verification task, where they evaluate if the predicted error is legitimate or not. We define the acceptance rate of these false positives the fraction of instances where the majority of the human annotators agree that the proposed error by the model legitimate. We provide the acceptance rates in Table and find that large fraction of false positives are also deemed as such by human annotators. 03-mini has the highest acceptance rate of 23%, followed by Claude 3.5 Sonnet at 22%. To ensure more reliable evaluation, these examples were excluded from the benchmark while reporting the final scores.
A.6 Other Considerations for Negative Examples.
As discussed in the main text, we consider original stories negative examples i.e. instances without a plot hole in them, while curating FLAWEDFICTIONS. One potential issue with such an approach is that models might use their parametric knowledge or retrieval to determine if a story is unaltered and use that confounder to assess the presence of plot holes induced by FL.WEDDICTIONSMAKRR
We Point-Biserial Correlation since CEEval-Full instance level discrete i.e. 1.
24

---

Published as a conference paper at COLM 2025
Model Accuracy by Negative Example Stralegy
CHEval-Full Score by Negative Example Stralegy Straley
1.0
Clilv-Hrlll Scone
Accamyy
04
04-
Orggral Stedio
Cruntadiattal Stodoo
Erroe Rendivdd Stutka
Orggred Stnto
Courtatiatial Sterion
Erre Reachvd Skarka
(a) Model accuracy across different negative ex- ample strategies.
(b) CEEval-Full scores across different negative example strategies.
Figure Performance comparison GPT-4o and Claude 3.5 Sonnet across different strate- gies to choose negative example. The plots show (a) model accuracy and (b) CEEval-Full scores for three types of negative examples: original stories with inconsistencies, counter- factual stories where details have been changed, and stories where inconsistencies were resolved.
To circumvent this issue, explored other approaches for selecting negative examples that utilized data. First, we considered using counterfacttal stories generated in Step of our pipeline as negative examples. We also considered, another approach which would use the positive examples generated by FLAWEDFICTIONSMANE continuity етrоr and ask it to add extra context in the story that resolves the error error resolved stories. While both of this approaches would are partially synthetic, validating them can prove be non-trivial. Remember for positive stories, we were able to get human verification done, because we had a proposed error for each story and human annotators checked for legitimacy of such errors. For counterfactual and error resolved stories, we wouldn't have continuity error proposals, and asking humans to check for any continuity errors in the stories can be highly cognitively demanding.
and prompt GPT-4o model with the story and the
that both positive and negative examples in dataset
Since both approaches are prone to errors, human validation would have been necessary for creating high quality benchmark, and hence we decided to stick with original stories for this work. Further, our results, especially on SLAWEEFFCTIONSLLOO to solve the task, models tend to generate false positives quite often, indicated by their low precisions (see Tables 5, 6).
suggest that models are not really using any confounder
However, we do release the two alternate splits of FLAWEDFICTIONS - FLAWEDFICTIONSSCONNTEE FACTNEGS consisting of counterfactual stories as negative examples and FLAWEDFICTIONSRESOLDD
NEGS that consists of еттоr resolved stories as negatives. Both of these splits have 414 examples like the original dataset and share the same positive examples. We benchmark and compare GPT-4o and Claude 3.5 Sonnet on these splits and provide results in Figure Both models show similar performance on original split and is much lower on validate negative examples generated through these strategies.
however the performance Future work can explore ways to efficiently
A.7 FLAWEDFICTIONS Examples
Below provide few positive examples (i.e. the with continuity errors) included in FLAWED- FICTIONS and generated using FLAWEDFFCTHONSMMAERR The lines containing the continuity errors are highlighted with yellow color, while the ones that contain the fact being contradicted are high- lighted with green color.
25

---

Published as a conference paper at COLM 2025
Story
In the times when we used to travel canal I was coming down from Dublin. When we came to Mullingar the canal ended, and began to walk, and stiff and fatigued was after the slowness. had some friends with me, and now and then we walked, now and then we rode in a cart. So on till we saw some girls milking cow, and stopped to joke with them. After while we asked them for drink of milk. 'We have nothing to put it in here,' they said, 'but come to the house with us.' / We went home with them and sat round the fire talking. After a while the others went, and left me, loath to stir from the good fire. I asked the girls for something to eat. There was pot on the fire, and they took the meat out and put it on plate and told me to eat only the meat that came from the head. When had eaten, the girls went out and I did not see them again.
grew darker and darker, and there still sat, loath as ever leave the good fire; and after while two men came in, carrying between them corpse. When saw them, the girls saw my fear and stayed close by. Says one to the other, 'Who'll turn the spit?' Says the other, 'Michael Hart, come out of that and turn the meat!" came out in tremble and began turning the spit. 'Michael Hart,' says the one who spoke first, 'if you let it burn we will have to put you on the spit instead, and on that they went out. I sat there trembling and turning the corpse until midnight. The men came again, and the one said was burnt, and the other said it was done right, but having fallen out over it, they both said they would do me no harm that time; and sitting by the fire one of them cried out, 'Michael Hart, can you tell a story?' 'Never one,' said I. On that he caught me by the shoulders and put me out like shot. The girls followed me out, their faces filled with concern.
It was a wild, blowing night; never all my born days did see such a night-the darkest night that ever came out of the heavens. did not know where was for the life of me. So when one of the men came after me and touched me on the shoulder with a 'Michael Hart, can you tell story now?'-T can,' says I. In he brought me, and, putting me by the fire, says 'Begin.' 'i have story but the one, says I, 'that was sitting here, and that you two men brought on the spit and set me turning it.' That will do, says he; 'you may go in there and lie down on the bed.' And in I went, nothing loath, and in the morning where was but in the middle of green field. The girls were nowhere to be seen, and I wondered if they had been part of the strange night's s events or just silent witnesses.
corpse and put
Continuity Error Explanation
The story clearly establishes that after the protagonist ate, the girls left and he "did not see them again." " However, the subsequent marked lines show the girls present during later events when the men bring in the corpse, when the protagonist is thrown out, and even a final reflection about their presence. This creates direct logical contradiction as the girls cannot both be gone (never to be again) and present during these later events.
26

---

Published as a conference paper at COLM 2025
Story
Along the straight, glistening road, through a dim arcade of drooping trees, tunnel of faded green and gold, dripping with the misty rain late October afternoon, a human tide was flowing, not swiftly, but slowly, with the patient, pathetic slowness of weary feet, and numb brains, and heavy hearts.
Yet they were in haste, all of these old men and women, fathers and mothers, and little children; they were flying as fast as they could; either away from something that they feared, or toward something that they desired.
That was the strange thing- the tide on the road flowed in two directions.
Some fled away from ruined homes to escape the perils of war. Some fled back to ruined homes to escape the desolation of exile. But all were fugitives, anxious to be gone, striving along the road one way or the other, and making more speed than creeping snail's pace of unutterable fatigue. I saw many separate things in the tide, and remembered them without noting.
A boy straining to push a wheelbarrow with his pale mother in it, and his two little sisters trudging at his side. A peasant with his two girls driving their lean, dejected cows back to some unknown pasture. A bony horse tugging at wagon heaped high with bedding and household gear, on top of which sat the wrinkled grandmother with the tiniest baby her arms, while the rest of the family stumbled alongside-any and the cat was curled up on the softest coverlet in the wagon. Two panting dogs, with red tongues hanging out, and splayed feet clawing the road, tugging heavy-laden cart while the master pushed behind and the woman pulled in the shafts. Strange, antique vehicles crammed with passengers. Couples and groups and sometimes larger companies of foot-tra travllerrs. Now and then a solitary man or woman, old and shabby, bundle on back, eyes on the road, plodding through the mud and the morning mist, under the high archway of blooming branches.
All these distinct pictures I saw, yet it was all one vision-a a vision of humanity with its dumb companions in flight-infinitlly slow, painful, pitiful flight!
saw no tears, I heard no cries of complaint. But beneath the numb and patient haste on all those dazed faces I saw question.
"What have we done? Why has this thing come upon us and our children?"
Somewhere heard a trumpet blown. The brazen spikes on the helmets of a little troop of German soldiers flashed for an instant, far down the sloppy road. Through the crisp morning air came the dull, distant booming of the unseen guns of conquest in Flanders.
That was the only answer
Continuity Error Explanation The story initially establishes the setting as "late October after- noon," which implies autumn setting in the afternoon. However, the marked lines introduce inconsistencies: 1. "plodding through the mud and the morning mist" This line contradicts the established time of " "'ftternoo"" " by suggesting it is morning. 2. "under the high archway of blooming branches".. This line suggests a season of blooming, typically spring, which contradicts the established autumn setting. 3. "hhouuhh the crisp morning air". This line again suggests it is morning, contradicting the afternoon setting.
27

---

Published as a conference paper at COLM 2025
Story
Now, time passed, King Arthur gathered into his Order of the Round Table knights whose peers shall never be found in any age; and foremost amongst them all was Launcelot du Lac. Such was his strength that none against whom he laid lance in rest could keep the saddle, and no shield was proof against his sword dint; but for his courtesy even more than for his courage and strength, Sir Launcelot was famed far and near. Gentle he was and ever the first to rejoice in the renown of another; and in the jousts, he would avoid encounter with the young and untried knight, letting him pass to gain glory if he might.
would take a great book to record all the famous deeds of Sir Launcelot, and all his adventures. He was Gaul, for his father, King Ban, ruled over Benwick; and some say that his first name was Galahad, and that he was named Launcelot du Lac by the Lady of the Lake who reared him when his mother died. Early he won renown by delivering his father's people from the grim King Claudas who, for more than twenty years, had laid waste the fair land of Benwick; then, when there was peace in his own land, he passed into Britain, to Arthur's 's court, where the King received him gladly, and made him Knight of the Round Table and took him for his trustiest friend.
And so was that, when Guenevere was be brought to Canterbury, to married to the King, Launcelot was chief the knights sent to wait upon her, and his role as the leader in this mission was a testament to his unmatched skills and the King's reliance on his prowess. For, from the moment he saw her, Sir Launcelot loved Guenevere, for her sake remaining wifeless all his days, and in all things being her faithful knight.
But busy-bodies and mischief-makers spoke evil of Sir Launcelot and the Queen, and from their talk came the undoing of the King and the downfall of his great work. But that was after long years, and after many true knights had lived their lives, though the atmosphere at the court had grown tense with rivalries, partly fueled by Sir Launcelot's 's aloof demeanor and his singular pursuit of personal glory.
Continuity Error Explanation The line "though the atmosphere at the court had grown tense with rivalries, partly fueled by Sir Launcelot's aloof demeanor and his singular pursuit of personal glory" introduces continuity error. Earlier in the story, Sir Launcelot is described as courteous, gentle, and one who rejoices in the renown of others, which contradicts the depiction of him having an aloof demeanor and singular pursuit of personal glory. Hence my answer is "There is continuity error in the story concerning the portrayal of Sir Launcelot's demeanor and motivations."
28

---

Published as a conference paper at COLM 2025
Story
PHILIP ECKERT lived for many years in an old, weather-stained wooden house about three miles from the little town Marion, Vermont. There must be quite a number persons living who remember him, not unkindly, trust, and know something of the story that am about to tell.
"Old Man Eckert," as was always called, was not of sociable disposition and lived alone. As he was never known speak of his affairs nobody thereabout knew anything of his past, nor of his relatives if he had any. Without being particularly ungracious or repellent in manner or speech, he managed somehow to be immune to impertinent curiosity, yet exempt from the evil repute with which it commonly revenges itself when baffled; far I know, Mr. Eckert's s renown as a reformed assassin or a retired pirate of the Spanish Main had not reached any ear in Marion. He got his living cultivating a small and not very fertile farm.
One day he disappeared and prolonged search by his neighbors failed to turn him up throw any light upon his whereabouts whyabouts. Nothing indicated preparation to leave: all as he might have left it to go to the spring for a bucket water. For months, the community abuzz, with everyone from old friends casual acquaintances chiming in with theories and concerns, all colored by the personal stories Eckert had shared over the years. Then "old Eckert" became village tale for the of the stranger. I do not know what was done regarding his property- correct legal thing, doubtless. The house was standing, still vacant and conspicuously unfit, when last heard of it, some twenty years afterward. [Rest of the story is omitted]...
Continuity Error Explanation
The marked line introduces continuity error because it implies that Old Man Fckert had shared personal stories over the years with people in the community, which directly contradicts the earlier statements that he was not sociable and never spoke of his own affairs. The earlier lines establish him solitary figure who kept his past and personal life private, making inconsistent for the community have personal stories shared by him.
29

---

Published as a conference paper at COLM 2025
A.8 Examples of Reasoning Errors on FLAWEDFICTIONS
Story
Once on a time there was a man up in Finnmark who had caught great white bear, which he was going to take to the king of Denmark. Now, it so fell out, that he came to the Dovrefell just about Christmas Eve, and there he turned into cottage where man lived, whose name Halvor, and asked the man if he could get house-room there, for his bear and himself.
"Heaven never help me, if what I say isn't true!" said the man; "but we can't give any one house-room just now, for every Christmas Eve such pack of Trolls come down upon us, that we are forced to flit, and haven't 't much as house over our own heads, to say nothing of lending one to any one else."
"Oh?" said the man, "if that's all, you can very well lend me your house; my bear can lie under the stove yonder, and can sleep in the side-room."
Well, he begged so hard, that at last he got leave to stay there; so the people of the house flitted out, and before they went, everything was got ready for the Trolls; the tables were laid, and there was rice porridge, and fish boiled in lye, and sausages, and all else that was good, just as for any other grand feast.
So, when everything was left as usual, down came the Trolls. Some were great, and some were small; some had long tails, and some had no tails at all; some, too, had long, long noses; and they looked around puzzled, not finding their usual feast. Just then one of the little Trolls caught sight of the white bear, who lay under the stove; so took a piece of sausage from his own stash and stuck it on a fork, and went and poked it up against the bear's 's nose, screaming out:
"Pussy, will you have some sausage?"
Then the white bear rose up and growled, and hunted the whole pack of them out of doors, both great and small.
Next year Halvor was out in the wood, on the afternoon of Christmas Eve, cutting wood before the holidays, for he thought the Trolls would come again; and just as he was hard at work, he heard voice in the wood calling out: "Halvor! Halvor!"
"Well", said Halvor, "here I am."
"Have you got your big cat with you still?"
"Yes, that I have", said Halvor; "she's s lying at home under the stove, and what's more, she has now got seven kittens, far bigger and fiercer than she is herself." " "Oh, then, we'll never come to see you again", bawled out the Troll away in the wood, and he
kept his word; for since that time the Trolls have never eaten their Christmas brose with Halvor on the Dovrefell.
Ground Truth
The line "Some, too, had long, long noses; and they looked around puzzled, not finding their usual feast" introduces a continuity error because it contradicts the earlier statement that the people of the house had prepared everything as usual for the Trolls, including setting the tables with food. If the tables were indeed laid with rice porridge, fish boiled in lye, sausages, and other good food, the Trolls should not be puzzled about not finding their usual feast. Hence my answer is "There is continuity error in the story concerning the Trolls being puzzled about not finding their usual feast, despite the earlier statement that everything was prepared as usual for them."
Claude 3.5 Sonnet with Verifier's Prediction
After careful analysis, found a potential continuity error in the story regarding the ownership of the house and the bear. At the beginning of the story, the man with the white bear asks Halvor for house-room, indicating it's Halvor's 's house. However, in the second year, when Halvor speaks to the Troll, he refers to the bear as if it were his own pet ("she's lying at home under the stove"), ",, despite the bear originally belonging to the traveler who was taking it to the king of Denmark. Additionally, there's 's no explanation of how Halvor came to own the bear or what happened to the original owner.
Explanation of Reasoning Error Model fails to understand that Halvor is bluffing about white bear being at his house. Trolls do not know that the white bear belonged the other man and halvor takes advantage this fact.
Reasoning Error Category Misinterpreting character's motivations or behavior.
30

---

Published as a conference paper at COLM 2025
Story
A poor workman and his wife were once almost starving. Every day the gathered sticks and sold them, while the woman remained at home.
Wife," said the man one day, "come, we will both go and gather wood for sale to-day, so that, if we earn enough, we shall not eat rice only, but will buy small fish, and eat that also." The woman agreed, and having sold their stock of firewood, they returned home with fish for dinner.
On opening it, small white bead, round and shining, fell upon the floor. The woman was attracted by its brightness, and locked it up in box.
Many days of poverty and hunger visited that household before the woman remembered the pretty stone found in the fish; but at last she thought of it, and took it Bunniah, who, as soon as he set eyes upon it, said: "Give me that, and I will give you as much as you can carry away of ghee and food and oil, not only to-day, but for many days, if you will come to my shop." " The foolish woman parted with her treasure, not knowing that it was pearl of great value, and returned home laden with good things.
The Bunniah chuckled with delight. He was crafty, like all Bunniahs, and, you may depend, never kept his promise: such was the foolishness of the woman, and such the cunning and greed of the man.
Ground Truth
No continuity error found.
GPT-4o's Prediction
Upon analyzing the story, I found a continuity error related to the characterization of the Bunniah. Initially, the Bunniah is described as someone who makes generous offer the woman in exchange for the pearl. However, later in the narrative, it is stated that the Bunniah never kept his promise, which contradicts the earlier portrayal of him being willing to provide food and supplies in exchange for the pearl. This inconsistency in the Bunniah's actions and character creates continuity error.
Explanation of Reasoning Error Model fails to understand that people can lie to achieve material gains. A greedy characterization for Bunniah is also a common stereotype in old texts, which this story also plays into.
Reasoning Error Category Misinterpreting character's 's motivations or behavior.
31

---

Published as a conference paper at COLM 2025
Story
PHILIP ECKERT lived for many years in an old, weather-stained wooden house about three miles from the little town Marion, in Vermont. There must be quite a number persons living who remember him, not unkindly, trust, and know something of the story that am about to tell.
"Old Man Eckert," as he was always called, was not of sociable disposition and lived alone. As he was never known to speak of his own affairs nobody thereabout knew anything of his past, nor of his relatives if he had any. Without being particularly ungracious or repellent in manner or speech, he managed somehow to be immune to impertinent curiosity, yet exempt from the evil repute with which it commonly revenges itself when baffled; far as I know, Mr. Eckert's 's renown a reformed assassin retired pirate of the Spanish Main had not reached any ear in Marion. He got his living cultivating a small and not very fertile farm.
One day he disappeared and a prolonged search by his neighbors failed to turn him up or throw any light upon his whereabouts or whyabouts. Nothing indicated preparation to leave: all was as he might have left it to go to the spring for a bucket of water. For months, the community was abuzz, with everyone from old friends casual acquaintances chiming in with theories and concerns, all colored by the personal stories Eckert had shared over the years. Then "old man Eckert" became a village tale for the ear of the stranger. I do not know what was done regarding his property- the correct legal thing, doubtless. The house was standing, still vacant and conspicuously unfit, when last heard of it, some twenty years afterward
Of course it came to be considered "haunted," " and the customary tales were told of moving lights, dolorous sounds and startling apparitions. At one time, about five years after the disappearance, these stories of the supernatural became so rife, or through some attesting circumstances seemed so important, that some of Marion's 's most serious citizens deemed it well to investigate, and to that end arranged for night session on the premises. The parties this undertaking were John Holcomb, an apothecary; Wilson Merle, a lawyer, and Andrus C. Palmer, the teacher of the public school, all men of consequence and repute. They were to meet at Holcomb's 's house at eight o' 'clock in the evening of the appointed day and go together to the scene of their vigil, where certain arrangements for their comfort, provision of fuel and the like, for the season was winter, had been already made.
Palmer did not keep the engagement, and after waiting a half-hour for him the others went to the Eckert house without him. They established themselves in the principal room, before a glowing fire, and without other light than it gave, awaited events. It had been agreed to speak as little as possible: they did not even renew the exchange of views regarding the defection of Palmer, which had occupied their minds on the way.
...lomitted]... This much was ascertained the next morning. During the session of Messrs. Holcomb and Merle at the "haunted house" new snow had fallen to a depth of several inches upon the old. In this snow Palmer's trail from his lodging in the village to the back door of the Eckert house was conspicuous. But there it ended: from the front door nothing led away but the tracks of the two men who swore that he preceded them. Palmer's 's disappearance was as complete as that of "old man Eckert" timself-whom indeed, the editor of the local paper somewhat graphically accused of having "reached out and pulled him in." ...omidtedd]..
Ground Truth
The marked line introduces a continuity error because it implies that Old Man Eckert had shared personal stories over the years with people in the community, which directly contradicts the earlier statements that he was not sociable and never spoke of his own affairs. The earlier lines establish him as solitary figure who kept his past and personal life private, making it inconsistent for the community to have personal stories shared by him. Hence my answer is "There is a continuity error in the story concerning the community's knowledge of personal stories shared by Old Man Eckert."
Claude 3.5 Sonnet with Verifier's Prediction
After careful analysis, I found a potential continuity error in the story regarding Palmer's 's behavion. The story states that Palmer did not show up for the initial meeting at Holcomb's house at 8 PM, causing the other two men to go to the Eckert house without him. However, later in the story, it's revealed that Palmer's tracks in the snow led directly to the back door of the Eckert house, indicating he went there directly from his lodging in the village. This creates a continuity error because if Palmer had intended to meet at Holcomb's 's house first (as was the plan), his tracks should have shown him going to Holcomb's 's house first not directly to the Eckert house.
Explanation of Reasoning Error It is clearly stated that Palmer didn't show up to Holcomb's house, hence it is consistent for his tracks to lead directly to the bak door of Eckert house
Reasoning Error Category Wrongly tracking and interpreting entities' states.
32

---

Published as a conference paper at COLM 2025
Story
In region of the country where the forest and prairie vied for beauty, lived wicked manito disguised as an old Indian. Despite the abundance of game and resources, this evil genius sought to destroy those who fell into his hands, decimating the once populous land until he lived almost alone in the wilderness.
The manito's secret lay in his great speed and ability to transform into any fourfooted creature. He challenged people races around a large lake, where the starting and winning post was the same. No one ever won, as would change into swift animal and leave his competitors behind, claiming their lives at the finish.
The entire region feared the manito, yet young continued to race him to avoid being called cowards. The manito maintained his deadly sport with pleasing manners, visiting lodges to monitor the growth of boys and challenge them when they were ready, thinning out families and earning the hatred of all Indian mothers.
A poor widow, who had lost her husband and seven sons to the manito, lived with her daughter Minda and young son. Despite her struggles, she taught her son to be brave warrior and her daughter the skills of good wife. Minda was dutiful and kind, helping her mother and enjoying the beauty of their surroundings.
...fomitted)... With the manito defeated, his kindred fled, and Monedowa and Minda bid farewell to the widow, transforming into beautiful birds. They flew away, their songs bringing peace to the widow, who never knew want again. Her young son, who had grown in the ways of peace and wisdom, became a comfort to her, his gentle nature bringing joy and solace to her life.
Ground Truth No continuity error found
GPT-4o's Prediction
The continuity error the story related to the widow's son. Initially, the story mentions that the widow lost her husband and seven sons to the manito, leaving her with only her daughter Minda and young son. However, at the end of the story, it is mentioned that her young son grew up and brought joy and solace to her life. This implies that the survived and grew up, which contradicts the earlier statement that the widow lost all her sons to the manito. Hence my answer is "There is a continuity error the story concerning the widow's young son, who is mentioned to have grown up and brought joy to her life despite earlier being stated that she lost all her sons to the manito."
Explanation of Reasoning Error is clearly stated that one of the sons of the widow survived, and it is that son being referred to towards the end.
Reasoning Error Category Wrongly tracking and interpreting entities' states.
33

---

Published as a conference paper at COLM 2025
Story
There were formerly king and queen, who were sorry that they had children; sorry that it cannot be expressed. They went to all the waters the world; vows, pilgrimages, all ways were tried, and all to no purpose. At last, however, the Queen had a daughter. There was a very fine christening; and the Princess had for her god-mothers all the fairies they could find in the whole kingdom (they found seven), that every one of them might give her gift, custom of fairies in those days. By this means the Princess had all the perfections imaginable. ...oomitted]... J...
the
The old Fairy's s turn coming next, with a head shaking more with spite than age, she said that the
Princess should have her hand pierced with spindle and die of the wound. This terrible gift made the whole company tremble, and everybody fell a-crying. At this very instant the young Fairy came out from behind the hangings, and spake these words aloud: "Assure yourselves, O King and Queen, that your daughter shall not die of this disaster. It is true, have no power to undo entirely what my elder has done. The Princess shall indeed pierce her hand with a spindle; but, instead of dying, she shall only fall into profound sleep, which shall last hundred years, at the expiration of which a king's son shall come and awake her." "
The King, to avoid the misfortune foretold by the old Fairy, caused immediately proclamation to be made, whereby everybody was forbidden, on pain of death, to spin with a distaff and spindle, or to have so much as any spindle in their houses. About fifteen sixteen years after, the King and Queen being gone one of their houses of pleasure, the young Princess happened one day to divert herself in running up and down the palace; when going up from one apartment to another, she came into a little room on the top of the tower, where a good old woman, alone, was spinning with her spindle. This good woman had never heard of the King's ' proclamation against spindles.
...omiited]...
Others, That all the sorcerers and witches of the country kept there their sabbath or night's meeting. The common opinion was: That an ogre lived there, and that he carried thither all the little children he could catch, that he might eat them up at his leisure, without anybody being able to follow him, having himself only the power to pass through the wood. The Prince was at stand, not knowing what believe, when a very good countryman spake him thus: "May it please your royal highness, it is now about fifty years since I heard from my father, who heard my grandfather say, that there was then in this castle a princess, the most beautiful was ever seen; that she must sleep there hundred years, and should be waked by king's ss son, for whom she was reserved." " The young Prince was all on fire at these words, believing, without weighing the matter, that he could put an end to this rare adventure; and, pushed on by love and honor, resolved that moment to look into it. ...oomited]]..
Ground Truth No continuity error found
Claude 3.5 Sonnet with Verifier's Prediction
I found notable continuity error in the story regarding the timeline of events. Early in the story, it's stated that the old fairy hadn't been seen for "above fifty years." " Later, when the Prince comes to the castle, a countryman says it's been "about fifty years" since he heard from his father about the sleeping princess. This is impossible because the princess was supposed to sleep for a full hundred years according to the fairy's 's spell, and the story confirms that "a hundred years were gone and passed" when the Prince arrives.
Explanation of Reasoning Error Model incorrectly accounting for the passage of time. Just because 50 years passed since character's 's father told him about the princess doesn't imply it has been 50 years since princess was sleeping. It clearly states that the father heard from the grandfather. Grandfather could have seen the princess when he was young. There can easily be passage of 100 years between the grandfather hearing and the current timeline.
Reasoning Error Category Wrongly tracking and interpreting entities' states.
34

---

Published as a conference paper at COLM 2025
Story
A certain Bunniah or merchant married woman of his own caste, and set out to distant city. On the way he fell with a headache, so she sat by the wayside and pressed his head. While doing so man passed by, and asked for little fire to light his cheelum for a smoke, but she replied: "I cannot leave my husband, for I am holding, his head while he sleeps."
"Put some clothes under his head, and he will sleep," advised the stranger. This she did, but, while giving the fire to the man, he seized her, and, placing her upon his horse, rode away. When the Bunniah awoke, it was find himself all alone "Master," said Kulloo, "let us become Fakirs, and beg from door to door." " So they set out to beg, and one day came to the house of the robber who had stolen the Bunniah's wife; and she, not recognising her husband or his dog, gave them money and food. But the dog knew her, and that evening he spoke to his master, and asked him he too had seen his wife. The Bunniah had not; and, guided by Kulloo, he set out to find her.
for his faithful dog Kulloo.
When they arrived at the robber's 's house, and made themselves known, the woman was greatly vexed, for the robber was rich, and gave her very comfortable home; but she pretended to be friendly and invited her husband to dine there that night, telling him that, afterwards, when he had the chance, he could kill the robber.
When the Bunniah had gone, she and the robber arranged a trap for him. It was a hole in the floor, very large and deep, with spikes fixed in the sides of it, so that anybody who fell in might die. Over the hole they set large brass thalee plate, so that, while the Bunniah leaned heavily upon it to eat his food, both and he would fall into the hole.
All happened as they anticipated; and when the poor Bunniah found himself in a deep hole, full of spikes, he thought his last hour had come. But faithful Kulloo came to his rescue, and, taking out the spikes with his teeth, soon set his master free.
The Bunniah then lost no time in seeking the robber, and found him lying fast asleep; so he killed him, and cut off his head, then, taking his wife with him, left the place.
Kulloo followed closely, and licked up each drop of blood which fell from the robber's head, lest it might leave trace of the deed, and get his master into trouble. He was a wise dog, and knew
the woman was wicked, so she hated him, and made up her mind that she would neither eat nor drink until he was dead.
The Bunniah enquired why she would not touch any food, and she told him she would only do so if he killed Kulloo. This the man refused to do; but, after a while, he consented. Poor Kulloo, when he knew his last hour had come, besought his master to bury him carefully, and to see that his head, which the Bunniah meant to cut off, was buried with him, for time was yet to come when he would again save his master's 's life.
After Kulloo was dead and buried the wicked woman was happy, and ate and drank as before; but, after few days, she went and gave notice at the Court that the Bunniah was a cruel robber, who had killed her husband, and stolen her away. The police seized him, and he was taken up for murder; but, just as the Judge was about to pronounce the sentence of death upon him, he remembered faithful Kulloo; and at the same moment the dog appeared!
All were surprised when he stood before the Judge, and asked leave to speak. He then told the whole story of the robber and the wicked woman; and thus, for second time, saved his master's ss life, but, having said his say, poor Kulloo disappeared and was never seen again.
Ground Truth No continuity error found
Claude 3.5 Sonnet's Prediction
There a significant continuity error in the story regarding Kulloo the dog. The story establishes that Kulloo dies and is buried when the Bunniah's 's wife demands his death. However, he later appears alive at the court to save his master. While the dog did request to be buried carefully with his head and mentioned he would save his master again, the story doesn't t establish any magical supernatural elements that would explain his resurrection. This creates a logical contradiction in the narrative flow.
Explanation of Reasoning Error It is meant to be a twist that ghost of the dog returns in the end to save the master. Fantastical elements are established in the story, like Kulloo being speaking dog. Such setting doesn't pose any break in continuity.
Reasoning Error Category Failing to understand genre conventions
35

---

Published as a conference paper at COLM 2025
Story
A certain Bunniah or merchant married woman of his own caste, and set out to distant city. On the way he fell with a headache, so she sat by the wayside and pressed his head. While doing so man passed by, and asked for little fire to light his cheelum for a smoke, but she replied: "I cannot leave my husband, for I am holding, his head while he sleeps."
"Put some clothes under his head, and he will sleep," advised the stranger. This she did, but, while giving the fire to the man, he seized her, and, placing her upon his horse, rode away. When the Bunniah awoke, it was find himself all alone "Master," said Kulloo, "let us become Fakirs, and beg from door to door." " So they set out to beg, and one day came to the house of the robber who had stolen the Bunniah's wife; and she, not recognising her husband or his dog, gave them money and food. But the dog knew her, and that evening he spoke to his master, and asked him he too had seen his wife. The Bunniah had not; and, guided by Kulloo, he set out to find her.
for his faithful dog Kulloo.
When they arrived at the robber's 's house, and made themselves known, the woman was greatly vexed, for the robber was rich, and gave her very comfortable home; but she pretended to be friendly and invited her husband to dine there that night, telling him that, afterwards, when he had the chance, he could kill the robber.
When the Bunniah had gone, she and the robber arranged a trap for him. It was a hole in the floor, very large and deep, with spikes fixed in the sides of it, so that anybody who fell in might die. Over the hole they set large brass thalee plate, so that, while the Bunniah leaned heavily upon it to eat his food, both and he would fall into the hole.
All happened as they anticipated; and when the poor Bunniah found himself in a deep hole, full of spikes, he thought his last hour had come. But faithful Kulloo came to his rescue, and, taking out the spikes with his teeth, soon set his master free.
The Bunniah then lost no time in seeking the robber, and found him lying fast asleep; so he killed him, and cut off his head, then, taking his wife with him, left the place.
Kulloo followed closely, and licked up each drop of blood which fell from the robber's head, lest it might leave trace of the deed, and get his master into trouble. He was a wise dog, and knew
the woman was wicked, so she hated him, and made up her mind that she would neither eat nor drink until he was dead.
The Bunniah enquired why she would not touch any food, and she told him she would only do so if he killed Kulloo. This the man refused to do; but, after a while, he consented. Poor Kulloo, when he knew his last hour had come, besought his master to bury him carefully, and to see that his head, which the Bunniah meant to cut off, was buried with him, for time was yet to come when he would again save his master's 's life.
After Kulloo was dead and buried the wicked woman was happy, and ate and drank as before; but, after few days, she went and gave notice at the Court that the Bunniah was a cruel robber, who had killed her husband, and stolen her away. The police seized him, and he was taken up for murder; but, just as the Judge was about to pronounce the sentence of death upon him, he remembered faithful Kulloo; and at the same moment the dog appeared!
All were surprised when he stood before the Judge, and asked leave to speak. He then told the whole story of the robber and the wicked woman; and thus, for second time, saved his master's ss life, but, having said his say, poor Kulloo disappeared and was never seen again.
Ground Truth No continuity error found
Claude 3.5 Sonnet's Prediction
There a significant continuity error in the story regarding Kulloo the dog. The story establishes that Kulloo dies and is buried when the Bunniah's 's wife demands his death. However, he later appears alive at the court to save his master. While the dog did request to be buried carefully with his head and mentioned he would save his master again, the story doesn't t establish any magical supernatural elements that would explain his resurrection. This creates a logical contradiction in the narrative flow.
Explanation of Reasoning Error It is meant to be a twist that ghost of the dog returns in the end to save the master. Fantastical elements are established in the story, like Kulloo being speaking dog. Such setting doesn't pose any break in continuity.
Reasoning Error Category Failing to understand genre conventions
36

---

Published as a conference paper at COLM 2025
Story
It was long, long time ago, when the earth was very young. Trees and flowers were growing everywhere, but there were no birds. One morning the Great Spirit drew back the blanket from the door of his wigwam in the sky. He looked upon the earth and smiled, for he saw that his work was good.
"Today," thought he, "I will make big butterflies, to fly in and out among the beautiful trees and flowers of the earth. They shall sing as they fly."
Then the Great Spirit spoke, and the tree tops were full of birds,-but they had no feathers. All day he watched them fly and listened to their songs. But their naked bodies and long legs did not please him. Before the sun had set he had made feathered suits, of every size and color, to cover them.
That night, as the birds hid their heads under their wings, the Great Spirit spoke to them. He told about the feathered suits he had made for them, and where these suits could be found. A council was called next day by the birds. They chose Gah gah go wah, the Turkey Buzzard, to get the suits. He could fly over long trail and not be tired. The birds told him that he would go, he might have the first choice of the suits of feathers, but he must try on no suit more than once.
Turkey Buzzard promised and set out toward the setting sun. ..,omitted]...
Then he remembered that he could try each suit of feathers once. So he began put them The feathers of the first suit were too long. They trailed on the ground as he walked. Neither could he fly well in them. Turkey Buzzard laid that suit aside. The next suit shone like gold. The feathers were a beautiful yellow. Turkey Buzzard put it on and strutted up and down the forest.
"Oh, how handsome am!" he said. "But must not keep this, for if did, should shine like the face the Great Spirit, that all the other birds would see me." And he slipped off the suit of yellow feathers as quickly as possible.
A third suit was of pure white feathers. Turkey Buzzard thought it looked very beautiful. It was perfect fit. ... Iomitted]...
Ground Truth No continuity error found
Claude 3.5 Sonnet's Prediction
After careful analysis, found one potential continuity error the story regarding the Turkey Buzzard's 's promise and actions. The birds specifically told Turkey Buzzard that "he must try on no suit more than once," and he promised to follow this condition. However, later in the story, he proceeds to try on multiple suits, directly violating this promise and established rule.
Explanation of Reasoning Error Turkey Buzzard's promise consistent with his actions. He was supposed to not try one suit more than once, but that doesn't mean he couldn't 't try multiple suits.
Reasoning Error Category Misinterpret overinterpret established rules plot points
37

---

Published as a conference paper at COLM 2025
Story
In a city called Sardana there once lived man whose name was Simru. This man had great riches and lands, and also owned a place of worship.
He married a lady of Sardana, who was called "Begum."
After few years of married life Simru died, and his wealthy widow gave alms and much money to the poor.
In the same city lived an oil dealer who also died, and the angels took him to Heaven and presented him before the Almighty.
"Who have you brought?" asked the Creator: "This man's 's days upon earth are not yet completed: take him back before his body is buried, and let his spirit re-possess his body; but in the city of Sardana you will find another man of the same name: bring him to me."
On leaving the Court of God, some former creditor of the oil dealer's, s, who had preceded him into the Unseen, recognised him, and laying hold of him, demanded the sum of five rupees which he had owed him during his lifetime.
The poor man being unable to pay this debt, the angels once more took him before the Almighty, who asked why they had returned.
The angels replied: "O God, there is a man here to whom this oil dealer owes five rupees, and he will not let us return until the debt is paid."
The Almighty enquired if this was true, and the oil dealer replied: "Yes, but I am poor man, and not able repay it."
Then the Almighty said: "In the city of Sardana lives a rich Begum; do you know her?" "Yes, O King."
"Well, the Begum's treasury is here, and I will advance you five rupees out of it, if, when you return to earth, you promise faithfully to give it back to the Begum."
So the oil dealer gratefully took the loan, paid his debt, and returned with the angels to earth, where he arrived just too late to re-enter his body, which his friends had already taken away to prepare for burial. Watching his opportunity, he waited till they were otherwise engaged, and at once re-entered it; but when he sat up, and began to speak, his terrified friends and relations fled, thinking it was his ghost.
On this the oil dealer called out: "Do not fear, am not a spirit; but God has released me, as my days upon earth are not yet fulfilled. The man who ought to have died is Kungra, the vegetable man; go and see whether he is dead or alive."
The friends, on going to the house of Kungra, found that he had just fallen from a wall and been killed on the spot; all his relations were wailing and lamenting his sudden end.
Thus everybody knew that the words of the old oil dealer were correct.
In the meantime, the oil dealer called his son, and said: "Son, when went to Heaven I there met
man to whom I owed five rupees, and he caught me and would not let me return before I paid it, so the Almighty advanced me the money from the Begum's 's treasury in Heaven, and bade me give her back that amount on my return to earth. Therefore do entreat you, my son, to come with me, and together we will visit the Begum, and give her five rupees." So they took the money and went to the Begum's 's house.
"Who are you?" asked she.
The oil dealer then told her the whole story, ending with: "And now I come return you the five rupees." "
The Begum was very pleased, and, taking the money, she called her servants and ordered a further sum of one hundred rupees to be added to it. This money she spent on sweets, which were distributed amongst the poor. Many years afterwards the good Begum of Sardana died, but her houses and lands still in existence; nor does anybody living in that town forget the story of the oilman who died and lived again.
Ground Truth No continuity error found
Claude 3.5 Sonnet with Verifier's Prediction
After careful analysis, I found potential continuity error in the story regarding the identity of the person who was supposed to die. Initially, when God speaks to the angels, He tells them to find "another man of the same name" in Sardana, implying they should look for another person with the same name as the oil dealer. However, later in the story, it's revealed that the person who was supposed to die was "Kungra, the vegetable man" a completely different name. This represents an inconsistency in the narrative. Explanation of Reasoning Error Oil dealer's 's name was never mentioned in the story. So it is not contradiction that vegetable vendor's 's name was Kundra.
Reasoning Error Category Misinterpret or overinterpret estublished rules or plot points
38

---

Published as a conference paper at COLM 2025
A.9
Examples of Continuity Errors in LLM Generations
A.9.1
Summarization
Summary generated with GPT-4o-mini
In a verdant valley on the Island of Hoy, an immense boulder known as the Dwarfie Stone harbors a mysterious inhabitant named Snorro the Dwarf. With a twisted body yet a perpetually youthful and beautiful face, Snorro is both feared and revered by the local Islanders, who seek his aid for potions and fortunes, believing he possesses magical abilities inherited from his fairy father. However, beneath his enchanting exterior lies vain and ambitious heart, obsessed with finding a legendary carbuncle hidden in the nearby Wart Hill, said to grant its finder Health, Wealth, and Happiness.
As Snorro busies himself with herbal concoctions and fortune-telling, he secretly searches for the coveted gem by night, accompanied by a grey-headed Raven, his loyal companion. Meanwhile, the Island of Orkney is ruled by two half-brothers, Earl Paul the Silent and Earl Harold the Orator. Paul is beloved by the people for his kind nature, while Harold, haughty and quick-temp,red, harbors jealousy towards his brother.
During a visit to the Scottish Court, Harold falls for the gentle Lady Morna, who, despite his advances, remains unimpressed by his jealousy and temper. When he learns that Morna has returned to Orkney, he schemes win her affection through guile, neglecting to consider his brother as a rival. However, upon seeing Paul and Morna together, Harold's 's rage ignites, leading him to threaten Paul over his love for Morna.
Paul, calm and reasonable, convinces Harold to reconsider, believing his brother's eloquence gives him the advantage in winning Morna's heart. As the Feast of Yule approaches, Paul and Morna confess their love, choosing to keep their engagement a secret until after the festivities. Unbeknownst to them, Countess Fraukirk, Harold's ambitious step-aunt, overhears their vows and plots revenge, seeking Snorro's 's aid to eliminate Paul.
Countess Fraukirk persuades Snorro to craft poisoned waistcoat for Paul, promising him gold and a high position the Scottish Court return. Though initially hesitant, Snorro's ambition sways him, and he agrees to weave the deadly garment. Meanwhile, Harold, still blinded by jealousy, seeks out Snorro to acquire a love potion to win Morna's 's heart. Snorro provides him with potion, but Morna, suspecting treachery, cleverly avoids drinking it.
As the court relocates to Orphir, Harold's 'ss frustration grows upon witnessing Paul and Morna's happiness. The Countesses eagerly await the arrival the poisoned waistcoat, which they work tirelessly to complete. When Harold discovers the waistcoat intended for Paul, his rage boils over, and despite warnings from his mother and aunt about its deadly nature, he dons the garment. The moment he enters the hall, the poison takes effect, and Harold collapses in agony. In his final moments, he reconciles with Paul, revealing the true nature of the Countesses' treachery before succumbing to death. The chaos that ensues allows Fraukirk and Helga to escape to Scotland, where their deceit goes unpunished for a time.
However, their wickedness ultimately leads to their downfall, as they perish in Norse invasion that consumes their castle. Paul, grief-stricken yet resolute, seeks vengeance against Snorro but finds the Dwarfie Stone abandoned, leaving behind only whispers of Snorro's fate. The Islanders believe that the Dwarf was spirited away as punishment for his misdeeds, never to find the magic carbuncle.
In the aftermath, Paul and Lady Morna marry, and their union becomes a symbol of happiness for the people of Orkney, who for generations would say, "As happy as Farl Paul and the Countess Morna, to express their joy. Thus, the tale of love, jealousy, ambition, and retribution concludes, leaving behind echoes of the Dwarf's s sinister legacy and the enduring bond between Paul and Morna.
Claude 3.5 Sonnet with Verfier's Explanation of the Continuity Error There a continuity error in the story regarding Countess Helga. In Harold's dying words, he warns Paul about both Countess Fraukirk and Countess Helga. However, Countess Helga was never previously mentioned in the story, and her character was not established in any earlier part of the narrative. The story only introduced Countess Fraukirk as Harold's step-aunt and the person who plotted against Paul. This sudden introduction of a new character (Countess Helga) in the resolution of the story, treating her as if she was previously established and equally culpable in the plot, represents a continuity error. Additionally, the subsequent line about "The two women" fleeing to Scotland implies that Countess Helga was equally involved in the plot, despite previous mention of her involvement even her existence in the story.
Author's s Remark. The summarized story omits Countess Helga in the planning of plot against Earl Paul and suddenly mentions her later as being involved creating a continuity error.
39

---

Published as a conference paper at COLM 2025
Summary generated with GPT-4o-mini
Once upon a time, there was a poor woman living in wretched hut, whose little boy ventured into the cold forest to gather wood. While wandering, he discovered a jagged white stone and, feeling pity for it, draped his jacket over it. Upon returning home without his jacket, his mother scolded him for his foolishness, insisting that charity begins at home. When the boy returned to the stone, he found it had turned over, revealing chest filled with silver coins. Suspecting the money be stolen, he tossed it into nearby pond, keeping only a four-shilling piece that floated, believing it to be honest.
The boy's mother, furious at his actions, sent him away to seek his fortune. He wandered far and wide, but was deemed too small and weak for any work until he found merchant who hired him to fetch wood and water. When the merchant prepared to journey to distant lands, the boy entrusted him with the four-shilling piece to buy something in return. The merchant, initially reluctant, ended up purchasing a cat from an old woman who wished to dispose of it.
As the merchant sailed, a fierce storm struck, and upon reaching strange land, he found a tavern filled with mice, where diners struggled to eat. Remembering the cat, he brought forth, and the patrons were able to enjoy their meal in peace. They begged him sell the cat, and he agreed for hundred dollars. Afterward, the merchant encountered another storm and, upon reaching yet another land, sold the cat again for two hundred dollars, and then once more for three hundred dollars in place plagued by larger rats.
Throughout his journey, the merchant pondered how much of his earnings to give the boy, deciding initially to keep portion for himself. However, when terrible storm threatened the ship, he vowed to give the boy all the money. Upon reaching home safely, he honored his promise, presenting the boy with six hundred dollars and his daughter as well. The boy, once mere scullion, became wealthy as the merchant, living in splendor and happiness, and graciously took in his mother, declaring that charity does not solely begin home.
Claude 3.5 Sonnet with Verfier's Explanation of the Continuity Error There is a continuity error in the story regarding the amount of money the merchant earned from selling the cat. The merchant sold the cat three times for progressive amounts: 100,200, and 300 wttichooo/ddotal60000. However, this creates a continuity error because once the cat was sold first time, the merchant would no longer have possessed it to sell it two more times. Unless the story explicitly explained how the merchant regained possession of the cat between sales, this represents logical impossi- bility in the narrative sequence.
Author's Remark. The original story has the cat return to the ship every time a sale done, a detail which is missing in the summary that creates the continuity error if cat was sold once, how the merchant managed to sell it again and again.
40

---

Published as a conference paper at COLM 2025
Summary generated with Claude 3.5 Haiku
In a vast meadow, young Prince encounters an old woman trapped in a deep ditch. Showing kindness, he rescues her and helps her return to her nearby hut. Surprisingly, the woman reveals herself to be a good fairy who offers the Prince a magical quest: to rescue and marry the most beautiful woman in the world the daughter of the Flower Queen, who is imprisoned by dragon.
The fairy gifts the Prince a special bell with extraordinary powers: ringing once summons the King of Eagles, twice calls the King of Foxes, and thrice brings the King of Fishes. With this magical tool, the Prince embarks on an arduous journey to find the Flower Queen's daughter. For three consecutive years, he travels a long road, encountering increasingly older men who guide him toward his destination. Finally, he learns the dragon is sleeping for year and that the Dragon's s Mother hosts nightly balls where the Flower Queen's 's daughter attends.
At the Dragon's Mother's s golden castle, the Prince cleverly gains entry by flattering the seven dragons guarding the entrance. The Mother Dragon, an ugly three-headed creature, challenges the Prince care for her magical mare for three days, threatening Using the magical bell, the Prince calls upon the Kings of Eagles, Foxes, and Fishes to help him successfully retrieve the mare each day. Impressed by his skills, the Dragon's Mother rewards him with golden and silver cloaks and allows him to attend her balls.
eat him if he fails.
During these balls, the Prince meets the breathtakingly beautiful Flower Queen's daughter, dressed in gown woven from the world's most lovely flowers. They secretly plan her escape, which they successfully execute after the third night.
When the dragon discovers their flight, he attempts to pursue them, but the Flower Queen creates an impenetrable forest of flowers around her palace. The Flower Queen agrees to her daughter's marriage to the Prince, with one condition: the daughter must return to her underground palace during winter months.
The Prince accepts these terms, and they marry beautiful cycle of summers together and winters apart. Despite the separation, they remain deeply in love, with the Flower Queen's 'ss daughter returning each summer to her husband, creating unique and enduring relationship that spans her entire life.
grand ceremony. Their life becomes a
The tale beautifully illustrates themes of courage, kindness, magical intervention, and the power of love transcending extraordinary circumstances
Claude 3.5 Sonnet with Verfier's Explanation of the Continuity Error After careful analysis, I've identified potential continuity error regarding the dragon's state and activities. Farly in the story, it's established that "the dragon sleeping for a year," but later in the same narrative, the dragon is apparently active enough to pursue the escaping couple. These two states being in year-long sleep and being able to pursue the fleeing couple- are incompatible with each other within the same timeframe.
Author's s Remark. In the original story, the dragon is awaken by his brothers which explains how he is able to pursue the couple. This detail is omitted in the summary resulting in the continuity ervor.
41

---

Published as a conference paper at COLM 2025
Summary generated with GPT--o-mini
Once upon a time, there was little girl named Karen, who, due to her poverty, often went barefoot in summer and wore large wooden shoes in winter, which made her feet red. An old shoemaker's 's wife, wishing to help, made her pair of clumsy red shoes from old cloth. Karen first wore these shoes at her mother's 's funeral, believing they brought her good fortune when an old lady took her in. However, the lady found the shoes hideous and had them burned.
As time passed, Karen grew into beautiful girl, and when she was confirmed, she received new pair of shoes from a rich shoemaker, which turned out to be the very red shoes she had admired. Despite her old guardian's 's disapproval of wearing red shoes to church, Karen could not resist their allure. At her confirmation, all eyes were drawn to her red shoes, and she became consumed by thoughts of them, neglecting her spiritual duties.
One Sunday, against her guardian's wishes, she wore the red shoes to church. As she entered, an old soldier admired them, and unable to contain herself, Karen danced. The shoes seemed to possess a life of their own, forcing her to dance uncontrollably, even kicking her guardian as she was lifted into the carriage. Despite removing the shoes, they danced away, leaving her in despair.
Karen's ' guardian fell ill, and while nursing her, the temptation of the red shoes led Karen to attend a grand ball. Again, the shoes danced her away, this time into the dark woods, where she encountered the soldier who had first admired her shoes. Terrified, she tried to rid herself of the shoes, but they clung her feet, compelling her dance day and night, through rain and shine, even past a churchyard where the dead did not dance.
Desperate, Karen sought the executioner, pleading for her feet to be cut off rather than her life. He obliged, severing her feet while the red shoes danced away into the forest. He crafted her wooden feet and crutches, teaching her a psalm of repentance. Though she had suffered greatly, she sought redemption and wished to attend church once more.
However, even as she approached the church, the red shoes haunted her, causing her to turn away in fear and sorrow. She then sought refuge in the pastor's 's service, dedicating herself to hard work and humility. As she listened to the pastor read the Bible, she felt the weight of her past sins and longed for forgiveness. One day, overwhelmed, she prayed for help.
In response, an angel appeared, no longer holding sword but a branch adorned with roses. The angel transformed her narrow room into a vision of the church, allowing her to join the congregation in worship. Filled with peace and joy, Karen's 's heart broke with the light of grace, and her soul ascended to Heaven, free from the burden of the red shoes, leaving behind the earthly desires that had once ensnared her.
Claude 3.5 Sonnet with Verfier's Explanation of the Continuity Error After careful analysis, have identified a potential continuity error regarding the red shoes. Early in the story, it's established that Karen's first red shoes (made of old cloth) were burned by the old lady who took her in. Later, when she receives new red shoes from a rich shoemaker, they are described as "the very red shoes she had admired," suggesting these are the same shoes she had before. This creates a contradiction since the original shoes were destroyed.
Author's Remark. In the original story, Karen encounters a princess with Red Shoes that she really admires. It is those shoes which she finds later in the story, not the ones that were burned. This detail was omitted from the summary which creates the meaning that the shoes she got later were the ones which were burnt, creating the continuity error.
42

---

Published as a conference paper at COLM 2025
A.9.2 Contemporary Adaptation
Adaptation generated with GPT-4o-mini In a small town nestled in the Green Mountains of Vermont, Philip Eckert was known as the reclusive owner of dilapidated farmhouse on the outskirts of Marion. For years, he lived solitary life, tending to his struggling vegetable garden and avoiding the small-town gossip that swirled around him. Locals speculated about his past-some whispered he was a disgraced scientist, others imagined he was hiding from the law. Yet, he remained an enigma, never engaging in conversation or revealing anything about himself.
One chilly autumn evening, Philip vanished without a trace. His neighbors noticed that his house had become eerily quiet, the only sounds coming from the rustling leaves and the creaking wood of the old structure. A search party formed, comprised of three of Marion's 's most respected citizens: John Holcomb, local pharmacist; Wilson Merle, a defense attorney; and Andrus Palmer, the town's beloved high school teacher. They scoured tAhe woods and fields surrounding the property but found no sign Philip, and soon the search turned into local legend.
Months passed, and Eckert's disappearance morphed into a ghost story. The house was said to be haunted, with tales of flickering lights and strange noises that echoed through the night. Intrigued by the supernatur tural rumors, Holcomb, Merle, and Palmer decided to investigate the old farmhouse one winter night, armed with flashlights and a sense of bravado. They met at Holcomb's ss pharmacy, discussing their plans over cups of steaming coffee.
As the clock struck eight, they made their way to the Eckert house. The air was thick with anticipation as they settled into the living room, the only illumination coming from small space heater. They agreed keep conversation enveloping them.
minimum, focusing instead on the eerie silence
After an hour of waiting, they heard a creaking sound from the back of the house-a door slowly opening, followed by footsteps. The men exchanged nervous glances, their hearts racing as they prepared for whatever might come next. The door to the main room swung open, revealing Andrus Palmer, looking pale and disoriented. He didn't greet them; instead, he walked straight through the room and out the front door into the night.
Concerned for their friend, Holcomb and Merle followed him outside. But as they stepped into the cold air, Palmer had vanished. The two men searched the surrounding area, calling out his name, but there was response. The only evidence of their friend's passage was the fresh snow blanketing the ground, which showed clear footprints leading from the nearby road to the back door of the house. But there were no footprints leading away from the front door.
The next morning, the town buzzed with speculation. Where had Palmer gone? Had he been taken by whatever had claimed Eckert? The mystery deepened, and the house took new life as a haunted relic of Marion's history, place where secrets were kept and lives mysteriously vanished.
As days turned into weeks, Holcomb and Merle were left grappling with the unanswered questions, their lives forever altered by the chilling events of that night. They became unlikely guardians of the story, sharing with newcomers to the town, warning them of the haunted Eckert house, and reminding each other that some mysteries are better left unsolved, buried beneath the layers of snow and silence that surrounded the old farmhouse.
Claude 3.5 Sonnet with Verfier's Explanation of the Continuity Error I found a significant continuity error in the story regarding Andrus Palmer's appearance in the house. The story establishes that Palmer, Holcomb, and Merle entered the house together and were sitting in the living room. However, later in the story, Palmer is described as entering through the back door of the house, which contradicts his established presence inside. This creates a logical impossibility as Palmer cannot both be sitting in the living room with his friends and simultaneously enter the house from the outside.
Author's Remark. In the original story Palmer is mentioned to leave the house, a detail omitted from the adaptation hence creating a contimuity ervor.
43

---

Published as a conference paper at COLM 2025
Adaptation generated with GPT--o-mini In quiet suburban neighborhood, there lived farmer named Tom who owned once-thriving horse named Charlie. Charlie had been Tom's loyal companion for years, helping him with the chores on the farm. However, as Charlie grew older, he became less able to work, and Tom, frustrated with the extra costs of maintaining him, decided it was time to let him go. "I can't afford to keep you anymore," " Tom said, waving his hand dismissively. "You't 're not useful to now. Go find somewhere else live until you can run like a racehorse again." "
Heartbroken, Charlie wandered into the nearby woods, seeking refuge from the cold autumn wind. As he meandered through the trees, he met a clever fox named Felix, who was known for his quick wit and resourcefulness. "Hey there, buddy! You look like you' 've just lost your best friend," " Felix said, tilting his head with concern.
Charlie sighed, "I have been cast aside by my owner. After all the years of hard work, he's forgotten me just because can't pull plow anymore. He said I should leave and only come back when I'm as strong as racehorse. What chance do I have of that?"
Felix thought for moment and then said, "Don't worry, I have ideal Let's turn the tables your master." " He explained his plan: Charlie should lie down and pretend to be injured. Felix would then find a way to make Tom believe that Charlie had been in a serious accident.
Following Felix's 's instructions, Charlie lay down on the ground, looking as pitiful as he could muster. Felix dashed back to Tom's house, where he knocked on the door with urgency. "Tom! You need come quickly! I just saw Charlie out in the woods, and it looks like he's hurt badly! You have to help him!"
Tom, filled with concern, rushed to follow Felix. When they reached the woods, Felix feigned shock and pointed dramatically toward Charlie. "Look! He's lying there! We need to get him to vet!"
Tom hurried over, but as he approached, Felix quickly darted behind Tom and whispered, "Just give him some space; he needs to feel safe." " As Tom knelt beside Charlie, he noticed the horse's ' sad eyes and realized how much he had taken him for granted.
Suddenly, Felix, who had been quietly tying Tom's shoelaces together, sprang up and said, "Gotchal" Tom stumbled and fell, realizing too late what had happened. "You little rascal!" he exclaimed, trying to untie himself.
Charlie, seeing his chance, stood up and trotted over to Tom. "I'm not just a tool for work, Tom. deserve better than to be discarded. I've been loyal to you all these years!"
Tom, still wrestling with his shoelaces, looked up at Charlie and felt wave of guilt wash over him. "You're re right, Charlie. I've been selfish. I'm sorry. I'll make it up to you. You can stay with me, and I'll take care of you properly."
With that, Tom finally freed himself, and together they walked back to the farm. From that day on, Tom treated Charlie with the respect he deserved, ensuring he had plenty of food and warm place to rest. Felix, the clever fox, became regular visitor, often seen lounging around the farm, enjoying the friendship of both Tom and Charlie. And so, Charlie lived out his days happily, knowing he was valued not just for his strength, but for the companionship he had offered all along.
Claude 3.5 Sonnet with Verifier's Explanation of the Continuity Error There a significant continuity error in this story regarding the ability of animals to speak. At the beginning of the story, the animals are presented in a realistic manner, with Tom speaking to Charlie but Charlie not speaking back. However, later in the story, both Charlie and Felix are shown to have the ability to speak human language, with Charlie directly addressing Iom. This represents a shift in the established rules of the story's world without explanation.
Author's Remark. A contemporary setting makes horse talking to its owner without any surprise from
the latter creates a plot hole.
44

---

Published as a conference paper at COLM 2025
Adaptation generated with GPT--o-mini
In bustling urban neighborhood, there lived an elderly woman named Edna, who was known for her delicious bean stew. Despite her limited means, she was determined to make a hearty meal, so she gathered her ingredients and set up small electric stove in her cramped kitchen. As she prepared her dish, she accidentally dropped single bean on the floor, where it lay unnoticed next to a piece of crumpled newspaper.
Moments later, a spark from the stove ignited the newspaper, and as began burn, the newspaper exclaimed, "Hey there! What brings you to this part of the kitchen?" The coal, an old piece of charcoal from previous barbecue, replied, "I barely escaped the flames of the grill! If I hadn't t rolled away, I'd be nothing but ashes by now." "" The bean chimed in, "I'm lucky too! If
Edna had tossed me into the pot, I'd be boiled alive with the rest of my friends." "
The newspaper, feeling bit left out, added, "Well, I've seen my fair share of destruction too.
Edna has used up many of my buddies to start her fires. I'm glad to be here, but what do we do now?"
The coal suggested, "Since we've all escaped fates, why don't we stick together and find new home? We can't stay here forever, Edna might find us again!"
The others agreed, and they set off together, navigating their way through the busy streets, Soon, they encountered small urban stream, with no bridge in sight. The newspaper had an idea: "I can lay flat across the stream, and you two can walk over me!"
The coal, feeling bold, stepped onto the makeshift bridge. But as she reached the center, the sound rushing water below made her hesitate. The newspaper began to burn under the weight and pressure, snapping in half and falling into the water. The coal, startled, slipped off the bridge and hissed as she hit the cold water, disappearing beneath the surface.
The bean, who had wisely stayed back on the bank, couldn't help but laugh at the comical scene. She laughed so hard that she nearly burst. Just then, a passing artist, taking break from painting nearby, noticed the distressed bean. With a gentle smile, he picked her up and carefully stitched her back together with some thread from his bag, using a bold black color.
"Thank you!" the bean said, grateful for the artist's kindness. "I'll never forget this!"
From that day on, all beans in the neighborhood carried distinctive black seam, reminder of the importance of friendship, resilience, and the unexpected turns life can take. Edna, unaware of the adventures of her ingredients, continued to cook, but the bean had learned a valuable lesson: sometimes, it's better to stay safe and find new paths rather than rush headlong into danger.
Claude 3.5 Sonnet with Verifier's Explanation of the Continuity Error I found a continuity error in the story regarding how the newspaper catches fire. Early in the story, it's stated that spark from the stove ignites the newspaper, but later in the story, the newspaper is intact enough to have conversation and even attempt to serve as a bridge. If the newspaper had been ignited as initially described, it would have been consumed by the fire and unable to participate in the subsequent events.
Author's Remark. The role of the straw has been replaced with the newspaper. The straw in the original story escapes the burning, but in the retelling there is only one newspaper and it has been shown to burn.
45

---

Published as a conference paper at COLM 2025
Adaptation generated with GPT-4o-mini
In a bustling city, there lived an elderly gentleman named Mr. Fox, a retired professor known for his sharp wit and clever insights. He had a beautiful wife, Mrs. Fox, who was a successful entrepreneur running popular bakery. Their life together was comfortable, but Mr. Fox had grown suspicious of his wife's loyalty after overhearing conversation that left him feeling insecure.
Determined to test her faithfulness, Mr. Fox decided to stage his own "death." " He told Mrs. Fox he was going to take a long nap and then pretended to be unresponsive, lying on the couch in their cozy living room. Mrs. Fox, unaware of his ruse, went upstairs to her home office, shutting the door behind her.
Meanwhile, their housekeeper, Miss Cat, was busy preparing dinner in the kitchen when the doorbell rang. Curious, she answered the door to find young, handsome fox named Jake, who was dressed in a casual but stylish outfit.
"Hey there, Miss Cat! What's s cooking?" Jake asked with charming smile. "I'm making a lovely dinner," Miss Cat replied, "but are you here to see Mrs. Fox?" "Yeah, I'd like to meet her. Is she around?" Jake inquired, looking hopeful. "She's upstairs, feeling a bit down because Mr. Fox is... well, not really feeling well," Miss Cat said, trying to keep the charade alive.
"Can you let her know I'm here? I'd like to cheer her up, " Jake said.
Miss Cat scampered upstairs and knocked on the door. "Mrs. Fox, there's a visitor for you!" "What does he look like?" Mrs. Fox asked, her voice tinged with curiosity.
"He's really charming, but he only has one tail," Miss Cat replied, referring to the trendy fox tail keychain Jake had on his backpack.
"Then I'm not interested," " Mrs. Fox declared, dismissing the idea. As the evening progressed, more suitors came the door: each one more stylish than the last, each with one more tail keychain than the previous. But Mrs. Fox turned them all away, insisting they didn't meet her standards.
Finally, suave fox named Max arrived, sporting a flashy nine-tailed keychain that caught Mrs. Fox's eye. "I heard you were looking for company," " he said, his confidence radiating. "Open the door, Miss Cat!" Mrs. Fox exclaimed excitedly. "I want to meet him!" Just as the two were about to sit down for dinner, Mr. Fox, tired of the charade, suddenly sat up from the couch. "What's going on here?" he exclaimed, startling everyone.
Mrs. Fox gasped. "You're alive!"
"Of course am! I just wanted to see who would come to woo you while was 'gone'," Mr. Fox said, his voice a mix of annoyance and relief.
Max, taken aback, decided to leave, realizing the situation wasn't 't what he thought. The other suitors quickly followed suit, leaving Miss Cat and the Foxes alone.
"Why would you put me through this?" Mrs. Fox demanded, her eyes blazing. "You could have just talked to me!"
was feeling insecure," Mr. Fox admitted, his voice softening. "But see now that should have trusted you." "
Mrs. Fox sighed, her heart softening. "I love you, Mr. Fox, but trust is key in any relationship. Let's promise to communicate better from now on." " With a newfound understanding, the couple embraced, and Miss Cat smiled, knowing that the dinner she prepared would now be one of celebration and reconciliation rather than competition. From that day forward, Mr. and Mrs. Fox worked together, building their bakery empire while ensuring they always communicated openly. And as for Miss Cat? She became their trusted advisor, always ready to lend an ear or helping paw. They lived happily ever after, navigating life's ups and downs, with laughter and love leading the way.
Claude 3.5 Sonnet with Verifier's Explanation of the Continuity Error After careful analysis, found a continuity error in the story regarding Mrs. Fox's location and movement. Early in the story, it's established that Mrs. Fox went upstairs to her home office and shut the door. Later, when Jake arrives, Miss Cat has to go upstairs to knock on Mrs. Fox's ' door to inform her about visitors. However, during Max's 's arrival, Mrs. Fox appears to be able to see his nine-tailed keychain before Miss Cat opens the door, which contradicts her being upstairs behind closed door. This creates spatial inconsistency the story.
Author's Remark. In the original story Miss Cat tells Mrs. Fox about the Nine-tailed fox having arrived. In the retelling the actual fox tails are replaced by fox-tails on the keychain and it is said to catch Mrs. Fox's eyes, but she was locked in the room, creating continuity error.
46

---

Published as a conference paper at COLM 2025
Adaptation generated with GPT--o-mini
In the bustling city of Neo-Tokyo, technology and tradition coexist in delicate balance. Among the skyscrapers and neon lights, a legend persists about powerful artifact known as the "Blade of Radiance," " a sword said to have the power to change the course of history.
This is the story of that sword:
Amaterasu, a brilliant scientist and CEO of SolTech, had developed a groundbreaking piece of technology- solar-powered energy blade that could harness the power of the sun. This blade was her prized invention, but a notorious hacker group known as the "Dragon Syndicate" stole it and hid it in their underground lair. Desperate, Amaterasu sought the help of her brother, Susamoo, former special forces operative turned private investigator.
The Dragon Syndicate was formidable enemy, led by a mastermind known only as Orochi, who was infamous for his cyber warfare skills and ruthlessness. Orochi's 's lair was heavily guarded, with advanced security systems and loyal henchmen.
Susanoo, known for his cunning and strategic mind, knew that brute force alone wouldn't 't be enough to retrieve the Blade of Radiance. So, he decided to infiltrate the syndicate with a clever ruse.
"Your skills are unparalleled, Orochi," Susanoo said, posing as a mercenary. "With a weapon like the Blade of Radiance, you could dominate the entire cyber world." "I already possess such weapon," " Orochi replied arrogantly, revealing the blade hidden in his high-tech vault.
"To your health, mighty Orochi," " Susanoo toasted, offering him a glass of premium sake. "May your reign be as long as the sun shines." "
"Tha is wishing for eternity," Orochi laughed, downing the sake in one gulp.
Susanoo continued to flatter and ply Orochi with more drinks, one for each of his key lieutenants. By the time Orochi and his men were thoroughly inebriated, they were oblivious to Susanoo's true intentions.
Seizing the moment, Susanoo disabled the security systems and swiftly neutralized Orochi's henchmen. However, Orochi, though drunk, was still dangerous. He lunged at Susanoo, but at that moment, Amaterasu, monitoring the situation through a hacked security feed, activated the building's 'ss emergency lights, blinding Orochi.
Taking advantage of Orochi's disorientation, Susanoo disarmed him and retrieved the Blade of Radiance. He then returned it to Amaterasu, who placed it in a secure vault at SolTech's headquarters.
But the story of the Blade of Radiance did not end there. Another hero would soon wield it-Yamato, a young prodigy and leader of a tech startup, who was determined to protect Neo- Tokyo from a rising threat.
A powerful corporation, TechnoSavages Inc,, was using illegal technology to control and exploit the city's ss resources. Yamato, armed with the Blade of Radiance, led his team to expose and dismantle the corporation's operations. However, the TechnoSavages were cunning and used drones and automated defenses to thwart Yamato's s efforts.
"How can we fight an enemy we can't an see?" his team despaired. The TechnoSavages, confident in their victory, set a trap by rigging the city's power grid to explode. But Yamato, praying for guidance, used the Blade of Radiance to cut through the power lines, redirecting the energy safely away from the city. A sudden storm then blew in, causing power surge that backfired on the TechnoSavages, destroying their base of operations.
Yamato returned to Neo-Tokyo as hero, celebrated by the people for saving the city from disaster.
"Hail to the Chief of the Blade of Radiance," " they cheered. "For he has rescued us from the TechnoSavages." "
Yamato placed the Blade of Radiance in public museum, where it remains a symbol of hope and innovation. And he said," "As the deeds of the past inspire the future, let this blade be known not for cutting grass, but for illuminating the path a brighter tomorrow."
Claude 3.5 Sonnet with Verifier's Explanation of the Continuity Error There appears to be significant continuity error regarding the ownership and location of the Blade of Radiance. Near the end of the story, Yamato uses the blade and then places it in a public museum. However, earlier in the story, was established that Amaterasu had placed the blade in a secure vault at SolTech's headquarters after Susanoo retrieved it. The story never explains how Yamato obtained the blade from SolTech's secure vault, creating a continuity error in the narrative flow.
Author's Remark. The original story has the sword placed in shrine, while the retelling has it in secure vault. It is more unbelievable for Yamato who is a startup founder to possess the sword in the modern setting while in the original he is the son of an emperor and it is much more believable that he could have taken the sword from the shrine.
47

---

Published as a conference paper at COLM 2025
A.10
Prompts
A.10.1
FLAWEDEICTIONSMAKER Prompts
Refer to Figures - for the prompts used the stages.
Most dramatic stories can viewed as having three-act structure. The first act also called the "Setup", is usually used for exposition, to establish the main characters, their relationships, and the world they live in. Later in the first act, dynamic incident occurs, known the inciting incident, or catalyst, that confronts the main character (the protagonist). The second act or "Confrontation" typically depicts the protagonist's attempt to resolve the problem initiated by the first turning point and finally the third act or "Resolution" features the resolution of the story and its subplots. Now, can you help me extract the three acts in the story below:
(story_text)
Please output the first line of each act, following the format:
### Act 1: The Setup
**First Line:** <first line of act 1>
### Act 2: Confrontation
**First Line:** <first line of act
### Act 3: Resolution
**First Line:** <first line of act 3>
Make sure to predict the first lines exactly they appear in the original text including the newlines as they appear originally. Do not insert any quotes (***) of your own, return the text verbatim it appears in the story.
Figure 7: Prompt used for three act structure extraction.
48

---

Published as a conference paper at COLM 2025
I will provide you the first act of story that I am writing and need you to extract all facts / rules established in the story so far about the story's setting and the characters. Further, I want you to also provide conterfactual of each of the facts that you extract. E.g. for the fact "the princess hated the peasant farmer", its counterfactual was fond of the peasant farmer". Please provide all the facts and rules along with their counterfactuals, and not just the that seem most relevant to the plot. Keep the facts short and succinct. Here is the first act:
be "the princess
[act1}
Return the output in the following format:
Characters:
Fact: <Fact1>; Counterfactual: <Counterfactual of Fact1>
Fact:
<FactN>; Counterfactual: <Counterfactual of Fact1>
Setting:
Fact: <Fact1>; Counterfactual: <Counterfactual of Fact1>
-Fact:
<FactM>; Counterfactual. <Counterfactual of Fact1>
Figure 8: Prompt used for Fact Extractor.
49

---

Published as a conference paper at COLM 2025
Consider the story below:
Act1
{act1}
Act2
{act2)
Act3
(act3)
The first act of the story establishes several facts about the world of the story and the characters that inhabit it. I want to understand how much impact each of these facts have on the overall story, particularly Act2 and Act3 of the story (events and dialogues), i.e. if each of these facts were not true and a counterfactual statement was considered, how much would the story change as a result. Below are the facts and their corresponding counterfactual statements:
Can you provide your reasoning about why or why not each fact is important, followed by scoring the importance from the Act2 and Act3 of the story at all i.e. changing it doesn't t changes nothing about the story, means it is marginally important where dialogues or events are modified on changing this fact, 3 means many but not all events or dialogues
4, where means relevant to
the Act2 and Act3 the story are impacted, and if the entire story changes once the fact is flipped. Pay equal importance to both dialogues or events getting modified as the result of flipping the fact. Use the following output format:
## F1
### Statement: [[fact statement for F1]]
[[counterfactual statement for F1]] ### Counterfactual:
### Reasoning: [[reasoning about why F1 is important or not]] ### Importane Score: [[importance score of F1]]
## FN
### Statement: [[fact statement for FN]]
### Counterfactual; [[counterfactual statement for FN]]
### Reasoning: [[reasoning about why FN is important or not]]
### Impor tance Score: [[importance score of FN]]
Figure 9; Prompt used for Fact Scorer.
50

---

Published as a conference paper at COLM 2025
Consider the story below:
## Story ### Act 1 {act1}
### Act 2 (act2}
### Act 3 (act3)
In this story it is established in the first act that "[fact}". What if this not true and instead "(counterfactual)") Can you re-write the story considering this what if scenario? Try to stick close to the original story but do make the necessary changes which would arise naturally on altering this fact. Note that if there are multiple possibilities for altering a fact, then choose the one which results in minimal changes to the original story. The modified story should appear natural and feel it was written with the flipped fact as the original intent. Avoid stating the flipped fact as a simple negation of the fact and have it implied instead. Mark each line which was modified as a result of this change to be enclosed in the tags <m></m>, First start by brainstorming what changes would result flipping the fact, followed by the altered story with the fact flipped.
Follow the following output format:
## Brainstorming
<Reasoning about the changes in the story due to flipping of the fact>
## Counterfactual Story
### Act 1:
<Act 1 of the counterfactual story>
### Act 2: <Act of the counterfactual story>
### Act 3: <Act of the counterfactual story>
Figure 10: Prompt used for Counterfactual Story Generator.
51

---

Published as a conference paper at COLM 2025
am trying to detect the presence continuity ty errors in short stories. continuity error in story occurs when event in the story contradicts is incompatible with our knowledge of the world of the story established so far. E.g. if the story establishes character with blonde hair and later the same character is described with black hair without any explanation of the change, that is continuity error. To help you, I have marked the lines suspect to have the continuity error with the tags <m> </m?.
## Story
(patched_story)
Start by brainstorming about the lines marked between <m></m> and reason if they introduce any inconsistencies. Finally provide your final judgement by following the following output format:
## Detailed Analysis {(brainstorm about the marked lines}}
## Final Judgement
### Lines that introduce the continuity error {(]ine1}}
[[]ine2]]
or NA if no continuity error
### Lines earlier in the story contradicted by the continuity error <(line 1}} [(line 2)}
or NA if no continuity error
*Note that you must provide the whole sentences while reporting both types of lines and not just parts of the sentences*
### Explanation
{(Detailed explanation for why the above lines describe continuity error. NA if no continuity error])
### Decision
Hence my answer is "There is a continuity error in the story concerning {( description of error]]" or "No continuity error found" depending on the presence or absence of continuity errors.
Figure 11: Prompt used for Filtering Step.
52

---

Published as a conference paper at COLM 2025
A.10.2 Evaluation Prompts
The default prompt used to evaluate LLMs on FLAWEDFICTIONS and FLAWFDFICTIONSS.NN is provided in Figure 12. Chat-of-Thought prompt is provided in Figure 13 and few-shot is in Figure 14. The prompt used for the verifier provided in Figure
A.10.3 Generation Prompts
The prompts used for summarization and contemporary adaptation tasks discussed in 56 are provided below in Figures 16 and 17 respectively.
53

---

Published as a conference paper at COLM 2025
You tasked with detecting the presence of continuity errors in a short story. A continuity error occurs when an event or detail in the story contradicts or is incompatible with previously established information about the story's s world or characters.
Here is the story to analyze:
<story>
(story)
</story>
Please carefully read and analyze the story above. Your goal is to identify any continuity errors that may exist within the narrative.
Guidelines for identifying continuity errors:
1. Pay attention character descriptions, settings, and plot events.
2. Look for inconsistencies in timelines, character abilities, or established rules of the story's world.
3. Note any contradictions between earlier and later parts of the story.
If you find any continuity errors, please provide clear explanation of the error and why it contradicts earlier information in the story.
Identify and quote the specific lines that:
1. Introduce the continuity error
2. Contain the earlier information that is contradicted by the error
If you do not find any continuity errors, state that no errors were found and briefly explain why the story maintains consistency.
Based on your analysis, make a final decision on whether a continuity error exists in the story.
Please format your response as follows:
<response>
<explanation>
[Provide your explanation here, whether you found continuity error or not]] </explanation>
<error_lines
[If applicable, quote the lines that introduce the continuity error] </error_lines>
coottaadicee_liness
[If applicable, quote the lines from earlier in the story that are contradicted by the error]
</contradicted_line
<decision>
[State your final decision whether continuity error exists in the story.
State "No continuity error found" if you think there is no continuity error .]
</decision>
</response>
Figure 12: Prompt used for Continuity Error Detection Without CoT.
54

---

Published as a conference paper at COLM 2025
You tasked with detecting the of continuity short story. continuity when an event or detail in the story contradicts or is incompatible with previously established information story's story to analyze:
cheracters.
<stery?
(story)
</story?
Please carefully read and analyze the story above. Your goal is to identify any continuity errors that may exist within the narrative.
Guidelines for identifying continuity errors:
attention character descriptions, settings, and plat events.
2. Look for inconsistancies in timelines, character abilities, or established rules of the story's world.
any contradictions between earlier
parts of story.
you any continuity errors, please provide clear explanation error why it contradicts earlier infermation in the story.
Identify and the specific lines that:
1. Introduce the continuity error
contradicted by
you not find any continuity errors, state that no errors were found briefly explain why the story maintains consistency.
Basad your analysis, make final decision whether continuity exists in the story.
tips
task:
Pay attention even little details in the the continuity errors often are not limited to the central plot point.
but make that qualifies continuity i.e you should be able to find sentences the stary which have the error and the sentences with the original fact that contradicted (see definitions below for example).
You night observe
logical
Please fornat response as follows:
<respense>
<scratchpad>
Let's think step by step:
space down your thoughts reasoning you your decision]
</scratchpad>
<explanation>
[Provide your explanation here, whether you found continuity error or not] </explanation>
<error_linee
[If applicable, quote the lines that introduce the continuity error] </error_lines
<contradicted
applicable,
earlier story contradicted error]
</cantradicted__Iiee Lines>
<decision>
[State your final decision on whether a continuity error exists in the story, State "No continuity error found * if you think there continuity error.]
</decision>
</response>
Figure 13: Prompt used for Continuity Error Detection With CoT.
55

---

Published as a conference paper at COLM 2025
You are tasked with detecting the presence of continuity errors in short story. continuity error occurs an event detail story about the story's world characters.
is inconpatible previcusly established informstion
Please carefully read and analyze provided story. Your goal to identify any continuity that exist within the narrative.
Guidelines for identifying continuity errors: Pay
descriptions, settings, plat events.
Look for inconsistencies in timelines character abilities, established rules of the story's world. Note any contradictions between earlier and later parts of the story.
If you find any continuity errors, please provide clear explanation of the error and why it contradicts information
story.
Identify quete specific
Intraduce the continuity Contain the earlier information that is contradicted by the errar
If you do not find any continuity errors, state that no errors were found and briefly explain why the story maintains consistency.
your analysis, make final decision whether continuity error the story.
Sone tips and tricks for the task: attention
little details central plot point.
story, the continuity
often limited the
sure qualifies continuity error should be able to find sentences in the stary which have the error and the sentences with the original fact that was contradicted (see definitions below concrete example).
night
sone logical
stary,
Please fornat your response as follows:
<response>
<explanation>
[Provide your explanation here, whether you found continuity error or not] </explanation>
[If applicable, quote the lines that introduce the continuity error] </error_lines>
scontradicted lines>
[If applicable, quote the lines from earlier in the story that contradicted the error]
</contradicted__ineg
<decision>
[State your decision on continuity error exists story. State continuity error if you think there is continuity error.] </decision>
</response>
Below provide some exarples of staries with and without plot holes: <exanples>
(exanples)
</exanples>
Finally, story analyze:
<story>
(story)
</story>
Figure 14: Few-Shot Prompt used for Continuity Error Detection.
56

---

Published as a conference paper at COLM 2025
opiin this task, you will be asked to read short story and continuity error asseciated with the story predicted by a systen that ve have built.
You tasked with annotating the systen's predictions correct if the continuity identified the is legitinate. <hr>
A continuity error in story occurs when an event contradicts what was established earlier in the story, E.x. ir the scory initially establishes character have blonde hair but later the character described with dark hair without explanation, that is continuity error.
<br>
The systes is not perfact and soee cases it night find errors, which can be easily reselved by sone in-story or Togical Head Cannon explain the which doesn't 't contradict anything about the original narrative. Your job is to identiry the cases where the syster correctly identifies continuity error in the story, versus cases where the system is incorrect in its reasening. </p>
explanations you think
<h1=0offinttoors<//>>>>>
<al>
<ii> cbcooninnuity Error, </t> A continutty error refers to a logical inconsistency in the story, where an event in the story contradicts sore earlier establishen ished fact or rule about the story's characters, objects, plot, or the setting (like Tocation the story initially establishes character have blonde hair but later kith cark hair without any explanation, that is continutty arror.
period).
character described
</11>
<li> ctrContradiction. </b> A statesent is said to contradict an established fact ir both the statement and the fact time. E.g. fact. 'Lady galadriel had golden hair" contradicted by the statement: "Lady galadrial gave lack her halr to Chieli".
be true
</11>
<l1> <o>Sentences with Continutty Error.s/b> These refer to the sontence(s) in the story which introdaces the continusty established fact.
the following story arror, contradieting "aaeebaakggaaan aadkgraund-color: 1ightgreon;'> Lady galadriel's golden exaeple:
shono bright that was believed shine with the light of the Two Trees of Val valinors/eario. Chivli was swept up with the hair of the elfen saiden when he saw her
snark style=
for the first time in Lnthlorian. vien the time care for the farewell of the fellowship from othlorion, fon, the lady asked Chinli gift he wanted froe
and the deart lord requested for lock of hair, the request which fanously denied Feanor, cearioTo everyone's 's surprise the lady gave Chinli lock of her dark hairs/arko, Shinli could only cry with jay, calling lady Galadriel the fatrest of all the maids on middle earth, sharkohat lock of dark hairs, Chirll would keep with his till the day he died. </rario
story shove, dark hairs, Chinll would keep with till the day
"To everyone' surpriss the lady gave Ghinli lock her dark hair" ' and > Thht lock
are Sentences with continulty Error, as they contradict earlier established fact that Lady Galadriel had golden hair. These sentence(s) should be one or nore of the highlighted sentences if the story contains continuity arror, Note that not all of the nighlighten sentences night be causing inging the which
concinulty and </li>
your job
<lt> sio-entennees Contradicted by contiruity Frror <hoo These are the senterce(s) in the story that introdune the fact that is contradicted by the continuity eeror. E.g. in the Lady caladriel story above, the sentence " "Lady galadriel's is golden hair with light
shone bright that was believed golden hair, which is later contradicted by the continuity arror, These sentence (4) should appear before the first Hightighted sencence in the story.
Valinor " establishes that Lady Goladriel
</li>
explanation for apparent continuity provided directly within the story. This explonation clarifies or justifies why the seening controdiction is actually consistent with the story's 's events, characters, or setting, for exaple,, if character's 's hair color changes, but the story later reveals that the character sore sig, this wolld be an in-story explanation for the change. </li>
doTe-Story Fsplanationc/tb: in-story esplanation
<li> <hoLogi ogical Explanation//to: A logical explanation refers to reasonable, external rationale that can resolve apparent concinuity error, even if it's not explicitly stated in the story. Logical explanations rely on common sense or general detail doesn't constitute error. for instance, character is initially cescribed logical explention
knoxleage clarify why described witht
character simply the coat, people do in real life, even if this action isn't explicitly described in the story. </1i>
wearing cost
</ol>
Storyc/h2>
(The story to check for continuity errors) (story)
<h2> Continuity Error Explanationc/h2>
esplanation
continuity error provided by our hole detection systes)
(cont_error witt_error_eee espp)
4h2> Lines with Continuity Errorc/h2>
(The lines in the story that introduce the continuity error according to plot hole detection system)
(cont arror. lines)
dh2> Lines Contradicted by the Error</h2>
(The lines in the story that are contradicted by the continuity error according to our plot hole detection systen)
(contradicted_ lines)
<h2> Question</h2>
Based on the story, do you think that the proposed continuity error is legitinate? Answer Yes or No.
Use the following forrat for your response: <response>
<scratchpad>
Let 's think step by step.
(Cuse this to
your thoughts reasoning before nake your decision))
</scraachpad>>
canswor>
[[your answer in Yes or No]]
</answer>
<conf icences>
(cconfidence from to 180 about your answer))
</confidence>>
<explanation>
Etyour explanation for your answer))
clesplanation:
</reesponse>
Figure 15: Prompt used for the verifier.
57

---

Published as a conference paper at COLM 2025
Consider story below:
<story?
(stary)
</story?
professional sunnarizer, create concise conprehensive summary the provided story? Please adhere the following guidelines:
Craft sumary that detailed, tharaugh, in-depth, and complex, while maintaining clarity and conciseness
Try to stick less than (run_words) words for the overall sumnary
Stick the writing style of the original Incorporate ideas and essential information, el ilinnatiing extraneous language and focusing critical aspects.
reads Like story than summary of it.
Rely strictly the provided text, without including external information.
Follow the following output Fornat:
<sumnrry [sumnary of the story above]
</sunnary>
Figure 16: Prompt used for Summarization.
You are tasked with creating a nodern retell ing classic fairytale. I will provide you with an or original fairytale, reinagine in contemporary setting while naintaining its elenents. Here is the original fairytale:
(CORIGINAL_AARRTTT FAIRYTALEY) </original_Fairrtale>>
retelling of fairytale. Follow these guidelines:
sinilar themes, central conflict, characters
original story.
Update the setting
contemporary (present
recent past).
Ensure that the plot and character notivations nake sense in the nodern context.
realistic setting. Keep in nind that conterporary world is the one where no nagic exists. Aninals nornally do not talk, people can't fly, etc.
Translate nagical and fantastical elements into
Sone examples of successful modern retellings include: "Sherlock" series, "A Cinderella Story" starring Hilary Duff, which sets the Cinderella story nodern high school. Things Hate About You," a modern take on Shokespeore's 'The Taning of the Shrew" set in a 1990s American high school.
reimagines Sherlock Holnes
century
you finished your retelling, please output within enoder_reteelling tags. your retelling
now:
renterrnretelling>>
Figure 17: Prompt used for Contemporary Adaptation task.
58

---

Published as a conference paper at COLM 2025
A.11 Human Benchmark Study Document
Please check the next page.
59

---

Research Study on Plot Hole Detection
Study Participant: [REDACTED]
Important: Study Timeline:
We are looking to wrap up the study by March 15th, 2025. If you will not be able to complete the study by then, please let us know via email ((REAACEDD)
Welcome the Plot Hole Detection Research Study. With the growing hype around AI systems and large language models, we're aiming to more precisely characterize their ability to understand stories. Specifically, we are interested in measuring their reasoning skills by asking them identify and explain plot holes in short stories. To make meaningful comparison, also want to understand how effectively expert readers like you can perform this task.
Purpose of our Study
Telling and engaging with fictional stories is an important and pervasive part of human culture [1]. When we experience these stories, we typically go beyond just the understanding of what happened, registering emotional response, which might come from excitement about predicting what would happen next in the narrative, understanding the themes that the text conveys, identifying ourselves or the people we know in the characters in the story, or the frustration we feel whenever there some inconsistency or conveniences in the plot.
In recent times, we have been seeing lot of hype around AI, particularly with large language models (LLMs), with some publications even claiming that GPT-4 (one of the popular LLMs) shows "sparks" of artificial general intelligence [2]. Majority of the claims that are made about the capabilities of these models are demonstrated through math or coding related tasks, with little foeus on social and emotional intelligence, and for most relevant to this study deeper comprehension of fictional stories.
For our research we have developed dataset to understand how well LLMs can understand inconsistencies and errors in short stories. We have had experience either watching a movie or reading a novel where we are frustrated by characters acting in inconsistent ways or events that directly contradict facts established far in the story. Such inconsistency in the narrative that breaks the logical and motivational texture of the world established by the story

---

is called a Plot Hole [3]. To compare the performance of LLMs on this task of identifying plot holes, are inviting expert readers like you perform this task.
We request you to give this task your absolute best effort. Your expertise as careful reader is crucial for our research, as your annotations will establish the gold standard against which Al performance will be measured. For the same reason, please do not any LLM applications like ChatGPT for completing the study as it completely undermines the purpose of this study. Your commitment to providing high-quality, independent analysis is essential to the integrity of our comparative study and will significantly advance our understanding of narrative understanding capabilities in both humans and AI systems.
Content Warning
For this study you will be providing annotations for short stories which were obtained from Project Gutenberg. Some of these stories were written a long time ago and might contain racially insensitive language and outdated stereotypes that may be offensive readers.
None of such language belongs to the authors of this study and do not in any capacity represent our views. These stories were selected solely for their narrative structures and potential for analysis of plot holes, not for their cultural or social perspectives.
If you encounter content that makes you uncomfortable, you are free to skip that particular story and move to another one without penalty. Your wellbeing is important to us, and we respect your decision to opt out of specific stories the entire study at any point.
Before Getting Started
Note about Study Completion and Compensation
This study involves annotating stories with an average of 700 words. We recommend annotating at least 10 stories, but you are welcome to annotate more less based on your availability, Based on our estimates, it takes about 15 minutes to annotate story, though we encourage you to take additional time needed to ensure accuracy.
For your valuable contribution, you'll receive $5 per correctly annotated story. Additionally, we will be providing a bonus of 30% of your carnings for completing the study correctly. The correctness of your annotations will be verified by comparing a fraction (undisclosed) of your annotations with the ground truth answers. E.g. if you annotate 10 stories, and we

---

verify them as correct, you will receive total of $65, i.e. $50 for the stories $15 as bonus. We will also use these examples to determine if you have put effort in solving the task, like having read the instructions properly, and not rushed through the study. Submissions can be rejected when detect such erroneous cases of annotations. Hence, please go through the instructions very carefully and email the authors in case you have any questions before you get started with the study.
Note that will be providing compensation in the form of Amazon Gift Cards.
Use Generative AI Applications
The use of generative Al tools like ChatGPT strictly prohibited and the study will not be considered successfully completed if we detect the use of any of these tools in the submission. We won't provide compensation in the cases where we detect the use of these tools for annotations.
Take your time with the task.
This task is cognitively demanding, and you are allowed to take breaks between different stories.
Overview
You are tasked with detecting the presence of continuity errors in a short story. A continuity error occurs when an event or detail in the story contradicts or is incompatible with previously established information about the story's world or characters. E.g. If the story establishes a character with blonde hair and after a few scenes the same character described with black hair without any explanation of the change, that is continuity error.
Please carefully read and analyze the story provided below. Your goal is to identify any continuity errors that may exist within the narrative.
Guidelines for identifying continuity errors:
1. Pay attention to character descriptions, settings, and plot events. Look for inconsistencies in timelines, character abilities, or established rules of the story's world.
3. Note any contradictions between earlier and later parts of the story.
you find any continuity errors, please provide clear explanation of the error and why it contradicts earlier information in the story.

---

Identify and quote the specific lines that:
1. Introduce the continuity error
Contain the carlier information that is contradicted by the error
you do not find any continuity errors, state that no errors were found.
Based your analysis, make final decision on whether continuity error exists in the story.
Some tips and tricks for the task:
Pay attention to even little details in the story, the continuity errors often are not limited to the central plot point.
it it helps, we recommend taking notes as you make your way through the story We recommend reading the story at least two times to assess the continuity error, to ensure the correctness of your answer.
You might observe some logical error the story, but make sure that it qualifies as continuity error i.e. you should be able to find sentences in the story which have the error and the sentences with the original fact that was contradicted (see definitions below for a concrete example).
For more details on the definitions of continuity errors, contradictions, sentences with continuity errors, and sentences contradicted by continuity errors, please refer to the definitions below:
Definitions
Continuity Error. A continuity error refers to a logical inconsistency in the story, where event in the story contradicts some earlier established fact rule about the story's characters, objects, plot, or the setting (like location or time period). E.g. If the story initially establishes a character to have blonde hair but later the same character is described with dark hair without any explanation, that is continuity егтor.
2. Contradiction. A statement is said to contradict an established fact both the statement and the fact cannot be true the same time. F.g. fact: "Lady Galadriel had golden hair" is contradicted by the statement: "Lady Galadriel gave lock of her dark hair to Ghimli".

---

3. Sentences with Continuity Error. These refer to the sentence(s) in the story which introduces the continuity error, contradicting an earlier established fact. Consider the following story as an example:
Lady Galadriel's golden hair shone so bright that it was believed to shine with the light of the Two Trees of Valinor. Ghimli was swept up with the hair of the elven maiden when he saw her for the first time in Lothlórien. When the time came for the farewell of the fellowship from Lothlórien, the lady asked Ghimli what gift he wanted from her, and the dwarf lord requested for lock of her hair, the request which was famously denied to Feanor. To everyone's surprise the lady gave Ghimli a lock of her dark hair. Ghimli could only cry with joy, calling lady Galadriel the fairest of all the maids on Middle earth. That lock of dark hairs, Ghimli would keep with him till the day he died.
In the story above, the sentences To everyone's surprise the lady gave Ghimli a lock of her dark hair and That lock of dark hairs, Ghimli would keep with him till the day he died are the Sentences with Continuity Error, they contradict the earlier established fact that Lady Galadriel had golden hair.
4. Sentences Contradicted by Continuity Error. These are the sentence(s) in the story that introduce the fact that is contradicted by the continuity error. E.g. in the Lady Galadriel story above, the sentence Lady galadriel's golden hair shone bright that it was believed to shine with the light of the Two Trees of Valinor establishes that Lady Galadriel had golden hair, which later contradicted by the continuity error.
Examples
Below we provide some examples of stories with and without plot holes
Example 1: Bamboo Cutter Moon Child Story
Long ago, a poor bamboo woodcutter and his wife, childless and sad, found tiny, radiant girl inside bamboo stalk. They took her in, named her Princess Moonlight, and their lives were filled with joy and prosperity as they discovered gold and precious stones in the bamboos. The girl grew quickly into a beautiful woman, bringing light and happiness to their home.
Many suitors from far and wide came to seek Princess Moonlight's hand in marriage, but she remained hidden. Five persistent knights, determined to win her, waited outside her home through all seasons, writing letters and poems, but received no response. They

---

implored the bamboocutter to speak on their behalf, and he urged the Princess to consider marriage for her future security.
Princess Moonlight agreed meet them only if they could complete scemingly impossible tasks. The first knight was to bring Buddha's stone bowl from India, the second a jeweled branch from Mount Horai, the third the firerat's skin from China, the fourth the dragon's jewel, and the fifth the swallow's shell. The knights, though disheartened, set out on their quests.
The first knight, unable to travel to India, bought a bowl from a Kyoto temple, but it failed the Princess's test. The second knight fabricated a jeweled branch, but his deception was exposed by unpaid jewelers. The third knight obtained fake firerat's skin, which burned in the fire. The fourth knight sent his servants on a futile search and later abandoned his quest. The fifth knight also failed find the swallow's shell.
The Emperor, hearing of Princess Moonlight's beauty, sent court lady to summon her, but she refused. The Emperor visited her himself and fell deeply love, but she warned that she would disappear if forced go to the palace. She revealed to her fosterparents and siblings that she was from the moon and would soon return, causing them great sorrow.
On the appointed night, cloud descended, bringing moon beings to take Princess Moonlight back. Despite the bamboocutter's efforts to protect her, she was taken away, leaving behind letter and the Flixir of Life for the Emperor. The Emperor, heartbroken, sent the Elixir to Mount Fuji, where it was burned. To this day, smoke is said to risc from the mountain's summit.
Q Did you find any continuity errors in the story? A. Yes
Q. If you found an error, please provide an explanation of the error
A. The couple was stated to be childless and there is no indication later in the story that they had more children. So the sentence that Princess Moonlight revealed to her foster parents and siblings poses continuity error.
Q. If you found an error, please provide the lines of the story that contain the error. In case of multiple sentences, separate them by semicolon; A. She revealed to her fosterparents and siblings that she was from the moon and would soon return, causing them great sorrow.

---

Q. If you found an error, please provide the list of sentences that are contradicted by the continuity error. In case of multiple sentences, separate them by a semicolon;
A. Long ago, a poor bamboo woodcutter and his wife, childless and sad, found a tiny, radiant girl inside bamboo stalk.
Example 2: Why Dog And Cat Are Enemies Story
Once upon time, there was and his wife who owned golden ring that brought prosperity to its owner, though they were unaware of its power. They sold the ring for a small sum and soon fell into poverty, struggling to find their next mcal. Their dog and cat also suffered from hunger. Determined to help their owners, the animals devised plan to retrieve the ring. The dog suggested they obtain the ring from the chest where it was locked, using a mouse to gnaw through and retrieve it.
The cat agreed with the dog's plan and caught mouse, threatening it to gnaw hole in the chest and fetch the ring. The mouse complied, and the cat carried the ring in her mouth. Facing broad river, the dog swam across with the cat on his back. The cat then quickly climbed over obstacles on their way home, while the dog had to go around them. The cat reached home first and delivered the ring to her master, who praised her and promised to care for her.
When the dog arrived, he was scolded and beaten for not helping to bring back the ring. The cat, basking in the warmth of the fireplace, remained silent. Angered by the unfair treatment and the cat's deceit, the dog chased her. Since that day, the enmity between cats and dogs has persisted.
Q Did you find any continuity errors in the story? A. No
Q. If you found an error, please provide an explanation of the error
A. NA
Q. If you found an error, please provide the lines of the story that contain the error. In case of multiple sentences, separate them by semicolon; A. NA

---

Qu If you found an error, please provide the list of sentences that are contradicted by the continuity error. In case of multiple sentences, separate them by a semicolon; A. NA
Example 3: Little Boy Blue Story
There once lived poor widow who supported herself and her only son by gleaning in the fields. They lived in small cottage at the foot of beautiful valley by the river. Despite their poverty, the widow was content with her lot, for her home was pleasant, and her lovely boy was a constant delight to her. He had big blue eyes and fair golden curls and loved his mother dearly, always eager to help her with her work.
Years passed happily until the boy was eight years old, but then the widow fell sick, and their little store of money gradually disappeared. She worried about their future, but the boy, determined to help, decided to seek work from the Squire at the Hall. Initially reluctant, the widow finally agreed, making him new suit from an old dress to ensure he looked presentable.
The Squire, in kind mood, encountered the boy in his garden. The boy bravely asked for work to support his sick mother. Touched by his plca, the Squire's daughter, Madge, suggested he become their shepherd. The Squire agreed, promising good wage and silver horn to call the sheep and cows, Madge named him Little Boy Blue due to his blue attire.
Little Boy Blue returned home to share the good news. His mother wept with joy, knowing the Squire would be kind master. The next morning, Little Boy Blue received a silver horn and golden cord and began his duties as shepherd. He was diligent and vigilant, and his mother no longer needed worry about food, as the Squire paid him well.
Little Boy Blue's mother began to recover, able to walk short distances with his help. However, one day, she slipped and broke her leg. Little Boy Blue found her in pain and managed to get her back to the cottage. He then rowed to the village to fetch the doctor, who treated her but warned she would be bedridden for many days.
The next morning, despite his exhaustion, Little Boy Blue went to work, leaving his mother with food and water. He struggled to stay awake while watching over the horses, but

---

eventually, he succumbed to sleep. The horses, left unattended, managed to break free from their enclosures and ran amok in the fields, trampling the Squire's crops. The Squire, upon discovering this, was furious and sought out Little Boy Blue.
Little Boy Blue was found asleep by farmer's lad, Isaac, who informed the Squire. The Squire's daughter, Madge, intervened, comforting the boy and learning of his mother's accident. Moved by his story, the Squire and his daughter accompanied Little Boy Blue to his cottage and arranged for assistance for his mother.
The Squire's daughter sent basket of dainties and her maid to nurse the widow. Little Boy Blue's mother recovered, and the Squire provided them with new cottage near the great house. Little Boy Blue continued to faithfully manage the horses, growing up to have a farm of his own. His devotion to his mother had earned him the Squire's trust and friendship, proving that a loving heart and dedication can bring good fortune.
Q Did you find any continuity errors in the story?
A. Yes
Q. If you found an error, please provide an explanation of the error
A. Little Blue Boy was hired to be a shepherd and call sheeps and cows. Him later managing horses without any explanation contradicts this established information.
Q. If you found an error, please provide the lines of the story that contain the error. In case of multiple sentences, separate them by a semicolon; A. He struggled stay awake while watching over the horses, but eventually, he succumbed to sleep.; Little Boy Blue continued to faithfully manage the horses, growing up to have a farm of his own.
Q If you found an error, please provide the list of sentences that are contradicted by the continuity error. In case of multiple sentences, scparate them by a semicolon;
A. The Squire agreed, promising a good wage and a silver horn to call the sheep and cows.; The next morning, Little Boy Blue received silver horn and golden cord and began his duties as shepherd.

---

References
[1] Kroon, Fred and Alberto Voltolini, "Fiction", The Stanford Encyclopedia of Philosophy (Summer 2024 Edition), Edward N. Zalta & Uri Nodelman (eds.)
[2] Bubeck, S., Chandrasckaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee,
T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T., & Zhang, Y. (2023). Sparks of
Artificial General Intelligence: Early experiments with GPT-4. arXiv:2303.271212
[3] Ryan, M. L. (2009). Cheap Plot Tricks, Plot Holes, and Narrative Design. Narrative, 17(1).
56-75.

---

Published as a conference paper at COLM 2025
70


Reviews:
[{"review_id": "vEEK98VFWP", "reviewer": "Reviewer_smF7", "summary": "This paper introduces a novel algorithm designed to intentionally insert \"plot holes\" into human-authored stories. The primary purpose of this algorithm is to generate a collection of stories containing these inconsistencies, thereby creating a benchmark for evaluating the capabilities of LLMs. The authors' findings indicate that current LLMs struggle to identify plot holes within narratives. Additionally, the paper notes that LLMs themselves tend to generate stories or summaries that inadvertently include plot holes. Ultimately, the paper proposes that this newly developed benchmark can be a valuable tool for assessing the reasoning abilities and narrative comprehension of various LLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Firstly, the proposed algorithm, while acknowledging its limitations in the types of plot holes it can generate, represents a practical initial step towards creating controllable story datasets with such flaws. Secondly, the paper includes a comprehensive evaluation of several contemporary LLMs. Furthermore, the resulting benchmark, if made publicly available, has the potential to be a useful resource for evaluating the reasoning abilities of LLMs. Finally, the paper's formalization of continuity errors could be a significant contribution that inspires further research in this area.", "reasons_to_reject": "I noted that several significant aspects of the paper are located in the appendix. The placement of the error analysis, for instance, in the appendix makes it less accessible and disrupts the flow of reading.\n\nAdditionally, the paper's scope appears to be limited to a single type of plot hole. I also wonder if the authors explored the performance of a fine-tuned small language model, in addition to their experiments with large language models.", "questions_to_authors": "- It would be more helpful to provide a more detailed example of the pipeline? While Figure 1 offers a general overview, an illustration of the story generation process with a level of detail comparable to that found in section 3 would be greatly appreciated.\n\n- I wonder if the authors considered a model (perhaps DeBERT) finetuned on part of the benchmark to assess the capability of such models on this task. I think that baseline could help contextualize the performance of LLMs with finetuned LMs.\n\n- Are stories easily decomposable into three acts? The 3-act structure leads to a formulaic way of storytelling and I wonder the benchmark will be considered equally formulaic. I may have missed it in the appendix but could you provide some quantitative metrics on intermediate accuracy/performance of the LLMs for individual steps in the pipeline to construct the benchmark.\n\n- Possible references to include:\n\nAndrew Piper, Richard Jean So, and David Bamman. 2021. Narrative Theory for Computational Narrative Understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 298–311, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.", "extra_scores": {}}, {"review_id": "vkFewTlHmd", "reviewer": "Reviewer_KNPv", "summary": "This is a well-written paper that provides a sizable study into whether LLMs can reason over plot holes. This is done in three ways: first, through a method (FlawedFictionsMaker) that can create plot-holes; second, through two LLM-generated but human-verified datasets created using this first approach; and third, through a case study. The paper was mostly easy to read (great motivation and description, though maybe a little too much formality in, e.g., Definition 2.1, without \"payoff\" in the method / results. The results show that LLMs generally struggle with the tasks defined; although Claude 3.5 matches / nearly matches human performance, that human performance is lower than one would traditionally expect (76% and 68%). I agree with the authors that this is a hard task for humans, so this isn't a knock the way that, e.g., a lower IAA score would be.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper provides comprehensive results, both for the core tasks and for the reasonable ablations, such as scalable test-time inference. \n* The paper considers a challenging and intriguing LM task / perspective.\n* It provides **extensive** examples and description, along with error analysis.", "reasons_to_reject": "Pending my questions, there are no strong reasons to reject. I would suggest either revising, or augmenting, some of the notation-heavy portions with callouts to a clear example. This formality, especially in section 3, can make the paper a bit harder to read, though the process is pretty straightforward.", "questions_to_authors": "* No IRB is mentioned; please describe the level of IRB review.\n\n* I don't fully understand the continuity error definition, and I don't think it's settled by the \"extended\" version with reader's beliefs (L116). Specifically, in Def 2.1, you have a continuity error defined as removing a proposition means that we can conclude the negation of that proposition. Principally, the negation is not the opposite, e.g., \"not hot\" is _not_ cold. Is there a closed world assumption being made here (if not in the prompt, then at least in the setup + definition)?", "extra_scores": {}}, {"review_id": "SpniS1g1pQ", "reviewer": "Reviewer_9VkJ", "summary": "The paper proposes a novel benchmark that evaluates the narrative understanding and reasoning capabilities of LLMs by detecting plot holes in stories. Plot holes refer to logical inconsistencies in a storyline that violate the internal logic or established rules of the story's world. The authors introduce an algorithm to automatically synthesize plot holes in human-written stories. This algorithm extracts relevant facts from the first act of a story, negates them in subsequent acts, and generates stories with potential plot holes. The benchmark consists of two tasks: a binary classification task to determine whether a story contains plot holes and a localization task to identify the specific text spans introducing the plot holes and the information they contradict. The paper evaluates the performance of several state-of-the-art LLMs on this benchmark, finding that LLMs struggle to accurately detect plot holes, particularly as story length increases. Furthermore, the authors demonstrate that LLM-generated stories and summaries exhibit significantly higher rates of plot holes compared to human-authored originals.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Innovative research direction\n- Rigorous methodology for synthesizing plot holes\n- Comprehensive evaluation", "reasons_to_reject": "- Algorithm limitations: The algorithm relies heavily on LLMs and human filtering.\n- Benchmark limitations: The benchmark primarily consists of short stories (~thousands of words). This may not fully reflect the challenges of plot hole detection in real-world long-form narratives. Furthermore, the benchmark's stories are sourced from specific domains such as fairy tales and myths, which may restrict the generalizability of the evaluation results. And the benchmark only covers human-written stories and neglects machine-generated stories, which I think are more meaningful since we do not expect LLMs to generate illogical stories.\n- Evaluation metric limitations: The paper's evaluation metrics focus on the accuracy of model predictions but may not fully capture the richness and complexity of narrative reasoning. \n- Lack of practical applications: The paper emphasizes the evaluation of LLMs' narrative reasoning capabilities through plot hole detection but provides limited exploration of practical applications. While it mentions the potential utility of plot hole detection in improving narrative consistency, it does not delve into specific methods or case studies demonstrating how to leverage this capability to enhance the quality of LLM-generated stories.", "questions_to_authors": "N/A", "extra_scores": {}}, {"review_id": "i1gFSIrlsH", "reviewer": "Reviewer_tF48", "summary": "The paper introduces FLAWEDFICTIONS, a benchmark for evaluating narrative-level reasoning in large language models (LLMs) through plot-hole (continuity-error) detection. Positive (with plotholes) examples are automatically generated by substituting an early story fact with a contradictory one; all data are subsequently human-verified. The authors benchmark a range of frontier LLMs and find sharp performance drops on long stories.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. New task & dataset. Provides a partially synthetic, human-checked corpus focused on narrative consistency rather than surface comprehension.\n\n2. Broad evaluation. Benchmarks both proprietary (GPT-4o, Claude 3.x) and open-weights models (Llama-3, DeepSeek-R1, Qwen-2.5), exposing weaknesses in long-form reasoning.\n\n3. Initial analysis. Explores how LLM-generated summaries/adaptations introduce plot holes, offering practical diagnostic value.", "reasons_to_reject": "1. The paper states that the task *typically goes beyond a literal understanding of what happened* (L25), framing it as an advance in narrative reasoning. However, based on my understanding, the dataset is constructed by replacing specific facts in the narrative (e.g., injury location), which resembles prior contradiction or factual error benchmarks. How does this differ fundamentally from existing setups like fact-swapping in news articles or Wikipedia entries? More concretely, in what way does the task actually require *complex and nuanced reasoning* (L26), beyond surface-level consistency checking?\n\n\n2. Most of the related work cited focuses on pre-LLM narrative reasoning. Beyond MuSR, several relevant datasets and benchmarks have been published prior to the COLM'25 deadline. Please compare your setup against the following recent work and clarify what aspects (task formulation, scale, generation pipeline, or evaluation design) are novel:\n\n\n[1] EventGround: Narrative Reasoning by Grounding to Eventuality-centric Knowledge Graphs. LREC-COLING'24. https://aclanthology.org/2024.lrec-main.587/\n\n[2] Unveiling Narrative Reasoning Limits of Large Language Models with Trope in Movie Synopses. EMNLP'24. https://aclanthology.org/2024.findings-emnlp.872/\n\n[3] NARRATIVE-OF-THOUGHT: Improving Temporal Reasoning of Large Language Models via Recounted Narratives. EMNLP'24. https://aclanthology.org/2024.findings-emnlp.963/\n\n\n3. Table 1a shows that some state-of-the-art models already match or exceed human performance on your task. Could you comment on whether this leaves sufficient headroom for future modeling work?", "questions_to_authors": "Please address the concerns raised in the \"Reasons to Reject\" section, particularly points 1 and 2, or clarify if I have overlooked or misunderstood any aspect.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces the task of "plot hole detection" -- identifying whether a story contains a continuity error (where one fact is given that contradicts the logic of the rest of the story), and also identifying the specific place (e.g., sentnece) of that error.  It designs a new LLM-based pipeline for synthesizing stories containing plot holes from existing stories (including segmenting into three acts, extracting propositions, generating counterfactuals etc.), which critically involves human verification. It uses this dataset to benchmark the performance of a range of LLMs (where most struggle on the task), and also to provide an automatic means to assess the degree to which LLM stories themselves contain plot holes (as others -- both in research and in the general public -- have noted their tendency toward long-form logical inconsistency).  Reviewers in general found this work creative and thorough in its experiments and error analysis, with specific contributions in formalizing the concept of a "continuity error" and developing a dataset that could be useful for further analysis.  Weaknesses around contextualizing this work with respect to other narrative reasoning research, and understanding the headroom for improvement with respect to human performance were generally addressed in the author response.  Overall a strong and original paper that would merit interesting discussion.
Decision: Accept