CLIPPER: Compression enables long-context synthetic data generation
Authors: Chau Minh Pham
Yapei Chang
Mohit Iyyer
Venue: COLM 2025 (2025)
Keywords: synthetic data
fine-tuning
instruction-tuning
TL;DR: We introduce CLIPPER, a compression-based approach for generating synthetic data tailored to narrative claim verification—a task that requires reasoning over a book to verify a given claim.

Abstract: LLM developers are increasingly reliant on synthetic data, but generating high-quality data for complex long-context reasoning tasks remains challenging. We introduce CLIPPER, a compression-based approach for generating synthetic data tailored to narrative claim verification—a task that requires reasoning over a book to verify a given claim. Instead of generating claims directly from the raw text of the book, which results in artifact-riddled claims, CLIPPER first compresses the book into chapter outlines and book summaries and then uses these intermediate representations to generate complex claims and corresponding chain-of-thoughts. Compared to naive approaches, CLIPPER produces claims that are more valid, grounded, and complex. Using CLIPPER, we synthesize a dataset of 19K claims paired with source books and chain-of-thought reasoning, and use it to fine-tune three open-weight models. Our best model achieves breakthrough results on narrative claim verification (from 28% to 76% accuracy on our test set) and sets a new state-of-the-art for sub-10B models on the NoCha leaderboard. Further analysis shows that our models generate more detailed and grounded chain-of-thought reasoning while also improving performance on other narrative understanding tasks (e.g., NarrativeQA).

Published as a conference paper at COLM 2025
CLIPPER: Compression enables long-context synthetic data generation
Chau Minh Pham! Yapei Chang!
Mohit lyyer
Universitty of Maryland, College Park
University of Massachusetts Amherst
yapeiic,piiiiiii miyyer @umd. edu
Abstract
LLM developers are increasingly reliant on synthetic data, but generat- ing high-quality data for complex long-context reasoning tasks remains challenging. We introduce CLIPPER, a compression-basee approach for generating synthetic data tailored to narrative clain verification task that requires reasoning over book to verify a given claim. Instead of generating claims directly from the raw text of the book, which results in artifact-riddled claims, CLIPPER first compresses the book into chapter out- lines and book summaries and then uses these intermediate representations to generate complex claims and corresponding chain-of-thoughte pared to naïve approaches, CLIPPER produces claims that are more valid, grounded, and complex. Using CLIPPER, we synthesize a dataset of 19K claims paired with source books and chain-of-thought reasoning, and use it to fine-tune three open-weight models. Our best model achieves break- through results on narrative claim verification (from 28% to 76% accuracy on our test set) and sets new state-of-the-art for sub-10B models the NoCha leaderboard. Further analysis shows that our models generate more detailed and grounded chain-of-though reasoning while also improving performance on other narrative understanding tasks (e.g., NarrativeQA).
Com-
https: thub. com/ chmm/hhhmm223/LLPPE
1 Introduction
Due the high cost of human-annotated data, LLM developers increasingly rely on synthetic data (generated by LLMs) to boost instruction following and reasoning capabilities (Ding et al. 2023; Lambert al. 2024; Yang al. 2025, inter alia). As the context size of LLMs extends to millions of tokens, it is important to ensure that we have scalable and performant strategies to create synthetic data for long-contert tasks. Prior work creates such data by (1) selecting a long document or a smaller chunk within; and (2) prompting an LLM to generate imput/outpu pairs using the selected text (Bai al., 2024; Dubey et al., 2024).
While this strategy is effective for tasks like summarization and QA, we show that it breaks tasks like narrative claim verification, in which model must judge whether a statement about long input text is true or false. The majority of narrative claims in the NoCha benchmark (Karpinska et al., 2024), which was created by human readers of fictional books, can only be verified by global reasoning over events, characters, and relationships.? This poses challenge to even the best LLMs: OpenAl's ol-preview currently leads with an accuracy of 67 .44% (far below human performance). If no LLM can reliably solve the task, how can we produce and validate synthetic data for it?
down for more complex reasoning-orient
We tackle this challenge by introducing CLIPPER, a synthetic data generation pipeline that operates in two stages (Figure 1). First, long document is compressed by an LLM into
!CLIPPER stands for Compressing Long InPuts.
"Creating NoCha costly and challenging: annotators read full books and earn $1.70 per claim.
1

---

Published as a conference paper at COLM 2025
Stage Compression
Stage Syntheticc claim generation
Chapter outline
Book- claim pairs
Chain-of-though
[Toad fur
you devetic shere are the vinuur ani tie
Book tuxt 74,884 Yoklnns
Chapter 1
See Htalvar stallo..
Policiiss
pPogrnsss trem por ausin love to
CA Chieter Dosum on rejerta trr leosting
Cf sptar skyl's performane ace su Juler unsopeciadly
Bac Piccuncal
persal yhuu man
ihanel Oveann retal prohip with dity! progremve from contrnato offene arariggs
ca sutar Lord Henry ofform tor on
Hard, oyr nil and welt...
atborthpg Sequar
Chapter
actia nuriil charorn. colsppatation
(Trud Lort Henry coiio Rusd locrtts Ts Dual rnturns cliin permonal ressons.
A hundoree yours mes Dur on Orip, cares seftcct grive curenruu retuaing fus
the minnert celats
Sottte Lerd Herry
seggents
Ovear Wille
whhttt
pursal Dotten Oney goedd destroying
Lare ortiiiv
atorathg: ing The conteruan d subled wecly
revelb ausects Derien
Cc Roaal orluaan, clrin incestary sakin
Book summary 919 takions
Fasiooa for baiiing to eat des partils uispainattio
Chapter-leval claim paints
Figure 1: CLIPPER overview. (1) Compression: An LLM generates chapter outlines (8.7K tokens) and summaries (618 tokens) from books (90K tokens). (2) Claim generation: The LLM produces que/false claims with ofains-of-thought with the outlines and summaries.
summaries and/or or outlines that contain salient events. Then, the LLM generates claims based the compressed narrative, with optional instructions to write claims that require reasoning over multiple chapters to verify. Each claim is accompanied by a chain-of-though reasoning trace that grounds it within specific chapters where relevant events occur.
Compared to the naïve strategy of prompting an LLM with the entire (uncompressed) book, CLIPPER significantly reduces the error rate in the claims from 73.1% to 16.7%, while also producing more claims at a lower cost. Why does this work? Prior work has shown that LLMs are high-quality summarizers of long documents (Chang et al., 2024; Kim et al., 2024). By operating on compressed representations, we also address the known degradation of instruction-followe following in long-context settings (Wu et al., 2024b; Levy et al., 2024) and thus reduce the complexity of the claim generation process.
We use CLIPPER to generate a dataset containing 19K claims about public-domain fictional books. Fine-tuning open-weight models like -lama-3.1-8B-Insttee (Dubey et al., 2024), ProLong-512K-88 8B--Base (Gao al., 2024c) and Qwen2.5-7B-Instree this dataset yields large improvements to other narrative-related tasks (e.g,, NarrativeQA, MuSR). For instance, fine-tuning Llama- 3.1-8B-Instruct our data doubles its NoCha performance (from 16.5% to 32.2%) and almost triples its test set performance (from 27.9% to 76.0%). Our fine-tuned Qwen model sets new state of the art NoCha for <10B models, outperforming closed models like Gemini 1.5 Flash SB (Team et al., 2024) and OpenAl ol-mini (OpenAl, 2024).
(Qwen et al., 2024) on narrative claim verification and positive transfer
While our approach is promising for improving long-contess reasoning in open-weight models, our best models fall well short of the performance reached by closed LLMs such as ol on NoCha. The performance gap between NoCha and CLIPPER-test likely stems from the nature of the claims, as CLIPPER-test features synthetic claims based on model-generated outlines, while NoCha's 's human-written claims require reasoning about details often absent from such outlines. We analyze where our fine-tuned models can still improve, discovering that they benefit more from training on claims whose evidence localized chapter (and not more complex multi-chapter claims). We hope future work will use our data generation pipeline for fine-tuning larger models (e.g., >70B) on other long-context tasks to further improve global reasoning abilities. In summary, our contributions are:
single
1. CLIPPER: A compression-based pipeline for synthesizing grounded long-context data low cost, producing 19K claim-book pairs for narrative claim verification.
2. Fine-tuned open-weight models that advance claim verification and narrative un- derstanding, showing the benefits of training on data generated with CL ILPRR..
2

---

Published as a conference paper at COLM 2025
2 CLIPPER: generating high-quality synthetic data via compression
In long-context settings, synthetic datasets have typically been created by selecting lengthy documents from an existing corpus and using an LLM to generate input-output pairs given either the entire document (Bai et al., 2024) or random excerpts (Dubey et al., 2024; Yang et al., 2025). For our task, however, we show that these methods insufficient:
Providing the LLM with the entire document results in much noisier data, as we show that producing high-quality, complex claims about long narratives is fundamen- tally difficult task even for the best models ($2.2).
Providing only an excerpt from the long document, on the other hand, precludes the model from generating claims that require global reasoning across the entire book.
We thus develop two-stage strategy, CLIPPER, which first compresses the narrative into chapter outlines and summaries, then prompts an LLM to produce claims and chain-of- thoughts grounded in the compressed narrative ($2.3). We use the dataset generated by CLIPPER to fine-tune open-weight models in $3. All prompts can be found in $A.5.
2.1
Task setup
Before describing how CLIPPER works, we first establish the definition for narrative claim verification, then explain how we collect the books that serve the foundation for this task.
Task definition: In narrative claim verification, an LLM is given a book and a claim about the book. The task is determine whether the claim is true false while providing a clear explanation for its decision. A key aspect of the task is the inclusion of true/false narrative minimal pairs (Karpinska et al., 2024), where each false claim closely resembles its true claim counterpart but contains subtle inaccuracies (illustrated in Figure 1; see Table 4 for more examples). The model is considered accurate only correctly verifies both claims in a pair, which reduces the chances of the model being correct for the wrong reason.
Gathering public domain books: We collect 479 fictional books from Project Gutenberg, with an average length of 90K tokens² and 23 chapters. 4 While these texts might raise memorization concerns, $A.2 shows they do not affect baseline model performance."
Naïve claim generation using book texts
One simple data synthesis approach prompting an LLM to generate claims directly from the book text. However, this NAIVE method falls short in long-context setting, with 73.1% of the generated claims containing serious errors. In addition, NAÎVE produces fewer claims at higher price. These limitations motivate us to develop CLIPPER.
The NAIVE method: We provide Saaude-3.5-Sonnet-ee with the entire book text and prompt it to generate pairs of true/ false claims along with corresponding thain-of-thoughle reasoning in a zero-shot manner. Note that we cannot use few-shot prompting due to the books' length. We finally prompt Claude to remove any duplicated claims among the generated claims. Note that we do not ask Claude to validate the generated claims, as this is a much more challenging task (as seen by Claude's i 40.3% accuracy on NoCha) the very problem aim to address in this paper. In contrast, claim validation is very feasible with CLIPPER, as Claude's 's claim verification accuracy given outlines instead of text 98.6%.
All token counts are computed using o200k base from https://gi thub. com/openai/ttee stpeeai/tiktoken.
"We do not include books longer than 128K tokens many open-weight models cannot process anything beyond that number. We clean the manually downloaded books by removing supplementary content to prevent models from using these metadata as shortcuts for event retrieval.
httpps:/ thaagppiseattingtheoorll wtwprdd.oo//w-coeeentt. wp-content/ ptatteenttplodss//// 22/2/022///1 1225/01/Umma UnnreacteddRReply Plaintiffs-1.pp 1ddf shows that llama models might have been trained LibGen book data.
3

---

Published as a conference paper at COLM 2025
CATEGORY
NAIVE
CLIPPER
ERROR DEFINITION
EXAMPLE
Invalid
11.5% 9.1%
The claim is incorrect with respect to the book text, ur the true/fales claim pair is invalid.
Anne rejects three marriage proposals during her time Redmond College: from Charlie Sloane, Gilbert Blythe, and Roy Gardner, all because doesn't love them. (This fulse claim not extirely fülae besnuse Anne really didi'' love them or waan't initiaily auvre of her romantic feelings..
Mis- attribution
28.9%
4.6%
The claim is valid, but associated explana- tion does not cite the cur- rect chapters.
Dr. Sheppard.. the last person known to have seen Roger Ackroyd alive 8:50 PM on the night of the murder, assisted Hercule Poirut in the investigation while simultamcously concealing Ralph Paton
later
nursing home. (The explanation cifes Chapter 1, 4, 16, and 20, but misses Chapter 24, which mentions that Ralph in nursing home)
Explicit refer- ences
15.4%
0.0%
The claim is casier to ver- ify since it includes direct quotes and chapter ref- erences, eliminating the need for event retrieval.
Alice's pursuit of the White Rabbit, which begins with her following him down a rabbit hole Chapter continues throughout her adventure, including an encounter in the King and Queen of Hearts' Chapter 11 where the Rabbit
herald.
Duplication
17.3%
3.0%
claim describes same events as anuther. Although their content is similat, differences in wording may allow both to pass our dieduplicaiion process.
"Dorian Gray's criell rejection Sibyl after poor per- formance as Juliet leads to her suicide, which Dorian callously dismisses attending the opera the following night, resulting the first noticeable change in his portrait [...]" versus "Dorian Gray's cruel rejection of Sibyl Vane after her poor performance leads to her suicide, causing the first visible change in his purtrait [..] culminate in his murder of Basil Hallward years later [...]."
Juliet
Any
73.1%
16.7%
The claim is invalid, misattributed, duplicated, or contains explicit references.
Table 1: Error types among claims produced by NAÎVE (52) and CLIPPER (66) based on six books from Table 5. Examples are selected from NAÎVE claims.
Human validation of NAÎVE claims: We manually annotate 52 claims generated by NAÎVE
based on six books (Table 5).6 Across these claims, we identify four types of errors. Table shows that 73.1% of NAIVE claims contain an error. Specifically, there are 11.5% invalid claims, often due to mislabeled false claims that are actually valid. There is also a high number of misattributed (28.9%) claims that cite the wrong chapters in the produced chain-of-thought 17.3% of the claims are duplicated despite the deduplication step, because Claude frequently hallucinates source chapters. 15.4% of the claims also include explicit references chapter numbers or direct quotes? 7 which compromises the task by revealing the evidence location. Additionally, we observe that the events referenced in the generated claims are often the book's 'ss most major events, making the claims much easier for LLMs to verify. Beyond these errors, the NAIVE pipeline is costly at $0.07 USD per claim (< $1, 3,0 for 19K claims).
2.3 Claim generation with CLIPPER
To produce more valid and grounded claims, we use compressed representations of the book content, namely chapter outlines and book summaries. These intermediate forms help anchor claims to specific events in the book, reducing the need to search through the entire text for relevant details. Additionally, this approach makes it easier to generate claims about lower-level events, addressing major limitation of the NAÎVE approach. We first (1) compress the books into a chapter outline and book summary and then (2) generate pairs of true/faase claims at different scopes based on these compressed representations.
(1) Compressing books into summaries and chapter outlines: Book summaries provide a global context for claim generation to ensure that each claim is consistent with the entire book. We prompt GPT-4o to summarize the entire book into few paragraphs (s 618 tokens on average). Chapter outlines provide list of fine-grained events that can be used to construct grounded claims. We prompt Claude with each chapter text to generate an outline containing a synopsis, major events (5-7 per chapter), and character list. Our
We annotate each claim with its most major error type.
7hii happens despite explicit formatting instructions. We set temperature=0.0, nox.tookes==006. We use Claude instead of GPT-4o because Claude includes more concrete and objective events for the outline.
4

---

Published as a conference paper at COLM 2025
compression rate is 10.0%, calculated by averaging the ratio of outline length (8,745 tokens on average) to full book length (90,437 tokens) across all books.
(2a) Generating claims from compressed narratives: We use chapter outlines and book summaries to generate true/ false claims. We synthesize claims at two different scopes to enable reasoning across different token ranges:
Book-level claims: Claude is prompted to identify 2-3 key events from the outlines of at least chapters, then use them to generate a claim. These claims require models to have global understanding spanning multiple parts of the book.
Chapter-level claims: Given the book summary and a single chapter outline, Claude is instructed to identify 2-3 key events of the chapter and them to write a claim. While these claims do not necessitate global reasoning, they still require the model to search for correct chapter within long text and perform intra-chapter reasoning ($4.5).
(2b) Deduplicating and validating generated claims: Just as in NAIVE, we use Claude to remove duplicate claims. Additionally, we use GPT-4o" to validate the claims against the source chapter outlines by prompting it to assess whether all parts of claim are supported by the outline. This step is another advantage of CLIPPER: unlike the NAÎVE approach, our method allows for claim verification using the compressed chapter outline. To evaluate the reliability of LLM-based filtering, we manually review 72 claim pairs and only disagree in one instance, where GPT-4o deems a claim valid that we find too subjective. Overall, 59.4% of the original claims are removed duplicates, while 2.4% are filtered out invalid.
Human validation of CLIPPER claims: We use the same setup as described in $2.2 to manually evaluate 66 claims generated by the CLIPPER pipeline. Table 1 provides detailed breakdown of issues flagged in these claims, such as explicit references, invalidity, duplica- tion, misattribution Notably, 83.3% of the 66 claims are found to be completely free of errors-a significant improvement compared NAÎVE's 26.9%. CLIPPER also costs less at $0.05 USD per claim compared to NAÎVE ($0.07 per claim) and human annotators ($1.7 per claim based on Karpinska al. 2024). 10
2.4 Automatic validation of CLIPPER chain-of-thoughts
We assess the groundedness of chain-of-though (CoT) reasoning by prompting an LLM to verify whether each event in the CoT is supported by the chapter outline. Accuracy is measured as the percentage of events in true claim CoTs that are grounded in the book. To scale up evaluation, we et al., 2025). We find that 98.5% of CoTs are grounded. The remaining ungrounded CoTs typically involve events open to multiple interpretations. Compared to NAÎVE, CLIPPER's CoTs are significantly easier to verify due to their explicit chapter references.
LLM judge,
DinpseekRR--iistll--laaaa-
(DeepSeek-Al
3 Supervised fine-tuning for LLMs on CLIPPER data
Having shown that CLIPPER produces synthetic data of high quality, we now investigate the effects of training on such data. We apply supervised fine-tuning (SFT) to three models on our dataset: TroLoon-512K-8F
(Dubey et al., nearly three times the test set performance Our top model, Llaam--LLIPPER, achieves Lamma-nntructt-boottti accuracy from 27.9 17.9% to 7hiee showing substantial gains in long-context reasoning and narrative
(Gao et al., 2024b), 11 Llama-3.1-8B-Instree
2024), and Qwwn2.5-7B-Instruus (Team, 2024). 12
CChosen mitigate potential self-biases (Xu et al., 2024b; Panickssery al., 2024; Li al., 2025). 10See detailed cost analysis in $A.4.
11 Despite the name, this model has undergone instruction tuning before. The ProLong team ran continual pre- trainrain on 1amm---8B-Inntruuee to get this model. We will refer these models as ProLong- Llama-Instruct, and Qwen-Instruct.
5

---

Published as a conference paper at COLM 2025
understanding on tasks like NoCha, NarrativeAA, and MuSR. Moreover, all of our models outperform all existing <10B models on the NoCha benchmark.
3.1 Training setup
Data splits and hyperparameters: We divide our dataset into three parts: 16K claims (8K true/ false pairs) for training, 2K for validation, and 1K for testing. Notably, the books in the test set do not overlap with those in the training validation sets. For each entry, we combine the book text and claim to form the user prompt, and include the chain of thought reasoning along with the final answer as the assistant's message (see Figure 16). A learning
rate of 1e-6 and batch size of 16 yield the best performance on our dev set. 13 We fine-tune Qwen-Instruct, Llama-Instruct, and Prol.ongBBan -aase using this configuration for one epoch.
Ablation on the effect of claim scope: Our dataset consists of 8K book-level and 8K chapter-level claims. We fine-tune ProLong-Base separately on each claim scope subset, resulting in ProL.ong-CLEPEERcchatee and Prol.ong-CLIPPER-boool
Ablation on the effect of data length: Prior work shows that fine-tuning on short texts can improve long-context performance in tasks like QA and summarization (Dubey et al., 2024; Gao et al., 2024b). Since our dataset contains long documents averaging 90K tokens, we test whether short-text fine-tuning also helps with long-contexs claim verification. We use WritingPromppts (Fan et al., 2018), a dataset of 300K stories averaging 742 tokens, and extract claims directly without generating outlines or summaries." 14 We collect 19K claims and train on ProLong-Ban to get wra.onngwrinnggoooppss
3.2 Evaluation
Beyond claim verification, we expect that training on our synthetic dataset will also im- prove performance on related tasks. Therefore, include both reasoning and narrative understanding benchmarks that vary in input lengths and tasks.
Claim verification: To measure accuracy, we calculate the percentage of cases in which model correctly verifies both true and false claims within a given pair.
> CLIPPER-test contains 1,000 true/ false claim pairs drawn from 53 books, evenly split between book-level and chapter-level claims.
>
NoCha (Karpinska et al., 2024) consists of 1,001 true rrue/false false claim pairs about recent fiction books (up to 336k tokens). These claims, crafted by annotators familiar with the books, are much harder to verify compared to those in CLIPPER-test.
General narrative understanding: We use three existing benchmarks as detailed below.
> NarrativeQA (Kočiský et al., 2018) a long-form Q&A benchmark that requires models to process entire books movie scripts to answer provided questions. The benchmark consists of 1,572 stories and summaries as well as 46,675 human-written questions. We use the HELMET implementation (Yen et al., 2024) for this benchmark.
BeBnch QA (Zhang et al., 2024) is a long-form Q&A benchmark that requires models to answer 351 questions about novels. We use the HELMET implementation but use GPT-4o's judgment as a metric instead of ROUGE F1 (see SC.1 for explanation).
We perform hyperparameter tuning on learning rates of 1e-5, 1e-6, and 1e-7, along with batch sizes 16 and Tuning is done for one epoch on subset of 2K training samples. Due to high GPU costs (each epoch takes 5 hours), we only conduct hyperparameter tuning on ProLong- -Bae only.
14We a prompt similar to the one in $2.3.
15 After doing hyperparameter tuning on 2K training samples, we decide on the learning rate of le-5 and batch size of 16 as the best training configurations. We tested learning rates of 1e-5, 1e-6, 1e-7 and batch sizes of 8, 16, 32, 64.

---

Published as a conference paper at COLM 2025
Models
CLIPPER-test
NoCha
NarrativeQA
MuSR
coBench QA
Qwen2.55-B-Instruuc
51.0%
24.1%
40.3%
41.2%
35.3%
27.9%
16.5%
47.7%
40.3%
47.8%
BolLong-512K-BB-A
34.5%
16.9%
44.0%
42.3%
42.6%
Qwen2.5-7B-CLIPPE
73.9%
32.4%
46.0%
45.2%
42.3%
76.0%
32.2%
49.0%
43.6%
46.5%
MooLoog-512K-8B-CLLPPP ProLong-51288
75.0%
32.3%
49.0%
44.5%
38.5%
63.0%
24.1%
31.0%
43.2%
35.8%
Table 2: Model accuracy on claim verification (CLIPPER-test, NoCha) and narrative un- derstanding benchmarks (NarrativeQA, MuSR, coBench QA). Fine-tuning models using CLIPPER improves performance claim verification and narrative understanding.
MuSR (Sprague et al., 2024) includes 756 algorithmically generated problems such as murder mysteries, object placement questions, and team allocation optimization. We use the LM Harness (Gao et al., 2024a) implementation.
Results & analysis
Our fine-tuned models set new state of the art for <10B models on long-context claim verification while also improving baseline performance on narrative understanding tasks.
4.1 CLIPPER models outperform baselines on narrative claim verification
On CLIPPER-test, our fine-tuned models significantly outperform the instruct models they are initialized from (referred to as baselines), 16 as shown in Table
For example, Qwen-CLIPPER achieves over a 20% performance gain compared to Qwen-Instruct, while LIama-CLIPPER sees nearly triple the performance Llama- Instruct. These substantial improvements demonstrate the effectiveness of CLIPPER- generated data.
76.0
322
ProLong 512K CUPPER
m.een-255-BB. CUPPPE
Owen2.55 (reen2.578... 7-7--nnsttt
OpenAl al ol-Mini
487
28.6
45.0
Fine-tuning on our data improves perfor- mance on NoCha: A similar trend is ob- served on NoCha. The performance im- provements range from an 8% gain for strong baselines like Qwen-Instruct to twofold increase for weaker baselines such as Llama-Instruct and ProLong-Instree Instrucct It is worth noting that all baseline models ini- tially perform below the random chance baseline of 25%, but our fine-tuned mod- els consistently surpass this threshold.
26.2
Protang 512K Instruct
Liama-3 3.1-88 1.1-88 Instruct
30
50
60
70
Accuraty (76
Figure 2: Results on CLIPPER's test and NoCha for baselines, small closed models, and CLIPPER models. Fine-tuning on our synthetic data significantly improves narrative claim verification.
Performance gap between CLIPPER-test
and NoCha: We note that the performance gap between NoCha and CLIPPER-test is likely due the nature of the events involved in the claims. While CLIPPER-test consists of syn- thetic claims derived from events in model-generater outlines, NoCha's human-written claims may involve reasoning about low-level details that may not typically appear such
16ProLong-CLIn CLIIPERRRRRR is initialized from GroLong-Base instead of ProLong-Instrute However, since performing evaluation intended for instruct models on continually pretrained model may not be ideal, we exclude Prol ong-Base's results from Table 2. As shown in Table 19, langoonngass performs significantly worse than Prol.ngg Instrrut CLIPPER-test.

---

Published as a conference paper at COLM 2025
Claim: "Jacques consistently uses pastoral themes to express his love for Belle-bouche, first through metaphorical discussions of shopherds and shepherdesses, and later while walking through an orchard discussing Arcadian lovers." "
Ulmaa-nsttuct's CoT:
"The stalement claims that Jeeques
Llama- IPPPR''s CoT:
"Here are the relevant details from the lext:
consistently pastoral themes express his love for
Chapter 2: Jecques engages in metaphorical discussion about shep-
Belle-bouche. To evaluate this statemeni, we need ex-
herds shepherdesses, expressing his love for Belle-bouche.
amine the text and identify instances where Jacques uses
Chapter 17: Jacques and Belle-bouche walk through the orchard, discussing the twenticth of May day for lovers.
pastoral themes express love Belle bouche. The
text mentions that Jacques discusses crooks, flowers, and
Relationship: Recurring theme of pastoral love
shepherdesees with Belle bouche, and later, while walking through an orchard, he talks about Arcadian lovers, specif- cally mentioning Strephon, Chloe, Corydon, Daphnis, and Narcissus, These instances suggest pastoral themes to express his love for Belle-bouche. How- consider whether this consistent pat- tern throughout the text."
In Chapter Jacques engages metaphorical discussion about shep- herds and shepherdesses to express his theme continues in Chapter 17, where Jacques and Belle-bouche walk through orchard, discussing the twenticth of May day lovens. This connection shows Jacques' consistent use of pastoral imagery to his romantic feelings for Belle-bouche throughout the story." Answer: TRUE
Belle bouche. This
lacques does use
ever,
Answer TRUE
Table 3: Comparison of CoT responses from Llama Instruct and Llama-CL IPPER for a claim in John Esten Cooke's 'ss The Youth of Jefferson. Llama-CL IPPEER includes more explicit chapter references (green) and details than Llama-Instruct (red).
generated outlines. Future work could incorporate more low-level events into chapter outlines to create more diverse claims.
4.2 Fine-tuning on CLIPPER improves on other narrative reasoning tasks
Beyond long-context reasoning, our models also show improvements in narrative un- derstanding and short-context reasoning tasks. On NarrativeQA, which requires com- prehension of movie scripts or full books, our best-performing models, Llama-CLIPPER and Prolonn--CLPPER CLIPPE,,, achieve 2% and 5% absolute improvement over their respective baselines. Similarly, on MuSR, a short-form reasoning benchmark, our strongest model, Qwen-CLIPPER, achieves 45.2% accuracy, surpassing the 41.2% baseline. However, on ooBench QA, only Qwen-CLIPPER outperforms the baseline by approximately 7%. In con- trast, LIama-CLIPPER and ProLong-CL IPPER show slight performance declines of up 4%. Thus, while fine-tuning on CLIPPER data improves performance on reasoning and some aspects of narrative understanding, its transferability is not universal across domains.
4.3 Long-context claim data is more helpful than short-context data
Our results stand in contrast to prior studies suggesting short-form data benefits long- context tasks (Dubey et al., 2024; Gao et al., 2024b) more than long data. While ProLong- WritingPrompis, trained on short data, outperforms baselines, it underperforms across all four long-context benchmarks compared to models fine-tuned on our data. This underscores the need for high-quality long-conterd data generation pipelines like CLIPPER.
4.4 Fine-tuning on CoTs results in more informative explanations
We evaluate the groundedness of CoT reasoning generated by our fine-tuned models using DeepSeek-R1-Distilll Dit----Dssttil-L-la-0PP -Lamm--00 ($2.4). Here, a reasoning chain is counted as grounded when every plot event in the chain can be found in the chapter outline that it cites. Table 21 shows that fine-tuning significantly improves groundedness across all models, with ProLonn-CLIPPER achieving the highest rate (80.6%), followed closely by Llama-CLIPPER (75.9%). Looking closer at the content of the explanations (Table 3), the baseline model (Llama-Instruct) often gives generic response without citing any evidence, whereas Llama- CLIPPER explicitly references Chapter 9 and specifies the cause-and-effect relationship.
4.5 Small models struggle with book-level reasoning
Trained only on 8K chapter-level claims, CLIPPER-book on both chapter- and book-level test subsets (Table 20). This likely reflects the limitations of smaller models (7B/8)) in handling the complex reasoning required for book-level claims, in line with prior findings (Qi et al., 2024). The performance gap between
outperforms ProLong-
8

---

Published as a conference paper at COLM 2025
CATEGORY
FREQ (%)
TRUE CLAIM
FALSE CLAIM
Event
43.2
The Polaris unit, initially assigned to test new audio transmitter on Tara, explores the planet's surface using jet hoat without landing.
The Polaris unit, initially assigned to test new audio transmitter on Tara, explores the planet's surface by landing their spaceship.
Person
31.6
The cattle herd stolen from Yeager by masked rustlers is later found in General Pasquale's a possession at Noche Buena.
The cattle herd stolen from Yeager by masked rustlers is later found in Harrison's possession at Noche Buena.
Object
15.8
The alien structure Ross enters contains both chamber with jelly-like bed and control panel capable of communicating with other alien vessels.
The alien structure Ross enters contains both chamber with metal bed and con- trol panel capable of time travel.
Location
13.7
Costigan rescues Clio twice: first from Roger on his planetoid, and later from Nevian city using stolen space-speedster.
Costigan rescues Clio twice: first from Roger on his planetoid, and later from Triplanctary city using stolen space- speedster.
Time
6.3
Jean Briggerland's meeting with ex ox-convicts Mr. Hoggins and Mr. Talmot, where she suggests burglary target, follows a failed atlempt Lydia' ' life involving speeding car on the sidewalk.
Jean Briggerland's meeting with ex-convicts Mr. Heggins and Mr. Talmot, where she suggests a bunglary tanget, precedes failed attempt Lydia's life involving a speeding car on the sidewalk.
Affect
4.2
David Mullins, who initially expresses skep- ticism about Chester's 's hiring, later fires Chester false pretenses and immedialely replaces him with Felix.
David Mullins, who initially expresses en- thusiasm about Chester's 's hiring, later fires Chester false pretenses and immedialely replaces him with Felix.
Table 4: Taxonomy of perturbations causing false claims to be misclassified true. True and false details are highlighted in green and red, respectively. Frequencies may exceed 100% due to multi-labeling. See SD.2 for definitions and analysis.
the models is modest (4.2%), and we leave exploration of larger models (>70B) to future work due compute constraints.
4.6 Fine-tuned models have a difficult time verifying False claims
To study cases where fine-tuned models struggle, we analyze Qwen-CLIPPER outputs. Among 1,000 book-level claim pairs in CLIPPER-test the model fails to verify 37 true claims and 97 false claims, aligning with NoCha findings (Karpinska et al., 2024) that models struggle more with false claims. We investigate perturbations that make false claims appear true and present a taxonomy with examples in Table 4, with further details in SD.2.
Related work
Long-context language modeling: The context size of LLMs has expanded signifi- cantly (OpenAl et al., 2024; Dubey et al., 2024; Team et al., 2024; Yang et al., 2025), thanks to position inter- and extrapolation techniques (Press et al., 2022; Su et al., 2023; Peng et al., 2023) and efficient attention implementation (Dao et al., 2022; Dao, 2023; Liu & Abbeel, 2023). Longer data has been used during continual pretraining (Lieber et al., 2024; Xiong et al., 2023) or alignment stage (Bai et al., 2024; Xiong et al., 2024, An et al., 2024). CLIPPER augments existing long-form book texts with synthetic but challenging claims, which serves as the foundation of fine-tuning pipeline to improve LLMs' understanding and reasoning over long-context data.
Instruction-tuning data generation: Short-form data generation methods either induce instruction data from texts (Honovich et al., 2022; Zhou et al., 2023; Li et al., 2024) or generate instruction-outpui pairs simultaneously (Wang et al., 2023b). Long-context data is synthe- sized through induction from long-form documents (Pham et al., 2024; Köksal et al., 2023), random document segments (Xiong et al., 2023), or bootstrapping short documents (An et al., 2024; Xu et al., 2024a; Wu et al., 2024a; Wang et al., 2024). CL CLIPPER uses instruction

---

Published as a conference paper at COLM 2025
induction from compressed document representations to create instruction-tuning data for long-context LLMs.
Reasoning alignment: Previous work includes inference-time scaling (OpenAI, 2024; DeepSeek-Al et al., 2025; Muennighoff et al., 2025), prompting (Wei et al., 2023; Kojima et al., 2023; Yao et al., 2023; Wang et al., 2023a), and fine-tuning LLMs on reasoning data (Chung et al., 2022; Huang et al., 2023; Puerto et al., 2024; Yeo et al., 2025). These reasoning data are either human-writter rationale (AlKhamissi et al., 2023) or chain of thoughts distilled from larger models (Hsieh et al., 2023; Li et al., 2023; Ho et al., 2023; Zelikman et al., 2022). We find that fine-tuning models on CoTs improves the generated explanations for the validity of book claims.
Conclusion
We introduce CLIPPER, a compression-based pipeline for generating synthetic narrative claims. We create 19K true/fale claims at both book and chapter levels. Our fine-tuned models set new state-of-the-art among <10B models on claim verification and achieve improvement on narrative understanding tasks. Future work could examine the effect of book-level claims larger models and explore methods for generating harder claims to better match human-written benchmarks like NoCha.
Limitations
We only perform hyperparameter tuning on ProLong- Long-Base due to the high cost of the training process. To put things into perspective, training a model on our full test set requires approximately 50 hours 8 A100 GPUs, each costing $2 per hour to rent. Even training on our tuning subset takes 6 hours. Therefore, extending training further is prohibitively expensive.
Similarly, we do not hire human annotators to write claims for our dataset due to the prohibitive cost and the need for numerous annotators who have thoroughly read the books (Table 8). While this decision may result in less complex claims, our approach offers greater adaptability to new books while significantly reducing costs.
The compression stage can be challenging to fine-tune, subject to model biases, and prone to potential hallucinations. Due to the large volume of data, verifying its accuracy is also difficult, whether via prompting or human annotation. However, we note that prior research has demonstrated that LLMs are capable of producing high-quality summaries of long documents Chang et al. (2024); Kim et al. (2024). In addition, these compressed representations could still provide a strong foundation for claim generations, as most of CLIPPER's claims are grounded in the original book (subsection 2.3).
Acknowledgment
We extend special gratitude to Marzena Karpinska for benchmarking our models on NoCha and giving us helpful insights into the models' behaviors. This project was partially sup- ported by awards IIS-2046248, IIS-2312949, and IIS-2202506 from the National Science Foundation (NSF).
References
Badr AlKhamissi, Siddharth Verma, Ping Yu, Zhijing Jin, Asli Celikyilmaz, and Mona Diab. OPT-R: Exploring the Role of Explanations in Finetuning and Prompting for Reasoning Skills of Large Language Models. In Proceedings of the 1st Workshop Natural Language Reasoning and Structured Explanations (NLRSE), pp. 128-138, 2023, doi: (0.18653/v1/2000 orgp/b.//.......o. 12001. arX5v 12001 [cs].
nlrse-1.10. URL attp://arree http://aaaai..or/////255///
10

---

Published as a conference paper at COLM 2025
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. Make your Ilm fully utilize the context, 2024. URL https: httpss///axxiv... http://aaxivv.orab/////// wwwg/bs/////////o/ 16811.
Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. LongAlign: A Recipe for Long Context Alignment of Large Language Models!, January 2024. URL httt:://arxin www.oorggaooo..... ////b///2// 2401. 18058. arXiv:2401.1800 [cs].
Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit lyyer. BooookScore: A systematii exploration of book-length summarization in the era of LLMs, April 2024. URL http::/arxiii.orr abs/2310. 00785. arXiv:2310.00785 [cs].
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros,
Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling Instruction-Finetuned Language Models, December 2022. URL http: .rgg / 2210. 11416. arXiv:2210.11416 [es].
Tri Dao. Flashattentionn22: Faster attention with better parallelism and work partitioning, 2023. URL httpps://ariiii http://xxxv..or//bbbbe/a wrg/ 08691.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré, Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL https://arxiv.orrg abs/2205. 14135.
Dee,Seek-A,, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, and 190 others. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning, 2025. URL https:///rxivv.orr abs/2501. 12448.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing Chat Language Models by Scaling High- quality Instructional Conversations, May 2023. URL http://arxix org/ orgg/bss23055... 14233. arXiv:2305.14233 [cs].
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint ar Xiv:2400 21783, 2024.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Aunual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ 20/66533v1//11/ /tt/////1P11/////// URL https://aclanthhlogg 1rtp://attoollooo.r/1111111-
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 07 2024a. URL https://zenodo. org/recoods//1eeee 2628602.
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively), 2024b. URL ptttps://arxivv hbrgg/bss//11 2410, 02660.
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to Train Long-Context abs/2410 026600.
Language Models (Effectively)) October 2024c. URL http: arXiv:2410.02660.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the of gpt-3, 2023. URL https://arxiv org/ /209. 12356.
11

---

20/////vv/222.aae// Published as a conference paper at COLM 2025
Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14852-14882, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: acl-long.820 830/.
830. URL
Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions, 2022. URL https://///ttit.orrr abs/2205. 10782.
Cheng-Ye Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Yu Lee, and Tomas Pfister. Distilling step-by-step! out- performing larger language models with less training data and smaller model sizes, In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 8003-8017, Toronto, Canada, July 2023, As- sociation for Computational Linguistics. doi: 10.18653/ /22.2...22.2...... findings-acl.500. URL http:://aclanthollogy org/2023. findings-and acl. 507/.
Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 1051-1068, Singapore, December 2023. Association for Computational Linguistics. doi: 10.1 10.18653/v11 /ttp//v1122222////// 20m.2/00mmmmll 2023. em23pmmmm-mmin 67/.
URL attps://aclantthooog. org/
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models, October 2023. URL http://arxiv.. wwwpiiv...g//////////eee abggabs/2309 14509. M.Xiv:2309.14500 [cs].
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit lyyer. One Thousand and One Pairs: A "novel" challenge for long-context language models, July 2024. URL 16264. 2Xiv:2446.16664 [cs].
Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit lyyer. FABLES: Evaluating faithfulness and content selection book-length summarization, April 2024. URL http:///rxiv, wrw.l....r//bbba htgg/bb////aee 2404, 01261. arXiv:2404.01266 [cs].
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. URL htt.ttp://aaxii.or///aa/aa/ 11916.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA Reading Comprehension Challenge. Transactions of the Association for Computaatoonal Linguistics, 6:317-328, 2018. doi: 10.1162 (..112/tacc_a_000 tacl_a_00023 URL https: / /acllaaccccccc www.acaaaantllyyaooaaoooooo wrg/oo 1023. Publisher: MIT Press.
Place: Cambridge, MA
Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schütze. LongForm: Optimiz- ing Instruction Tuning for Long Text Generation with Corpus Extraction, April 2023. URL absg/abs//ee 2304. 08460. srXiv:2304.08466 [cs].
http:://axiv.org///
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjingg Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training 2024.
12

---

Published as a conference paper at COLM 2025
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models, February 2024. URL http://aaxxi.....bbs/////ss. 2402.. 10002..14445 14848. utXiv:2402.14845 [cs].
Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. Preference leakage: contamination problem in llm-as-a-judge, 2025, URL https:://arxn wwwperiii.oo///booeeeeeeeeeeee otg/ wta.///022.01111 01534.
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. Symbolic thain-of-thought distillation: Small models can also "think" step-by-step. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2669-2679, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 vv/ 2202..acl-ooog.. 1500. 150. URL https::/acc.attooogg..rrr///////
acl- 150/.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation, 2024. URL https:
// //arxiv.org/abeb htrrg/abs/23ee 06259..
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwarzz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A hybrid transformer-mablee language model, 2024. URL https: http://xwiw..or///////////////// 198877
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summa- rization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computa- tional Linguistics. URL https: https://aclantholoogg www.or///w/oo.. org/we4--0133
Hao Liu and Pieter Abbeel. Blockwise parallel transformer for large context models. Ad- vances in neural information processing systems, 2023.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint ar Xiw:2310.01882 2023.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Compulational Linguistics, 12:157-173, 2024.
Quinn McNemar. Note on the sampling error of the difference between correlated propor- tions or percentages. Psychometrika, 12(2):153-157, 1947.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL https://aaxxt org/ aaaaabbs/1111 19393.
OpenAI. ol System Card, 01 11-system--car-22000 c222--224dd22111000- 105. .ddf.
December 2024.
URL http::///cdn.opnnall
AI Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and 271 others. Gpt-4 technical report, 2024. URL wttttttss//aiiii wwwp:/iii..ogg/o/aeeee wttp://wwwww.////////..... 08774.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Lim evaluators recognize and favor their own generations, 2024. URL h
13076.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019. URL https: / http://axxvvvvrooo wtt.o/////////.//rrrrrro 01703.
13

---

Published as a conference paper at COLM 2025
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient Context Window Extension of Large Language Models, November 2023. URL http://arxiv.orr/ abs/2309.0001 arXiv:2309.0007 [cs].
Chau Minh Pham, Simeng Sun, and Mohit lyyer. Suri: Multi-constrrint Instruction Follow- ing for Long-form Text Generation, June 2024. URL arXiv:2406.19371 [cs].
19371.
Ofir Press, Noah A. Smith, and Mike Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation, April 2022. URL BtLLttp://aavii.ooggaaaa//// 12409. arXiv:2108. 12409 [cs].
Haritz Puerto, Tilek Chubakov, Xiaodan Zhu, Harish Tayyar Madabushi, and Iryna Gurevych. Fine-tuning with divergent chains of thought boosts reasoning through self-correction in language models, 2024. URL
Zhenting Qi, Hongyin Luo, Xuliang Huang, Zhuokai Zhao, Yibo Jiang, Xiangjun Fan, Himabindu Lakkaraju, and James Glass. Quantifying generalization complexity for large language models, 2024. URL https://arxix: http:///arxi..oor///////00 wttp:////aab//....11111
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Techni- cal Report, December 2024. URL http:://rxxi...t////////// orgziii.orr//aaa wbtp 15115. 15. arXiv:2412.15115 [cs].
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulies Mike Gatford, et al. Okapi at TREC-3. British Library Research and Development Department, 1995.
Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning, March 2024. URL http: //arxiv. www.//orr//aoe 23310 16049. arXiv:2310.1604. [cs].
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding, November 2023. URL http: wtt.orgg/bb//////// 09864. arXiv:2104.00 09864 [cs].
//arxiv.org/abs/
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Gar- rett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1127 others. Gemini 1.5:
Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv. wwtpaaawwivioogrrbrrraaaaaaa wwwg//b/////g.///// 05530.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https: //qwenlm. git.ub. hoollloog/we2/ 5/.
Liang Wang, Nan Yang, Xingxing Zhang, Xiaolong Huang, and Furu Wei. Bootstrap your 18860.
own context length, 2024. URL https://arxxv.org/
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023a. URL http:tss///axxvv.rrgaa//////////
11171.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-gennrated instructions, 2023b. URL https://arxxx http://rwiivvoog//////2/2/2/ 10560.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Lan- 11903. 2Xiv:2201.11900
guage Models, January 2023. URL
14

---

Published as a conference paper at COLM 2025
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution, pp. 196-202. Springer, 1992.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing, 2020. URL https://arxiv. org/abs/1910.037
Wenhao Wu, Yizhong Wang, Yao Fu, Xiang Yue, Dawei Zhu, and Sujian Li. Long context alignment with short instructions and synthesized positions, 2024a. URL https://arxiv. org/abs/2488 03939..
Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Xiangju Lu, Junmin Zhu, and Wei Zhang. LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios, November 2024b. URL http://arxiiv.orgabss///// 0.7300//////11177777777.
07037. arXiv:241..07037.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective ing of Foundation Models, November 2023. URL MrXiv:2309.16033 [cs].
Scal- 16039.
Zheyang Xiong, Vasilis Papageorgiou, Kangwook Lee, and Dimitris Papailiopoulos. From artificial needles to real haystacks: Improving retrieval capabilities in iims by finetuning on synthetic data, 2024. URL https://arxivv http://rxii..orr/b///////// orgg/bs///06. 19292.
Peng Xu, Wei Ping, Xianchao Wu, Chejian Xu, Zihan Liu, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa 2: Bridging the gap to proprietary Ilms in long context and rag capabilities, 2024a. URL https: //arxiv. www.worg//ass..... orgg/bs/2//7. 14482.
Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. Pride and prejudice: Llm amplifies self-bias in self-refinement, 2024b. URL
https://arxiv.org www.rriw..ooggb/////// 11436.
An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Xiafei Qiu, Xingzhang Ren, Xinlong Yang, Yong Li, Zhiying Xu, and Zipeng Zhang. Qwen2.5-1M Technical Report, January 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL https: //axxiv org/abs/2305 10601.
Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effectively and thoroughly. Xrv preprint arXiv:2410.026994 2024.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in lims, 2025. URL htttps:/aaaxw..rrg//bbsttst// 2332/
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022. URL https://arxiv
/tt///as//222// 14465.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. oobench: Ex- tending long context evaluation beyond 100k tokens, 2024. URL https: //ariv/.//iiv./// http//arxi..ogabs/////// 13718.
15

---

Published as a conference paper at COLM 2025
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers, 2023. URL
021900
16

---

35.6% Published as a conference paper at COLM 2025
A Data Collection
A.1 Books used in manual analysis
Table 5 lists six books used in our manual analysis. These books are chosen due to the annotator's 's familiarity with the content, which eases the manual verification process.
Title
Author
Publication Year
Number Tokens
Number of Chapters
Anne the Island
Montgomery
1915
111,337
Alice in Wonderland
Lewis Carroll
1865
36,691
12
The Murder Roger Ackroyd
Agatha Christie
1926
98,602
27
The Picture of Dorian Gray
Oscar Wilde
1890
105368
20
Frankenstein
Mary Shelley
1818
97,574
24
The Adventures Tom Sawyer
Mark Twain
1876
97,968
Table 5: Six books used in our manual analysis. Books are chosen due to familiarity with the content.
A.2 Does memorization have an effect on claim verification performance?
We measure the performance of the models used for fine-tuning on our test set, with and without book text. We provide the book title and author name where the book text is not provided. Our hypothesis is that if the model does better than the random chance baseline (25% accuracy) without the book text, then the claims are either too easy or can be verified without even reasoning over the texts.
Models
No Text
With Text
WroLonng-nnsruee
0.0%
Llama-Instruct
0.0%
32.8%
Qwen-Instruct 0.0%
51.4%
Table 6: Accuracy on CLIPPER's test set (with and without book texts).
Table shows that all baseline models perform below random chance, significantly trailing behind the performance achieved when the book text is included in the claim verification prompt. These results indicate that even if a model has memorized the book texts or generated claims, such memorization does not affect its performance
the task itself.
A.3 Are the True/False claims distinguishable without the book texts?
We ask the question of whether distinguishing between True and False claims is inherently too easy. If so, then the high performance of the fine-tuned models may be attributed merely to their ability to detect formatting cues rather than actually reasoning. To investigate this, prompt both baseline and fine-tuned models to verify claims without providing any book texts or metadata. Our hypothesis is that if a model performs better than random chance under these conditions, then the claims are likely too easily distinguishable based their formatting alone.
Models
Before SFT
After SFT
Lallon--nssuuc 0.0%
25.2%
Llama-Instruet
20.2%
13.8%
Qwen-Instruct 21.7%
22,9%
Table 7: Accuracy on CLIPPER's test set (no book text or metadata provided).
As shown in Table 7, even after fine-tuning, the models perform only marginally above random guessing. We conclude that, without the contextual information from the book text, muue/False claims are not easily distinguishable.
17

---

Published as a conference paper at COLM 2025
A.4 Cost Analysis
Table shows the cost incurred by running each stage of our data synthesis pipeline. With the exception of deduplication, which is done by GPT-4o, each stage of the pipeline is performed by Claude. Table shows the estimated per claim cost for the naïve versus main approach based on estimated cost for books. For human annotation, NoCha (Karpinska et al., 2024) reports that their total cost of annotating 1,001 claim pairs is $3,327 USD, so each claim costs around $1.7.
Stage
Cost
Book summary generation
$0.0021
Chapter outline generation
$0.0107
Book-level claim synthesis
$0.0129
Chapter-level claim synthesis
$0.0172
Deduplication
$0.0021
Verification
$0.0064
Total
$0.0514
Table 8: Cost to run pipeline per claim (in US dollars, rounded to four decimal places).
NAIVE
CLIPPER
Cost per claim (book-level) Cost per claim (chap-level)
$0.09
$0.07
$0.04
$0.02
Table 9; Estimated cost for our NAÎVE CLIPPER approach (rounded two decimal places)
A.5 Prompts
Table 10 shows stages to construct CLIPPER, mapped to their corrrespondigg prompts.
Prompt
Figure
Chapter outline generation
Book summary generation
Chapter-level claim extraction
Book-level claim extraction
Claim deduplication
7
Claim verification
8, 9, 10, 11, 12
Chapter-level claim extraction (NAĪVE)
15
Book-level claim extraction (NAĪVE)
Table 10: Figure references for each prompt.
A.6 Using DeepSeek-Distill to measure CoT groundedness
We evaluate the model on 66 annotated claims from $2.3 and measure its agreement with human annotations (Table 11). Among the models tested, DeepSeek-Die Distill aligns most closely with human judgments, with only one instance of disagreement, outperforming other models like GPT-4o (10 disagreements) and LLaMMA-.1-700-Insstruu (3 disagreements). Although Llama-70B performs comparably, it fails to provide clear explanations for its decisions and instead generating generic reasoning messages that lack specificity to samples. Therefore, use DeepSeek-Diste Distill to measure CoT groundedness in our dataset.
18

---

Published as a conference paper at COLM 2025
Prompt for Chapter Outline Generation
Your task is create detai led and objective outline for Chapter {order) of book. You will be provided with text Chapter (order). Ensure that your outline faithfully represents the content of the text.
First, carefully read the text of Chapter (order): <current_chapter> [curr]
<current_chapter.
Finally, create outline for Chapter (order) using the following format:
<synopsis>A oee-sent summary of the current haapter.</syoosiis.
<events>A chronological list of at most 7 major events in the chapter. The list should formatted numbered list. Each event should be sentence long, describing specific details on
happens, where it happens, and which characters are involved. DO NOT include subjective interpretation of the events </events>
<characters>A numbered list of characters in the chapter. Include only those that mentioned in the major events. Each character should have the format: [ character full name]: [character role/relationship with the main characters], physical appearance (if mentioned)], [personality (if mentioned)], first seen at [the first setting the character is in], last seen at [last setting the character is in]. </chrrcerrs
Now, create an objective outline Chapter (order) based on provided text. Ensure that your outline is concise, coherent, and accurately represents the content of the chapter
<synopsis>
[One-sentence summary of the current chapter.. ]]
</synopsis>
<events>
[Event Specific details on specific details on what happens, where happens, and which characters are involved.
2. [Event 2: Specific details on what happens, where it happens, and which characters are involved.]
[Event 3: Specific details what happens, where it happens, and which characters are involved.]
4. [Event 4: Specific details on what happens, where it happens, and which characters are involved.]
5. [Event 5: Specific details what happens, where it happens, and which characters involved.]
[Event 6: Specific details on what happens, where it happens, and which characters are involved.]
7. [Event 7: Specific details on what happens, where it happens, and which characters involved.] </events>
<characters>
1. [Character 1: Character role/relationship with the main characters, physical appearance (if mentioned), personality (if mentioned), first seen at the first setting the character is in, last the last setting the character is in.] (Repeat the format for each character mentioned the major events.) </characters>
Remember to focus the objective representation of the chapter content and avoid adding personal opinions or interpretations. Good luck!
Figure 3: Prompt for generating chapter outlines in our dataset.
19

---

Published as a conference paper at COLM 2025
Prompt for Generating Book Summary
Your task is to write a summary for the book below, make sure to include vital information related to key events, backgrounds, settings, characters, their objectives,
motivations. You must briefly introduce characters (with their full name), places, and other major elements they are being mentioned for the first time in the summary. The book may feature non-linear narratives, flashbacks, switches between al ternate worlds viewpoints, etc. Therefore, you should organize the summary it presents consistent and chronological narrative. The summary must within 1000 words.
summary should span multiple paragraphs and should be written as a single continuous narrative, not as a list of bullet points or an outline. DO NOT include the book name in the summary.
#Book
[book]
#Summary
Figure 4: Prompt for generating chapter outlines in our dataset.
Judge Models
% Agreement
GPT-4o
84.8%
Jlama-3.1-70bInsttue
95.5%
DeepSeck-Distill-LLann
98.5%
Table 11: Percentage of times LLM judges for chain thought groundedness agree with our manual annotation 66 samples in Section 2.3.
20

---

Published as a conference paper at COLM 2025
Prompt for Extracting Chapter-level Claims
Your task is to create factual statements that incorporate multiple events from Chapter (chapter) of a book. These factual statements must be objective and specific, grounded in the specific chapter, and consistent with the entire book. The included events must be specific. The facts cannot contain any interpretations, speculations, or subjective statements. In addition to each fact, provide a minimally corrupted version of the fact, which sounds plausible but is wrong based on the entire book.
To create valid facts, follow these guidelines step-by-step:
Carefully read through the book.
2. In a <brainstorm> section: Identify different events that are strongly related to one another in single chapter.
Consider the relationship between these events, but NOT include this brainstorming the resulting fact. If meaningful relationship is found, move on to the next step; otherwise, proceed to the next fact.
3. Formulate your facts based this analysis. Ensure that your facts:
Contain single sentence
Are coherent with the entire
Include multiple detailed and strongly related events from a single chapter in the book
Are self-contained and independent of other facts
No part the fact should be subjective interpretations speculations
Do not contradict or duplicate existing facts
Do not contain chapter information (e.g. "In Chapter "..")
Do not contain quotes from the book.
4. Formulate minimally corrupted version of each fact.
Only one aspect of fact, such as an atomic detail or relationship between details, should be altered.
The corrupted fact should sound plausible but be clearly wrong based on the entire book.
The corrupted fact should be similar with the original fact in of length and sentence structure to make harder to verify solely based on surface-lewel features.
First, read the book:
<book><book)</book> Finally, review the existing facts:
caaisting_aacts>(xxittttt fttpr//aacceeeeeitnnffeeeeee facts>
Now, generate as many valid facts as possible based on the provided chapter.
Return "No meaningful fact" if there is no valid fact. Present your facts in the following format:
<facts>
<fact_1>
<brainstorm> (Your brainstorm notes here]</brainstorm
Fact: [Your fact here]
Fact Reasoning: [Your explanation here (including the chapter associated with each event)]
Source: [Chapter involved]
Corrupted Fact: [Your corrupted fact here]
Corrupted Fact Reasoning: [Your explanation here]
</fact_1>
[Continue with additional facts...]
</facts>
Remember to create facts that are objectively valid, coherent with the entire book, and demonstrate strang, meaningful relationships between the events from Chapter (chapter). No part the fact can include subjective interpretations or generalizations. The relationship must be meaningful. The included events must be SPECIFIC and DETAILED!!!!!
Figure 5: Prompt for extracting chapter-level claims
21

---

Published as a conference paper at COLM 2025
Prompt for Extracting Book-level Claims
Your task is to create factual statements that incorporate multiple events from multiple chapters of a book. These factual statements must be objective and specific, grounded in the involved chapters, and consistent with the entire book The included events must be specific. The facts cannot contain any interpretations, speculations, or subjective statements. In addition to each fact, provide a minimally corrupted version of the fact, which sounds plausible but is wrong based on the entire book.
To create valid facts, follow these guidelines step-by-step:
Carefully read through the book.
2. In a <brainstorm> section:
Identify different events that are strongly related to one another in multiple chapters.
Consider the relationship between these events, but NOT include this brainstorming the resulting fact. If meaningful relationship is found, move on to the next step; otherwise, proceed to the next fact.
3. Formulate your facts based this analysis. Ensure that your facts:
Contain single sentence
Are coherent with the entire
Include mul multiple detailed and strongly related events from multiple chapters in the book
Are self-contained and independent of other facts
No part of the fact should be subjective interpretations speculations
Do not contradict or duplicate existing facts
Do not contain chapter information (e.g. "In Chapter "..")
Do not contain quotes from the book.
Formulate minimally corrupted version of each fact.
Only one aspect of fact, such as an atomic detail or relationship between details, should be altered.
The corrupted fact should sound plausible but be clearly wrong based on the entire book.
The corrupted fact should be similar with the original fact in of length and sentence structure to make harder to verify solely based on surface-lewel features.
First, read the book:
<book><book)<<book> Finally, review the existing facts:
(treeeissiin_faatssse/eiiissttee faa.fass/eexiitinnffaaaaaae facts>
Now, generate as many valid facts as possible based on the provided book. Return
"No meaningful fact" if there is no valid fact. Present your facts in the following format:
<facts>
<fact_1>
<brainstorm>>(oo (Your brainstorm notes here]</brainstorm Fact: [Your fact here]
Fact Reasoning: [Your explanation here (including the chapter associated with each event)]
Source: [Chapters involved]
Corrupted Fact: [Your corrupted fact here]
Corrupted Fact Reasoning: [Your explanation here]
</fact_1>
[Continue with additional facts...]
</facts>
Remember to create facts that are objectively valid, coherent with the book, and demonstrate strong, meaningful relationships between the events from multiple chapters. part
fact can include subjective interpretations or generalizations. The relationship must be meaningful. The included events must be SPECIFIC and DETAILED!!!!!
Figure 6: Prompt for extracting book-level claims
22

---

Published as a conference paper at COLM 2025
Prompt for Deduplicating Claims
You will be given list of facts. Your task is to identify all duplicate facts within this list. fact is considered a duplicate if it is exactly the same another fact, if it contains the information as another fact, even if worded differently.
Here is the list of facts:
fract_lis>> [fact_list]
</fact_list>
To identify the duplicate facts, follow these guidelines step by step:
Read through the list of facts carefully.
2. Identify any facts that are exact duplicates of each other, or that convey the same atomic information using different wording.
3. Return list of facts that duplicates of other, along with explanation why they are duplicates.
DO NOT return facts that are not duplicates of any other fact in the list.
Here's an example of how output should look:
<example>
dexample_fact_list
1. Jim worked hard, so he got a promotion.
2. Jim's hard work paid off, and he was promoted.
Jim and Sarah worked together the project.
Jim worked hard, so received praise from his boss, who promoted him. </example_fact_list>
<example_answar >
1, 2: These two facts convey the same atomic information but are worded differently.
Teexmmpleansseer
</example>
Remember, your goal is to identify duplicate facts, whether they are exact matches or convey the same information in different words. Be thorough in your analysis and clear in your explanations. If there is no duplication in the list, output "No duplicates found."
<answer>
[Index of duplicate facts, separated by commas]: [Explanation of why they are duplicates]
[Index of duplicate facts, separated by commas]: [Explanation of why they duplicates]
(... and so on) </answer>
Figure 7: Prompt for de-duplicating claims
23

---

Published as a conference paper at COLM 2025
Prompt for Verifying Claims with GPT-4o (Part 1)
You will receive book summary, chapter outline, and claim extracted from that outline. Your task is verify whether the claim contains detailed information, presents meaningful relationship, and shows consistency with both the book summary and chapter outline.
To verify the claim, follow these steps:
Read the summary, outline, and claim carefully understand context.
2. Decompose the claim into atomic parts.
3. Analyze each atomic part:
a. Is the part grounded in the events?
b. Does this part contradict any information in the summary outline? Keep mind that some books may have discontinuous plots or events, so just because detail is mentioned before or after another in the summary does not mean they are temporally related.
Evaluate the relationship between the atomic parts: a. the relationship objectively valid and meaningful?
relationship subjective interpretation or assumption not explicitly stated in the summary or outline?
b.
c. Does the relationship make sense based on the book summary and chapter outline?
Based on your analysis, provide your reasoning and verification result. Your reasoning should explain why you believe the claim is or is not valid based on the information provided in the book surmary and chapter outline.
First, read the book sunmary:
<book_summrry> (book_summary) </book_sunmary>
Next, review chapter outline:
<chapter_outline>
(chapter_outline)
Finally, consider the following claim: <claim>
(claim)
</claim>
Here are two examples of valid and invalid claims:
Figure 8: Prompt for verifying claims with GPT-4o (Part
24

---

Published as a conference paper at COLM 2025
Prompt for Verifying Claims with GPT--oo (Part 2)
<example_12
<example_summan summaryy
Laura Hand, Daniel Knowe, and Mo Gorch mysteriously return from realm of death with altered memories, orchestrated by their enigmatic music teacher, Mr. Anabin and the sinister Bogomil. As they grapple with their new realities, including Mo s discovery of his grandmother's death and Daniel's Ms complex feelings for Laura's s sister, Susannah, they face trials set by Anabin and Bogomil remain the living world. Alongside Bowie, another returned soul, they uncover the truth about their deaths while dealing with eerie supernatural encounters. Laura ss newfound magical abilities strain her relationship with Susannah, leading to increasing tensions Susannah begins to remember the truth.
As the trio navigates their altered lives, they become entangled in larger, dangerous game orchestrated by Malo Mogge, Anabin, and Bogomil, who guard the door be twenn 1ife and death. Mo and Susannah discover that a Harmony guitar, hidden by Susannah, is the key sought by Malo Mogge, powerful entity seeking immense power. The story culminates in chaotic battle in Lovesend, where Laura consumed by grief, vows to kill Malo Mogge. After absorbing Mogge's 's magic and becoming powerful goddess, Laura faces the challenge of balancing her divine powers with her passion far music, while the other characters embrace their new roles.
saxample_summary"
## Chapter 15 outline
1. Susannah gets frustrated about Laura and Daniel being close to each other and smashes Laura's old guitar a fit anger and frustration.
2. Mo eats a breakfast casserole made by Jenny and buys doughnuts and bagels on his way to Laura's house, feeling a mix of hunger and sadness.
3. Mo arrives at the Hands' house, where Laura and Daniel are eating ramen to satisfy their unusual hunger. Daniel reveals that
Laura have swapped ears due to Anabin's magic, and they discuss the implications of this mistake.
5. Mo shares his encounter with mysterious figure outside his house, leading to heated discussion about Bogamil and Mr. Anabin.
The trio creates list of goals navigate their situation, including magic.
staying alive, figuring out how they died, and learning
They attempt to perform magic by trying to transform a saltshaker into a hairless cat but fail, leading to further frustration.
Mo leaves the Hands' house, and they discover that the entire yard and house are covered thousands moths, adding to the surreal nature of their situation.
9. Laura finds the broken guitar pieces in her room, causing confusion and suspicion among the trio. eexxmmle_chapter_outllmm
claim_1>
Laura finds the broken guitar pieces in her room, which Susannah smashed in fit of anger and frustration. </example_claim_IP
eexample_vrrification.
<reasoning>
1. Analysis of each atomic part:
a. Laura finds the broken guitan pieces in her room: This part grounded in event of Chapter 15.
b. Susannah smashed the guitar in fit of anger and frustration: This part
is grounded in event 1 of Chapter 15.
Analysis of the relationship between atomic parts:
a. The relationship is temporal, thus valid.
Figure 9; Prompt for verifying claims with GPT-4o (Part 2)
25

---

Published as a conference paper at COLM 2025
Prompt for Verifying Claims with GPT-4o (Part 3)
b. The relationship is explicit and not subjective interpretation. c. The relationship is grounded in the chapter outline. There is contradicting information in the summary. Since all considerations are satisfied, claim VALID. </reasoning>
<result>
VALID
</result>
</example_verificattioo
<example_clain_2>
Mo shares his encounter with mysterious figure because Laura and Daniel discuss
implications their swapped hands Mr. Anabin 's magic. </example_clain_2>
<example.errifcaation.
<reasoning>
1. Analysis of atomic parts:
a. Mo shares his encounter with a mysterious figure: This part is grounded in event 5 of Chapter 15.
b. Laura and Daniel discuss the implications of their swapped hands due Mr. Anabin's s magic: There no mention of hands being swapped. Even though event discusses the swapping of ears, it does not relate to hands.
2. Analysis of the relationship between atomic parts:
The relationship is NOT VALID because there is direct 1ink between Mo sharing his encounter and Laura and Daniel discussing the implications of their swapped hands.
b. The relationship is a subjective interpretation and not explicitly grounded in the summary or outline.
c. The relationship is grounded contradicting information in the summary.
the chapter outline. There is
Since 1a., 2a., and 2b. are not satisfied, the claim is INVALID. </reasoning>
<result>
INVALID
</result>
</example_verfficcatoo....
Figure 10: Prompt for verifying claims with GPT-4o (Part 3)
26

---

Published as a conference paper at COLM 2025
Prompt for Verifying Claims with GPT--oo (Part
<example_cke caall__ccaim__
Jenny cooks breakfast for Mo because he feels a mix of hunger and sadness. </example_clai__3
cexample_verffication
<reasoning>
1. Analysis of atomic parts:
Jenny cooks breakfast for Mo: This part is grounded in event 2 of Chapter
15.
b. feels a mix hunger sadness: This part is grounded event of
Chapter 15.
2. Analysis of the relationship between atomic parts:
The relationship is INVALID. There is indication that Jenny cooked because felt mix hunger and sadness. The events are happening simultaneously but are not causally connected.
breakfast
b. The relationship is subjective interpretation and not explicitly grounded in the summary or outline.
c. The relationship is grounded the chapter outline. There is no contradicting information in the summary.
Since 2a. and 2b. are not satisfied, the claim is INVALID. </reasoning>
<result>
INVALID
</result>
eexxmple_veriffiaaton
</example_1>
<example_2>
<example_sumee suuleerrmrrr>>
Sarah Lee discovers an ancient wooden box with strange symbols in her attic, which contains journal revealing the history of secret society and prophecy about the return of powerful being known "The Shadow. events begin to plague her town, Sarah, along with her friend Mark, uncovers clues that connect these accurrences to the prophecy. They find a hidden chamber beneath the town containing ancient texts and artifacts, including a weapon capable of banishing "The Shadow. " With this weapon, they confrent member of the secret society attempts summon "The Shadow," and after tense battle Sarah successfully uses the weapon to banish the being, restoring peace to the town.
unusual
Throughout the story, the connection between the ancient artifact, the journal, and the unfolding events reveals the central role of the wooden box and the secret society in the impending danger. Sarah and Mark's journey highlights their struggle to protect their town from supernatural forces while deciphering the mysterious symbols and prophecies tied the powerful entity, "The Shadow. </example_summrry
<example_chapter <hapter_outtlne> ## Chapter 3 outline
Figure 11: Prompt for verifying claims with GPT-4o (Part 4)
27

---

Published as a conference paper at COLM 2025
Prompt for Verifying Claims with GPT--o (Part 5)
1. Sarah discovers ancient artifact in her attic, an intricately carved wooden box with strange symbols.
2. She finds old journal the box, detailing the history of secret society that once protected town.
3. The journal reveals prophecy about the return of powerful being known as
The Shadow."
Sarah decides keep the discovery herself, fearing that revealing it would cause panic.
## Chapter 8 outline
Sarah begins to notice strange occurrences around town, like unusual weather patterns and eerle shadows.
She consults the journal again and discovers a passage that seems to describe unusual event patterns as signs of "The Shadow's 's" return.
3. Sarah's 's friend Mark, who is local historian, suggests that they investigate further by visiting the town's library.
At library, they find more texts related the secret society and "The Shadow."
5. Mark went hame to rest after long day, where he met his mother. <example_chapeer.ootllin
<example_claim>
Sarah decides to keep the discovery to herself, which reveals the challenges Mark and Sarah's friendship.
</example_claim
<example_veri verificaiooo <reasoning>
1. Analysis of atomic parts:
a. Sarah decides to keep the discovery herself: This part is grounded in event of Chapter
b. "reveals the challenges in Mark and Sarah's friendship": This part a subjective interpretation and not explicitly grounded in the surmary or outline.
2. Analysis of the relationship between atomic parts:
a. The relationship is not meaningful, there is no direct connection
between Sarah keeping the discovery herself and revealing challenges Mark and Sarah's s friendship.
b. The relationship is subjective interpretation and not explicitly grounded in the summary outline.
C. The relationship does not contradict any information the summary or outline.
Since 1b., 2a., and 2b. are not satisfied, the claim is INVALID. </reasoning>
<result>
INVALID </result>
<example_claim
Sarah finds old journal in the room, which prompted Mark to go home and meet his mother.
</example_claim>
eexample_verification
<reasoning>
Analysis atomic parts: a. Sarah finds an old journal in the room: This part is grounded in event of Chapter 3.
b. Mark went home to rest after long day, where he met his mother: This part grounded in event of Chapter
Figure 12: Prompt for verifying claims with GPT-4o (Part 5)
28

---

summary or Published as a conference paper at COLM 2025
Prompt for Verifying Claims with GITT-4 (Part 6)
Analysis of the relationship between atomic parts:
a. The relationship is not meaningful, as there no direct connection
between Sarah finding the journal and Mark going home to meet his mother.
b. The relationship is subjective interpretation and not explicitly grounded in the summary outline.
c. The relationship does not contradict any information outline.
Since 2a. and 2b. are not satisfied, the claim is INVALID. </reasoning>
<result>
INVALID
</result>
</example_2>
Now, it's your turn to verify the claim based on the provided book summary, chapter outline, and claim. Present your response in the following format: <verification>
<reasoning>
[Provide your detailed reasoning here, explaining why the claim or not meaningful and coherent with the book summary and chapter outline.]
</reasoning>
<result>
[State whether claim VALID or INVALID. Use VALID if the claim portrays meaningful relationship and is coherent with the book summary and chapter outline. Use INVALID if it does not.] </result>
</verification>
Remember to base your verification solely on the information provided in the book summary, chapter outline, and the claim itself. Verify that the relationship makes sense and is objectively valid. Do not introduce external information or make assumptions beyond what given.
Figure 13: Prompt for verifying claims with GPT-4o (Part 6)
29

---

Published as a conference paper at COLM 2025
Prompt for Generating Book-level Claims in NAIVE
Your task is create factual statements that incorporate multiple events from multiple chapters of a book. These factual statements must objective and specific, grounded in the involved chapters, and consistent with the entire book The included events must be specific. The facts cannot contain any interpretations, speculations, fact, provide minimally corrupted version of the fact, which sounds plausible but is wrong based on the entire book.
subjective statements. In addition to each
To create valid facts, follow these guidelines step-by-step:
1. Carefully read through the book.
2. In <brainstorm> section:
Identify different events that are strongly related one another in multiple chapters.
Consider the relationship between these events, but DO NOT include this brainstorming in the resulting fact. If meaningful relationship is found, to the next step; otherwise, proceed to the next fact.
Formulate your facts based on this analysis. Ensure that your facts:
Contain a single sentence
Are coherent with the entire book
Include multiple detailed and strongly related events from multiple book
chapters
Are self-contained
independent of other facts
No part of the fact should be subjective interpretations or speculations
Do not contradict or duplicate existing facts
Do not contain chapter information (e.g. "In Chapter ...")
Do not contain quotes from the book.
4. Formulate a minimally corrupted version of each fact.
Only one aspect of the fact, such as an atomic detail or the relationship between details, should be altered.
The corrupted fact should sound plausible but clearly wrong based on the entire book.
The corrupted fact should be similar with the original fact in terms of length and sentence structure to make it harder to verify solely based on surface-level features.
First, read the book:
<book>(book)</bolk>>
Finally, review the existing facts:
Now, generate as many valid facts as possible based on provided book. Return "No meaningful fact" if there is no valid fact. Present your facts in the following format:
<facts>
<fact_1>
(rrainstor>>(oorr brainstorm notes here]</brainstorm Fact: [Your fact here]
Fact Reasoning: [Your explanation here (including the chapter associated with each event)]
Source: [Chapters involved]
Corrupted Fact: [Your corrupted fact here]
Corrupted Fact Reasoning: [Your explanation here]
</fact_1>
[Continue with additional facts...]
</facts>
Remember to create facts that are objectively valid, coherent with the book, and demonstrate strong, meaningful relationships between the events from multiple chapters. part
fact can include subjective interpretations or generalizations. The relationship must be meaningful. The included events must be SPECIFIC and !ITAILED!!!!!
Figure 14: Prompt for generating book-level claims in NAÎVE.
30

---

Published as a conference paper at COLM 2025
Prompt for Generating Chapter-level Claims in NAÎVE
Your task is to create factual statements that incorporate multiple events from Chapter (chapter) of a book. These factual statements must be objective and specific, grounded in the specific chapter, and consistent with the entire book. The included events must be specific. The facts cannot contain any interpretations, speculations, or subjective statements. In addition to each fact, provide a minimally corrupted version of the fact, which sounds plausible but is wrong based on the entire book.
To create valid facts, follow these guidelines step-by-step:
Carefully read through the book.
2. In a <brainstorm> section: Identify different events that are strongly related to one another in single chapter.
Consider the relationship between these events, but NOT include this resulting fact. If meaningful relationship is found, move on to the next step; otherwise, proceed to the next fact.
brainstorming
3. Formulate your facts based this analysis. Ensure that your facts:
Contain single sentence
coherent with the entire book
Include multiple detailed and strongly related events from a single chapter in the book
Are self-contained and independent of other facts
No part of the fact should be subjective interpretations speculations
Do not contradict or duplicate existing facts
Do not contain chapter information (e.g. "In Chapter x, ...")
Do not contain quotes from the book.
4. Formulate minimally corrupted version of each fact.
Only one aspect of the fact, such
atomic detail the relationship
between details, should be altered.
The corrupted fact should sound plausible but be clearly wrong based on the entire book.
The corrupted fact should be similar with the original fact in terms of length and sentence structure make harder verify solely based on surface-level features.
First, read the book:
<book><book)</bookk
Finally, review the existing facts:
cexisting_facts>(exiss wwt.eeeteceeiitnngffffeeeeiiiiie facts>
Now, generate as many valid facts as possible based on the provided chapter.
Return "No meaningful fact" if there is no valid fact. Present your facts in the following format:
<facts>
<>act_1>
<brainstorm>(Von brainstorm notes here)</brainstorm>
Fact: [Your fact here]
Fact Reasoning: [Your explanation here (including the chapter associated with each event)]
Source: [Chapter involved]
Corrupted Fact: [Your corrupted fact here]
Corrupted Fact Reasoning: [Your explanation here]
</fact_1>
[Continue with additional facts...]
</facts>
Remember to create facts that are objectively valid, coherent with the entire book, and demonstrate strong, meaningful relationships between the events from Chapter (chapter). part
fact can include subjective interpretations or generalizations. The relationship must be meaningful. The included events must be SPECIFIC and DETAILED!!!!!
Figure 15: Prompt for generating chapter-level claims in NAÎVE.
31

---

Published as a conference paper at COLM 2025
B
Training
B.1
Codebases
To fine-tune models in the Llama family, we adopt the ProLong codebase," 17 which inte- grates PyTorch (Paszke et al., 2019) and Hugging Face (Wolf et al., 2020) for model training, FlashAttention-2 (Dao, 2023) for efficient attention computation, and DeepSpeed Ulysses (Ja- cobs al., 2023) for sequence parallelism, enabling training across A100 GPUs. For fine-tuning Qwen-Instruct (Qwen-Instruct) we use the 360-LlamaPactory codebase, modification of Llama-Factory" that incorporates sequence parallelism via zigzag ring attention (Liu & Abbeel, 2023; Liu et al., 2023). We choose ProLong-Baas BBse (ProLong-baase over ProLong-Instruc Instruct Prooong-instrun
based on a small fine-tuning experiment, where fine-tune both ProLLonginstruc and ProLong-base 2K training examples. This ex- periment shows that ProLong-base outperforms ProLong-inss by 61.6% and 59.7%, respectively.
B.2 Hyperparameter Tuning
Table 12 summarizes the performance of each configuration from our hyperparameter tuning experiment on 100 samples from CLIPPER's dev set.
Learning Rate
Batch Size
Dev Set Accuracy
le-5
16
26%
1e-6
16
74%
1e-7
16
71 %
le-5
32
34%
le-6
32
73%
le-7
32
69%
Table 12: Hyperparameter tuning results. Each model is fine-tuned for epoch and tested on a subset of 100 samples from our dev set.
C Evaluation
C.1 Configuration for cBBnch QA Evaluation
In HELMET (Yen et al., 2024), for the ooench QA task, the default configuration sets the output maximum length to 10 tokens and uses ROUGE F1 (Lin, 2004) as the evaluation metric. Upon closer examination of the outputs from both models, we identify critical flaws in the default setup. These findings eventually motivate us remove the maximum length restriction and adopt the LLM-as-a-judge evaluation approach using GPT-4o the judge. Below, we provide more details on our analysis.
Setting max output length to 10 tokens frequently cause truncations: In Table 13, show the taxonomy we derive from our analysis. Here, we define a truncation to be when model's response is heavily cut off, making it impossible determine the correctness of the output. Out of all 100 evaluated examples, Qwen2.5-7B-Insttune times, while Qwen-CLIPPER's outputs get truncated 45 times. After removing the 10-token maximum length r we observe that of the 24 previously truncated outputs from Qwenn-nsttuucsshould be counted correct. For Qwen-CLIPPER this correction is even
outputs get truncated 24
httpss//ggghuhh.co/hhhygggg////
20 the maximum length limit, Owen2.5-7B-Instt outputs are average 20.9 tokens long, and Qwen-CLIPPER IPPER's outputs have 25.8 tokens on average.
32

---

Published as a conference paper at COLM 2025
Finetuning Prompt
System Message
You are an expert at verifying claims from fictional narratives.
User Message
You are provided with a context and a statement. Your task is to carefully read the context and then determine whether the statement is TRUE FALSE.
Answer TRUE the statement is true in its entirety based on the context provided. Answer FALSE if any part of the statement is false based on the context provided.
text)</context> <cortert-fooot_eexxx//ootttttt
paatement//caaintteee///stttttte
<question>Based on the context provided, is the above statement TRUE or FALSE? </question>
First provide an explanation of your decision-making process, and then provide your final answer. Use the following format:
YoXplanation>YOUR YOUR EXPLANATION<</expaanttio <answer> YOUR ANSWER</answer>
Assistant Message
<explanation>Here are the relevant detail from the text:
Chapter x:
Chapter .....
These chapters have relationship z, which makes the claim TRUE/FALSE.c/explanee
aanwee>TRRUFFALSSE//anwww
How we structure our fine-tuning prompts, which include system message, user message, and assistant message. Placeholders (colored in light cayenne) will be replaced with actual text from the dataset. The contents between <context>, <explanation>, and <answer> tags are generated (Section 2.3).
Figure 16:
33

---

Published as a conference paper at COLM 2025
CATEGORY
DERNITION
EXAMPLE QUESTION
EXAMPLE GROUND-TRUTH
EXAMPLE MODEL ANSWER
Cases that should be counted as incorrect
Wrong
model's answer is idently wrong.
the Edna moves into New Orleans called?
The pigeon
not provide information about Edna moving
The
Cases that should be counted as correct
Full model's answer perfectly matches the ground-truth
Which among nalisa, Seb, Peytun, Gannonmatie not Mrs. Bronwyn's child?
Peyton
Peyton
Cornect (Phras- ing)
The model's answer is curnect, has dif- ferent phrasing than the ground-truth
How old is Felicity of his ration?
Thirty
Felicity is 30 years
Correct (Long)
The model's answer is longer
What kind of pet does
cat
Madame Bowen keeps cat as a pet.
cornect, than the ground-truth.
Madame Bowen keep?
Correct (Short)
model's answer is correct, but it is shorter the ground-truth.
Why out jail?
has been re- Teigan
Paroled.
leased on parole.
Cases where correctness is ambiguous
Truncation
The model's answer has been heavily truncated, making it impossible to tell correctneess answer.
What is be built place Lars home Wickham Place?
Flats
The house Wick-
ham Place is to placed
Table 13: Taxonomy from our analysis on the coBench QA outputs of 1eemn..5-7B-nnstruee and Qwen-CLIPPER. Example model outputs are from Qwen-CLIPPER except the one for "Correct (Short)", which is from Owen2.5-7B-Instruu (all generated under the default setup where maximum output tokens is set to 10).
CATEGORY
QWEN-INST
QWEN-BC
Cases that should counted as incorrect
Wrong
26
Cases that should be counted as correct
Full match
17
Correct (Phras-
3
ing)
Correct (Long)
13
18
Correct (Short)
Cases where correctness is ambiguous
Truncation
24
45
Table 14: Raw counts of taxonomy categories for Qwen-Instruct and Qwen-CLIPPER, with outputs generated using the default maximum length of 10 tokens.
more significant, with 25 of the 45 truncated outputs being technically correct. We combine these numbers with numbers from the four rows in Table 14 that indicate correctness, and find that Qwen2.5-7B-Instrruc has an overall accuracy of 44%, while Qwen-CLIPPER has 54%.
ROUGE F1 is not reliable metric: If we use ROUGE Fl the metric, Qwen2.5-7B- Instruct achieves a score of 27.4, while ODwen-CLiPPERachieve a score of 18.0. This result sharply contrasts with the accuracies we obtain in the preceding paragraph, and does not
34

---

Published as a conference paper at COLM 2025
QUESTION
GROUND-TRUTII
MODEL ANSWER
ROUGE
EXPLANATION
How old Felicity at the start of his nar-
Thirty
Felicity is years old at
The model is correct, but it uses the numerical form of the number.
ration?
What gender does Harris predict Cal will be?
MALE
Harris predicts that 0 Cal will be boy.
The model is correct, but it phrases it differently, resulting in no word overlap.
When is Jarod's birthday?
NOVEMBER9
Jarod's birthday is on November 16
0.22
The model is completely wrong, but it gets the same score as the model answer the row below, which contains
which state Go- Minnesota pher Prairie located?
Gopher Prairic cated Minnesota. This is
The model is correct, but gets the same score the wrong model answer above, just because the output is much longer than the ground- truth.
Table 15: Examples showing that ROUGE-F1 is an unreliable metric.
reflect the actual performance of the models. Lots of prior work have shown that ROUGE correlates poorly with human judgment (Goyal et al., 2023; Chang et al., 2024). Our manual analysis reveals that this metric overly sensitive length, and does not capture the correctness of the model outputs. We show several examples in Table 15.
D Results
D.1 Impact of chapter distance and book length on test set performance
Figure 17 shows that test set accuracy peaks when the distance between chapters in a claim around 40-60K tokens (roughly the midpoint of a book). When that gap shrinks below or stretches beyond 60K tokens, performance dips by about 10%, leaving no definitive pattern beyond this sweet spot.
Average Accuracy
1.0
0.8
0.6
0.4
0.2
0.0
0-20K
20K-40K
40K-60K
60K-80K
80K+
Chapter distance (in tokens)
Figure 17: Accuracy of CL IPPEE-Prroloog-- balancee the test set (book-level claims), grouped by the distance (in tokens) between source events in each claim.
We also find that overall book length does not strongly influence accuracy, except in cases where the text exceeds 110K tokens. In these longer works, accuracy is about 5% higher than in shorter books, shown in Figure 18. While this slight edge may hint at advantages
35

---

Published as a conference paper at COLM 2025
in expansive narratives, the model's broader performance remains steady across most book lengths.
1.0
0.8
Accuracy
0.6
0.4
0.2
0.0
<90K
90K-110K Book Length (in Tokens)
>11K
Figure 18: ProLong-CLIPPER's performance on test set, grouped by the number of tokens in each book.
We finally examine the possible effect of event placement on ProLong- wwonnn-LLIPERRRR performance on the test set. Interestingly, there is strong "lost-n-thhemmidde effect regarding event placement in the book (Liu et al., 2024). As shown in Figure 19, accuracy is usually the highest when the claim involves events that appear at the beginning (0-0.4, around 82%) rather than the end the book (0.8-1, around 78%).
1.0
Average Accuracy
0.8
0.6
0.4
0.2
0.0
0-0.2
0.2-0.4
0.4-0.6
0.6-0.8
0.8-1
Event Placement in a Book (Chapter Position Scaled by Total Chapters)
Figure 19: Accuracy of Prol MooLoon--LIPPEA on the test set (chapter-level claims), grouped by the event placement the book (0-0.2 includes events are at the beginning, while 0.8-1 includes events towards the end).
36

---

Published as a conference paper at COLM 2025
D.2 False claim error analysis
In Table 16, we provide detailed definitions for each category from the false claim error analysis in $4.6. To explore instances where fine-tuned models still struggle, we conduct an in-depth analysis of Qwen-CLIPPER outputs. Of the 1,000 book-level claims in the test set, the model fails to verify 37 true claims and 97 false claims. This pattern is consistent with findings from NoCha (Karpinska et al., 2024), which highlight that models tend to have greater difficulty verifying false claims. Notably, in 95 cases, the model successfully validates the true claim but fails to validate the corresponding false claim. This raises an important question: What specific perturbations make a false claim appear true to the model?
Through careful manual analysis, we derive a taxonomy of such perturbations and present them in Table The most frequent perturbations are changes to events (43.2%) and people (31.6%), such altering actions or misattributing roles. Less frequent but notable are modifications to objects (15.8%), locations (13.7%), time (6.3%), and affect (4.2%). All these perturbations introduce plausible-sounding variations that the model may struggle to detect without fully understanding the narrative.
CATEGORY
DEFINITION
Event
Refers to the alteration or misrepresentation of the actions, occurrences, pro- cesses described in a claim.
Person
Involves substituting or misattributing individuals involved claim.
Object
Concerns the manipulation or substitution of physical items or artifacts mentioned in a claim.
Location
Relates to changing or misrepresenting the places where events occur.
Time
Pertains to the sequencing or timing of events being distorted swapped.
Affect
Deals with altering the emotional state, attitude, or disposition described claim.
Table 16: Definitions for each category of perturbations that cause false claim to be misclassified as true in the error analysis in $4.6.
D.3 Full results on LM Harness and HELMET
Table 17 shows the results of all models on popular short-form benchmarks. Overall, our fine-tuned models, especially baseline models even though it has been fine-tuned on longer data. Table 18 shows the results of HELMET on recall, RAG, passage re-ranking, and retrieval tasks. Overall, fine- tuned models achieve synthetic recall and RAG scores comparable to the baseline models, while generally delivering improved re-ranking and more robust ICL performance.
IPPÉR, do not degrade that significantly from the
Models
Math Ivi5
GPQA
MMLU-Pro
Are-Challenge
GSMSK
HellaSwag
WineGrande
Llama-Instruct
59.35
50.93
12.81
31.96
37.77
51.54
75.06
59:05
74.19
Owen--nstruct
54.00
54,60
24.80
33.40
43.80
53,20
77.70
61.80
69.20
Prolong instruct-noft
5.28
32.43
68.06
Owen-CLIPPRR
50.65
55.50
22.51
33.82
44.49
53.84
78.32
61.71
69.06
ProLong-CLIPPEE
7.91
48.03
5.42
27,88
32.35
50.68
60.80
60:44
73.24
Llama-CLiPPEr
12.77
Prol ong WritingPrompts
11.75
47.32
3.50
30,73
26.29
30.51
39.04
76.36
70.64
ProLong CL IPPER-book
6.39
49.31
5.53
29.47
32.38
54.78
62.02
79.27
72.93
Prol ong-El C1 TPPER-choptr
4.39
5.63
29.77
32.35
34.27
61.22
79.14
74.27
Table 17: Performance popular short-form benchmarks (evaluated using Language
Model Evaluation Harness).
D.4 Performance of ProLong-Bass Base on claim verification and narrative understanding benchmarks
Table 19 shows accuracy of ProLong-Basene long-context reasoning and narrative under- standing benchmarks. Even though ProLong-Bard Base's test set performance is much worse
37

---

Published as a conference paper at COLM 2025
Synthatic Recali (Rtuke)
RAG
He-noking
Model
reall
rihh,rv
160,40
rgp
Inrogg
hotpotge
mnmarooo
Liama-nntract
78,25
16
68.37
B0097
36
1506
72
Ovendsstued
388,
14.0
13.3
1838
35.3
47.3
26.0
764
2001
60
11.0
70
Froaann CLUPeON
99
20.25
99
82.5
51.58
$7.50
72
QwandLoMer
81
45
28.15
66.5
36
3.17
100
65.5
65.17
53.00
22.90
Table 18: Performance on HELMET for recall, RAG, passage re-ranking, and retrieval tasks. Fine-tuned models achieve synthetic recall and RAG scores comparable the baseline models, while generally delivering improved re-ranking and more robust ICL performance.
than ProLong-Instrucc performance on other narrative understanding tasks is comparable between the two models.
Narrative
MuSR
oeBench QA
23.9%
46.0%
39.8%
42.5%
Table 19: Performance ProLong- Base on long-conterd reasoning and narrative under- standing benchmarks. Even though ProLong-Basse test set performance much worse than ProLong- Mrnoon-nnttuut,, performance on other narrative understanding tasks is comparable between the two models.
ProLong-CP-book
ProLong-CP-chap
Test-book
74.8%
78.2%
Test-chapter 75.2%
80.2%
Overall
75.0%
79.2%
Table 20: Test set performance of models trained exclusively on either book-level claims or chapter-level claims, with accuracy measured for book-level, chapter-level, and overall claims. CP stands for CLIPPER.
D.5 Performance of retrieval-augmennte baselines CLIPPER-test
To better isolate the contribution of CLIPPER, we present the performance of retrieval- augmented approach on CLIPPER-test. Specifically, we report results of an approach where we use BM25 (Robertson et al., 1995) to retrieve the top 50 relevant book passages (each no longer than 256 words) for a given claim and prompting our original baselines with these passages instead of the full book text.
As seen in Table 22, these RAG baselines (denoted by BM25 model name) outperform our original LLaMA and ProLong baselines (+9%), but not the Qwen baseline (-15%). Compared to our CLIPPER models, however, these RAG baselines still lag by 22-50%. It important to note that RAG approaches do not consistently outperform long-context models in long-form claim verification. For instance, Karpinska et al. (2024) and Kim et al. (2024) benchmark similar RAG setup where GPT-4o is provided only with BM25-retrieved passages. As shown in Table 3 of Karpinska et al. (2024), the RAG versions (k 5, 25, 50) consistently underperform the setting where GPT-40 has no retrieval support.
D.6 Performance margin of CLIPPER
Table 23 lists statistical test results for the performance reported in Table 2. For CLIPPER- test, NoCha, and MuSR, which return binary True/Falls, predictions, use McNemar's 's test McNemar (1947). For NarrativeQA and InfiniBenchQA, which return ordinal scores ranging from 0 to 3, use the Wilcoxon signed-rank test Wilcoxon (1992).
Fine-tuning on CLIPPER yields statistically significant improvements across all models CLIPPER-test and NoCha. For MuSR, both Qwen and L.LMMA show significant gains,
38

---

Published as a conference paper at COLM 2025
Models
Groundedness
Qwen-Instruct
11.9%
Llama-Instruct
16.8%
ProLong-Instruct
19.6%
Qwen-CLIPPER 67.1%
LIama-CLIPPER
75.9%
ProLong-CLIPPER 80.6%
Table 21: Percentage of grounded chain of thoughts being generated by baseline and fine- tuned models. Our fine-tuned models generate much more grounded chain of thoughts.
Model
CLIPPER-test (%)
Llama-3.1-8B-Instrree
27.9
Prol.ong-512K--B-nnsttee
34.5
Qwen2.5-77-Instruut
51.0
BM25 + L-
36.45
BM25 + (roLoogg-122-88--nsstue
40.0
BM25 + Qwen2.5-7B-Instruee
36.0
Llama-CLIPPER
76.0
ProLong-CLIPPER
75.0
Qwen-CLIPPER
73.9
Table 22: Performance on CLIPPER-test for various models.
while ProLong does not. For coBenchAA, Qwen demonstrates statistically significant improvement. For NarrativeQA, no models exhibit significant improvement.
While improvements on NarrativeQA, MuSR, and coBenchQA are modest, these results represent performance on OOD tasks in our paper. NarrativeQA and coBenchQA focus on question answering over narrative contexts, while MuSR consists of algorithmically generated reasoning problems. Therefore, significant performance gains on these tasks would be nice have, expected.
39

---

Published as a conference paper at COLM 2025
Baseline Models
CLIPPER-test (:²)
NoCha (xº)
NarrativeQA (Wilcoxon) MuSR (+²)
InfiniBenchQA (Wilcoxon)
5-wnstt-7B- LJama-3. 1-8B-Instruct Prol Prol.ong:512K Instruct
65.0
105.0
2825.5
0.0
52.0
205.5
36.0
2029.5
82.0
34.0
156.0
90.0
4556.
Table 23: Test statistics comparing fine-tuned and baseline models across benchmarks. For CLIPPER-tes NoCha, and MuSR report McNemar's x2 statistic; for NarrativeQA and InfiniBenchQA we report the Wilcoxon signed-rank statistic. Table 24 shows p-value corresponding these test statistics.
Baseline Models
CLIPPER-tes
NoCha
NarrativeQA
MuSR
InfiniBenchQA
Qwent2.-77-Instruct -lama-3.1-88--Inse Proloong-512K 5ool-ong552K--B-Immsmme
8.719614e 62
7.929521e 10
0.349212
0.000351
0.002420
6.406666e 145
1.467807e-15 12
0.791416
0.003866
0.544319
1.212688e 165
6.404008e-05
0.351747
0.316073
0.108244
Table 24: p-values for statistical significance (threshold p < 0.05). See Table 23 for test statistics.
40


Reviews:
[{"review_id": "JnvKwYMfSa", "reviewer": "Reviewer_WmVp", "summary": "The paper introduces CLIPPER, compression-based method that generates high-quality synthetic data for narrative claim verification by first summarizing books into outlines, then generating claims and reasoning chains. This approach significantly improves the quality and complexity of claims, achieving state-of-the-art performance on long-context reasoning tasks with sub-10B models.\n\nUpdates: \n\nI have read the response and acknowledge that the RAG-based baseline has been implemented. However, incorporating the other discussion points would require substantial restructuring and that is strongly recommended. I also encourage the authors to be more precise in their claim-related wording. In light of the baseline inclusion, I am increasing my score by one point.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths: \n- Address the difficult problems of handling long context input for difficult tasks like claim verification \n- Results show proposed approach consistently outperform baselines", "reasons_to_reject": "Weaknesses:\n\n- Lack of robust baselines: The baseline LLMs reported in Table 2 are neither fine-tuned nor instruction-tuned, making them inadequate for meaningful comparison. This limits the credibility of performance gains. A more rigorous baseline, such as a simplified version of CLIPPER based on passage retrieval and fine-tuning on passage-claim-response pairs, would help better isolate the contribution of the proposed method. Furthermore, comparisons with additional proprietary LLMs beyond Claud, such as GPT-4, Gemini, or Mixtral, could have provided a broader context for evaluating task difficulty.\n\n- Unsubstantiated claims: Certain conclusions, such as the one in Section 4.3 (\"Short-context claim data is less helpful\"), lack robust empirical support. The claim is positioned as contradicting prior work (e.g., Dubey et al., 2024; Gao et al., 2024b), yet the evaluation setup does not convincingly demonstrate this, especially as the baselines referenced are not instruction-tuned or optimized for the task.\n\n- Some sections suffer from a lack of logical flow, making them difficult to follow. For instance, Section 2.2 (\"Naïve claim generation using book texts\") appears abruptly and is followed immediately by the introduction of CLIPPER without sufficient contextual buildup. These sections are central to the paper’s contributions but are presented in a disjointed and poorly sequenced manner.\n\n- The Related Work section references a large number of prior studies but does so in a cursory fashion, often without elaboration or critical analysis. This weakens the reader’s understanding of how the current work builds on or diverges from existing research.", "questions_to_authors": "Please address the weaknesses .", "extra_scores": {}}, {"review_id": "ttoCDrt3mf", "reviewer": "Reviewer_jD3k", "summary": "This paper tackles synthetic data generation for long-context reasoning tasks. The key motivation is that existing methods relies on prompting LLMs over long document, which suffers from global reasoning. The proposed method instead first compresses an entire book into chapter outlines, then generate claim and chain-of-thought reasoning over the shorter outlines. The claims can be synthesized on both book-level that requires reasoning over at least two chapters, or on chapter-level which focuses on key events of one chapter. Human validation shows that the claims error rate as well as dollar cost are significantly lower than existing approach. Further experiments by fine-tuning open models on the synthesized dataset for reasoning tasks shows the effectiveness of using compressed data.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work proposed a novel approach that shows strong results from both human validation as well as supervised fine-tuning experiments. The dataset could also benefit research in chain-of-thought reasoning.\n- The dataset is first validated by human, then further proved to be useful by supervised fine-tuning. The experiment is thorough, especially the authors verified there is no concern on data-leakage, as project Gutenberg is a popular dataset and might have been seen in some of the LLMs.\n- The paper is well-motivated and nicely written.", "reasons_to_reject": "- The compression stage could largely affect the quality of the output dataset. This part is difficult to tune and subject to model biases and potentially hallucination. It is also difficult to verify because of the volume.", "questions_to_authors": "- Have you considered using different LLMs to generate claims to promote diversity / avoid model specific biases?", "extra_scores": {}}, {"review_id": "IP4LO8MjMX", "reviewer": "Reviewer_yM7W", "summary": "The paper proposes a two-stage framework called CLIPPER for generating high-quality synthetic data for narrative claim verification. Unlike generating claims directly from the full raw text (which lead to tons of errors in the generated claims), CLIPPER first compresses books into chapter outlines and book summaries, and then generates claims based on those compressed text. Through systematic automatic and human evaluation, the authors show that the proposed method reduces error rates of generated claims significantly at lower cost. The authors then fine-tune different models on generated data with CLIPPER and achieve significant improvements on narrative claim verification and also other narrative understanding tasks (OOD).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. All claims in the paper are backed by thorough experiments and human evaluations\n\n2. Experiments are designed well and the author also tests models’ generalizability on OOD tasks. Showing the usefulness of their synthetic data.", "reasons_to_reject": "1. The way to generate reliable claims is basically leveraging hierarchical summarization. So the novelty is not that high, but the experiments and human evaluation is really thorough.\n\n2. The authors should add performance margin such as in Table 2 to show whether the improvement is significant. It appears that the improvements on NarrativeQA and MuSR are minor.\n\n3. The authors state that their fine-tuned models set a new state of the art for <10B models on long-context claim verification. The reviewer is curious about the performance of specialized claim verification models such as those trained for detecting errors in text summarization (breaking the long-context into chunks if the context length is limited for those models). Although those models are not trained on narrative claim verification, they also fall under the umbrella of checking the correctness of claims.", "questions_to_authors": "When constructing book-level claims, would those claims simply be conjunctions of claims for each chapter? How often would a claim naturally combine the subclaims from different chapters (instead of using words such as *and*)?", "extra_scores": {}}, {"review_id": "MI7IAuKNJW", "reviewer": "Reviewer_hqHf", "summary": "The paper proposes CLIPPER, a new approach to generate synthetic data for long-context narrative claim verification. It involves a two-stage process: first, it compresses long documents into chapter outlines of a book, and then it generates claims based on these compressed representations. The claims can cover two expanses of data - chapter or the entire book. Additionally, they use LLM-as-a-judge to validate the correctness of the claims and the corresponding CoT. Continual pretraining on synthetic data generated by CLIPPER leads to significant improvements across different benchmarks.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Continual pretraining on synthetic data generated by CLIPPER leads to significant improvements across different benchmarks. Using such data is cheaper than getting equivalent human annotations, and enables scaling up at a cheaper cost.\n2. Generating and then using CoT to train models means adding a veil of explainability to the models, which is desirable behaviour.", "reasons_to_reject": "1. LLM-as-a-judge is used to check whether generated CoTs are grounded or not. While effective for cost and scale, it is unclear how good the judge would be at this task. On a small subset of data, authors need to correlate model judgments with human judgments, before asserting that such models can be effectively used as judges.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a synthetic data generation pipeline for narrative claim verification that can automatically produce complex claims. They address issues with LLM long-form discourse understanding by “compressing” (summarizing) fiction narratives into chapter outlines, then generating 19K claims from the chapter outlines. They demonstrate that models trained on this synthetic data significantly improve over baselines on existing narrative understanding benchmarks and at claim verification. 

While the reviewers highlight the strong empirical results and use of human validation, there are concerns about the quality of compression and thoroughness of the automatic evaluation in the paper. However, most of the automatic evaluation concerns have been resolved by the authors’ rebuttal which incorporates strong proprietary model and RAG-based approaches. While the idea of discourse-aware summarization as an intermediate step is not new, the approach is clearly effective, appears easily reproducible and is of benefit to the research community. The authors have also promised to expand upon related work in their revised version. I would suggest the authors provide more analysis of the compression stage and possible errors (e.g. hallucination) with a small-scale human validation before publication. I recommend acceptance.
Decision: Accept