Learning to Reason for Long-Form Story Generation
Authors: Alexander Gurung
Mirella Lapata
Venue: COLM 2025 (2025)
Keywords: story generation
reasoning
reinforcement learning
long-context generation
TL;DR: We propose a long-form story generation task, Next-Chapter Prediction, and a novel reward formulation that allows us to learn reasoning traces which improve predicted chapters without a labeled dataset.

Abstract: Generating high-quality stories spanning thousands of tokens requires competency across a variety of skills, from tracking plot and character arcs to keeping a consistent and engaging style. Due to the difficulty of sourcing labeled datasets and precise quality measurements, most work using large language models (LLMs) for long-form story generation uses combinations of hand-designed prompting techniques to elicit author-like behavior. This is a manual process that is highly dependent on the specific story-generation task. Motivated by the recent success of applying RL with Verifiable Rewards to domains like math and coding, we propose a general story-generation task (Next-Chapter Prediction) and a reward formulation (Verified Rewards via Completion Likelihood Improvement) that allows us to use an unlabeled book dataset as a learning signal for reasoning. We learn to reason over a story's condensed information and generate a detailed plan for the next chapter. Our reasoning is evaluated via the chapters it helps a story-generator create, and compared against non-trained and supervised finetuning (SFT) baselines. Pairwise human judgments reveal the chapters our learned reasoning produces are preferred across almost all metrics, and the effect is more pronounced in Scifi and Fantasy genres.

Published as a conference paper at COLM 2025
Learning to Reason for Long-Form Story Generation
Alexander Gurung, Mirella Lapata School of Informatics
University of Edinburgh
Edinburgh, UK a. a.gurung-1ee 1@sms, ed.ac.uk, mlap@inf .ed. ac. uk
Abstract
Generating high-quality stories spanning thousands of tokens requires competency across variety of skills, from tracking plot and character ares to keeping a consistent and engaging style. Due to the difficulty of sourcing labeled datasets and precise quality measurements, most work using large language models (L.Mss) for long-form story generation resorts to combinations hand-designed prompting techniques to elicit author- like behavior. This is manual process that is highly dependent specific story-generation task. Motivated by the recent success of applying RL with Verifiable Rewards to domains like math and coding, propose a general story-generation task (Next-Chapter Prediction) and a reward formulation (Verifiable Rewards via Completion Likelihood Improvement) that allows us to use an unlabeled book dataset as a learning signal for reasoning. We learn reason over a story's ' condensed information and generate a detailed plan for the next chapter. Our reasoning is evaluated via the chapters it helps story generator create, and compared against non-trained and supervised fine-tuning (SFT) baselines. Pairwise human judgments reveal the chapters our learned reasoning produces are preferred across almost all metrics, and the effect is more pronounced in Sci-Fi and Fantasy genres.
the
1 Introduction
Long-form story generation is difficult modeling task that requires synthesizing thousands of tokens of rich and subtle text into coherent and character-aware narratives. High-quality writing balances interesting characters with engaging world-building and satisfying plot arcs (Jarvis, 2014; Kyle, 2016), posing problems of long-term dependencies and accurate Theory-of-Mine modeling Although Large Language Models (LLMs)) have shown recent promise in writing long texts (Yang et al., 2022; 2023; Bai et al., 2025; Xie et al., 2023; Shao et al., 2024a), their stories still struggle on a variety of criteria like originality, plot, character development, and pacing (Chakrabarty et al., 2024; Wang et al., 2023; 2022; Ismayilzada et al., 2024), in addition to more fundamental long-form generation flaws like repetition and quality degradation (Que et al., 2024; Wu et al., 2024).
Outside of long-form story generation, Reinforcement Learning (RL) has been successfully applied to post-training L.MSs across various tasks, from optimizing general human prefer- ences to improving performance in specialized domains like code generation and math (Li et al., 2025; Lambert et al., 2024; Kumar al., 2025). Although some RL work briefly touches on creative writing, these efforts typically focus on short-form tasks and are framed as an overall preference alignment (Nguyen et al., 2024; Zhao et al., 2024). In general, RL methods for LLMs have necessitated at least one of the following: (a) high quality datasets labeled with preferences, rewards, or verified reasoning traces or (b) high-quality reward models, often in the form of verifiable reward functions. Large-scale datasets have yielded strong results for broader tasks like peaaaan-reeeeeen--lignnnee
(Rafailov al., 2023; Ethayarajh et al., 2024;
We release reproduction and training code at github.com/ SalooommAlee-uuuanne Alex-Guruxg//////ee Rea.onngg//aS..nCC.......

---

Published as a conference paper at COLM 2025
Story Information
Policy Model
Ou
Reference Model
VR-CU
Rassoring
A
Dutaset FC. yo B
Genorator n²
VR-CLI
Deteried Plan
Group
[(rl4,4) F R(F)
Computation
Key
Outputoo
AG
Peiley Xa60
Reasoning Traest Answer
From Datanet
Trained Model
du
(a) NCP GRPO Training
(b) VR-CLI Reward
Figure 1: Next-Chapter Prediction (NCP) training procedure (a) using our VR-CLI reward paradigm (b), with GRPO (Shao al., 2024b). Our reward uses reference model to get the improved likelihood of the true next chapter. The reference model is a copy of the policy model frozen at the start of training, used both as our story generator π and for computing KI-divergence. The policy model is our reasoning model n trained to produce detailed plans of the next chapter. VR-CLI is described in Section 6 and training in Section 5.
Cui et al., 2024; Wang et al., 2025a), while recent efforts in the math and coding domains have achieved significant success through the use of high-quality rewards (Gehring et al., 2025; DeepSeek-Al et al., 2025; Shao et al., 2024b; Lambert et al., 2024; Kimi Team et al., 2025; Li et al., 2025; Kumar et al., 2025). In particular, there has been recent surge of interest in enhancing the reasoning capabilities of LLMs using verifiable rewards (Lambert et al., 2024). Instead of relying on learned reward model, recent work uses deterministic verification functions - usually rule-based methods that assess answers for correctness and formatting.
Neither approach is practical for long-form story generation: story 'correctness' is ill-defined, and collecting large datasets of labeled story completions is challenging, The multifaceted and nuanced nature of narratives also makes the formulation of singular reward functions difficult. Instead, the 'quality' of stories or their continuation is usually assessed through pairwise comparisons acToss multiple metrics (Yang et al., 2022; Huot et al., 2025; Yang et al., 2023; Xie & Riedl, 2024). Most current work on story generation has therefore avoided RL, and instead leaned into desiddd systems that mimic different parts of the human writing process, like drafting, editing, and planning. (Fan et al., 2018; Zhou et al., 2023; Wang et al., 2023; Xie Riedl, 2024; Wang et al., 2025b). Huot et al. (2025) train LLMs to model human story preferences across metrics like creativity and language use, but they do not train story generation models using these evaluators.
Rather than predicting and evaluating entire book-length generations, we propose Next- preceettiii Chapter Prediction (NCP) as a more tractable and informative task. Inspired by the human book-writing process (Kyle, 2016; Jarvis, 2014) that uses both high-level sketch of the story and more tinenerieeed information about characters and plot progression, we model story generation predicting the next chapter given similar collection of story information. Fine- tuning directly on this task, quickly leads to overfitting and fails to capture the underlying reasoning behind the story-writing process. Instead, we propose novel reasoning paradigm akin to Reinforcement Learning from Verifiable Rewards (RLVR; Lambert et al. 2024) that allows RL training of reasoning traces for story generation.
We introduce Verifiable Rewards via Completion Likelihood Improvement (VR-CLI), reward modeling paradigm designed to learn reasoning traces that enhance generator's 's ability to reproduce given dataset (see Figure 1). Our key assumption is that increasing the generator's 's likelihood of predicting the next chapter will, in turn, improve the quality of its generations. Accordingly, our reward is defined with respect to the improvement in predicting a gold next chapter, which can be naturally expressed in terms of per-token perplexity. We use VR-CLI to train Qwen 2.5 models (Qwen et al., 2024) to produce traces that cite and reason about the given story information before constructing detailed plan, which is then fed context to the base model for generating the chapter continuation. Our
2

---

Published as a conference paper at COLM 2025
results show that these models outperform reasoning and non-reasoning baselines and models trained via SFT on next-chapter prediction. Our contributions are as follows:
We introduce Next-Chapter Prediction, a new task for long-form creative writing.
We propose VR-CLI, a proxy reward formulation for reasoning that relies only on a high-quality dataset to mimic completions.
We show that training using this objective improves the generated next-chapters, as judged in pairwise human evaluations.
2
Related Work
Story Generation Story generation has long been studied as a completion task (Mostafazadeh et al., 2016; Fan et al., 2018), but only with recent advances in long-context modeling have language models begun producing stories of considerable length. A large body of previous work has shown that incorporating condensed story information, such as plot outlines (Fan et al., 2018; Zhou et al., 2023; Wang et al., 2023; Xie & Riedl, 2024; Wen et al., 2023; Yoo & Cheong, 2024), as well as setting and character details (Yang et al., 2022; 2023), significantly improves generation quality. Further improvements have been achieved by explicitly breaking down the writing process into sub-tasks handled by different 'agents' or models (Huot et al., 2025; Peng et al., 2022). However, these approaches largely rely hand-crafted prompting techniques to generate and refine both the condensed story information and the story itself. Progress in long-form story generation has been hindered by challenges in evaluation, which is inherently subjective, nuanced, and time-consuming. Human evaluation has become the de facto standard for assessing machine-generated sto- ries, typically through pairwise judgments across key dimensions such as coherence, plot development, creativity, and characterization (Yang et al., 2022; 2023; Huot et al., 2025; Xie & Riedl, 2024; Chhun al., 2022).
Reinforcement Learning for LLM Reasoning Fine-tuning LLMs with Reinforcement Learning (RL) has become an increasingly popular method for improving performance on variety of tasks. Traditionally, RL has been used align model generations with human preferences (RLHF; Grattafiori et al. 2024; Achiam et al. 2023; Ziegler et al. 2019; Ouyang et al. 2022; Stiennon et al. 2020; Christiano et al. 2017). Building on these online methods, recent work has introduced algorithms that can also learn policies from static preference datasets (Rafailov et al., 2023; Ethayarajh et al., 2024).
RL techniques have further led to significant advancements in verifiable and executable domains such as math, science, and coding (Gehring et al., 2025; Simonds & Yoshiyama, 2025; Li et al., 2025; Kumar et al., 2025). Several recent large-model releases have also emphasized the use of RL to improve performance on math and programming benchmarks (DeepSeek-Al et al., 2025; Kimi Team et al., 2025). Many of these approaches build upon the Reinforcement Learning from Verifiable Rewards (RLVR) paradigm introduced in Lambert et al. (2024), which trains LLMs to produce useful reasoning traces for tasks with easily verified answers via a binary or scaled reward.
We not aware of previous work that learns reasoning traces for creative long-form gener- ation, possibly due to the challenge of defining rewards with objective correctness criteria. However most similar in approach, Hu et al. (2024a) use LLM-based GFLowNets (Bengio et al., 2021) for sentence continuation and 5-sentence-story infilling using the ROCStories dataset (Mostafazadeh et al., 2016). By representing many LLM tasks as latent variable modeling problems, they connect RL approaches to the field of probabilistic inference.
Concurrent work has also begun exploring the idea of using a model's likelihood of the solution to optimize reasoning. JEPO (Tang et al., 2025), VeriFree (Zhou et al., 2025), LATRO (Chen et al., 2024) and INTUITOR (Zhao et al., 2025) all propose similar policy optimizers based on this latent-variable perspective, and largely apply their methods to math tasks. Our VR-CLI formulation differs in its reward shaping, lack of SFT-objective, and its application to long-form creative writing.
3

---

Published as a conference paper at COLM 2025
Story Information
Sample Sentence
Global Story Sketch
As the story progresses, it becomes clear that Barbara's family is complex and troubled.
Previous Story Summary
As the campers settle into their routine, Tracy becomes friends with Barbara, who arrives at the camp on foot (Chapter 7).
Character Sheets
She wants to be good mother to Bear and seeks validation through her role as mother.
Previous Chapter
The implication, Alice understood, was that to have more than one boy would complicate matters when it came to passing on the bank. Peter tells Carl that his son is missing.
Next Chapter Synopsis Next Chapter
And when a voice came through the wires, it wasn't 't a member of the staff, but Peter Van Laar himself- whom Carl nodded each time they crossed paths at the Preserve, but to whom he had actually spoken maybe twice in his life.
Table 1: Sample sentences from each Story- Inoyrmanoormatio element; data for The God of the Woods (by Liz Moore), Chapter 15. Note the different style of text (e.g., detailed prose vs high-level descriptions) and different focus (e.g., character-focused vs plot-focused).
3 Next-Chapter Prediction
Instead of generating and evaluating entire book-length stories at once, we propose Next- Chapter Prediction (NCP) more tractable task. We draw inspiration from the methods real authors adopt when writing a book (Kyle, 2016; Jarvis, 2014) which may involve creating a high-level global story outline, along with more detailed plans for individual chapters.
We define the 'writing process' as following this general structure: first, an author creates high-level sketch of the story. As the writing progresses the author refines their plot and character development. Before writing each chapter, the author prepares a short synopsis of what should occur. Based on this writing process, we formalize our NCP task as follows.
Let SI; collectively denote Story-Informatee at chapter index i. Specifically, SI, represents:
A global sketch of the entire story (see Table 1);
A summary of the previously written chapters (see Table 1);
Character sheets, based on previously written chapters (see Table 1);
Previous story text, which we approximate by the previous chapter (see Table 1).
Synopsis of the next chapter (see Table 1);
At a chapter index i with Story-Information SI, we denote the predicted next chapter as ê++ When the next chapter given, not predicted, we denote it C1+1- We denote the story-generator model that predicts the next chapter nTς. The default setup for this task is to directly generate the next chapter from the story information, using the story generator:
c1+ <- T (SI)
(1)
Our claim in this work is that reasoning traces can lead to better next-chapter prediction; specifically, we use reasoning model πT to predict reasoning trace about the given story information, ending with a more detailed plan for the next chapter. We denote this predicted plan p. In reasoning model variants, we generate the next chapter using dkory-generatee model conditioned on both the story information and the plan:
p T R (SL)
G < 7o (S4 p)
We evaluate the effectiveness of story-generation and reasoning models via the quality of the next chapter, judged both on its merit and its ability to into the broader story context. Note that our proposed reasoning method optimizes the detailed plans, which are then used to improve the generated chapters. More details are provided in Section 7, 7.2, and example reasoning traces in Appendix G.
4

---

Published as a conference paper at COLM 2025
4
Dataset Curation
We collect dataset of 30 books published in or after 2024 (to mitigate information leakage). Details on book selection are in Appendix A.1. Our books range from 67k to 214k tokens, with mean of 139k. Rather than raw story text we operate representations as prior research has shown these to be useful for story tasks (Pham et al., 2025; Gurung & Lapata, 2024). We follow previous work in sourcing gold-standard chapter summaries from SuperSummary' (Xu et al., 2024; Yuan et al., 2024; Ou Lapata, 2025). We explain below how we use these in building the story-information SI₁.
higher-level plot and character
4.1 Story Information
We denote the following story information Sl, defined at given story and chapter index i
High-level Story Sketch This can be thought of as an author's 's sketch of how the story should unfold. As we do not have these sketches available, we mimic them by summarizing all of a book's s individual chapter summaries together into a high-level plan (see Table 1). We use Llama 3.3 70B (Grattafiori et al., 2024) to generate these summaries.
Previous Story Summary At a given chapter index i, we also summarize the previous chap- summaries < i to give a more detailed representation of what has already been written (see Table 1). Again we use Llama 3.3 70B. Hyperparameter details are in Appendix A.2.
Character Sheets Prior work has shown that that character information is useful for down- stream story tasks (Gurung & Lapata, 2024; Huot et al., 2025), so we incorporate character information into S1 by adapting the CHIRON character sheets for longer stories (Gurung & Lapata, 2024). These sheets contain four broad categories of character information and are automatically generated from story text before being filtered for accuracy using an entailment module. We create individual character sheets for the three main characters of each story, defined at each chapter index i based on the previous chapters < i. We use the Llama 3.3 70B model for both generation and entailment modules and add summarization step to consolidate the sheets. More details are in Appendix A.3.
Next Chapter Synopsis For given chapter index i, we call the summary of the next chapter a 'synopsis'. This contains a rough idea of what should happen in the chapter, which informs the path the story generations take. On average, the synopsis is 7.4% of the size of the next chapter (in tokens). We hypothesiize that by adding detail to the synopsis via reasoning (see Table 1), we can more effectively guide the generation of the chapter.
4.2 Dataset Statistics
The final dataset has 30 books, split 22-4-4 into training, validation, and testing. We split by book to evaluate our method's generalization capabilities across books and authors, as training may learn book or author-specific style or plot information. Further splitting into chapters and lightly filtering for chapter length (200- < words < 5, 000) and chapter location (2 < i< < |S| 2) gives us 1,004 training datapoints, 162 for validation, and 181 for testing. More details are provided in Appendix A.4 and Tables 6, and Example sentences from each element in our dataset are in Table 1.
5 Verifiable Rewards via Completion Likelihood Improvement (VR-CLI)
As explained earlier, it is challenging to define reward model for long-form creative writing in part due to the length of generations and the subtle nature of the task. Furthermore, collecting labeled datasets of story continuations is costly due to the sheer variety of possible
2Qur work primarily uses two models: Llama 3.3 70B and Qwen 2.5 3B/B-1MM Instruct. The Llama model has a knowledge cutoff of December 2023, and while the Qwen models were released in 2024, we find little evidence of data contamination. h /ssuppprrrummrryccoo//
5

---

Published as a conference paper at COLM 2025
story continuations. We propose circumventing these constraints by creating a proxy reward that incentivizes useful reasoning steps, utilizing the book dataset described in Section 4.
Taking inspiration from RLVR (Lambert et al., 2024) and the latent-variable modeling framing of Hu et al. (2024a), we introduce Verifiable Rewards via Completion Likelihood Improvement (VR-CLD. VR-CLI is a reward modeling paradigm that learns reasoning traces that help generator reproduce a given dataset without the need for labeled data or defined external rewards. This formulation relies on a key assumption: improving the generator's 's likelihood of producing the provided dataset will, in turn, improve the quality of its generations. We verify this assumption for NCP in Section We believe VR-CLI is promising reward paradigm for other tasks where completions are not easily verified.
Let represent a dataset of pairs (x, where x is the input prompt, and y is the gold completion. A reasoning model generates reasoning trace I which culminates in a final answer a. We define a generator model, T that is not trained but instead used to get the likelihood of generating y. For our NCP task, x is the story information, y is the gold next chapter, a is the detailed plan p, and our generator 9 is the story generator model.
We define the improvement I as the percent improvement in per-token perplexity (PPL) when generating y from 7e, conditioned on (x,a), compared the perplexity of y conditioned only on x. This measure indicates how much the inclusion of this answer increases the likelihood of generating y. Note that the perplexity is calculated from the probability distribution over tokens in y,, not the probability of the entire text. Additionally, the prompt formulation can differ between y|x and y|x,, a as necessary for the specific subtask.
I (,) PPL (u|x) PPL, (ylx,a) x 100 = [1
PPL (v)x, PPI (y(x)
x 100 (4)
P (y)x))
Positive values imply the perplexity with reasoning is lower than the perplexity without, and the reasoning improved the likelihood of generating the gold continuation. Negative values imply the opposite, that the reasoning worsened the likelihood. Note that PPL (y/x) does not depend on the policy model or answers, so it can be pre-computed before training. Although we use perplexity as our likelihood measure in this work, a very similar formula- tion could be constructed with average log-likelihood instead. We found both methods to produce similar results; more details are presented in Appendix A.8.
There are many ways to define our reward based on the improvement I, for example: (1) as piecewise function, like RLVR, subject to the magnitude of the improvement and custom thresholds (Fwo, (),)), or (2) as a raw value bounded by when necessary.
I(x, y, S wo
R(x,y,a)
B wo < 1(x,y,a) S w1 (5)
R(x,y,a) = (x,
(6)
w1 < 1(x,y,a)
The specific reward formulation will depend on both the downstream task and the RL- training algorithm used. Figure 1b illustrates the high-level reward paradigm. The reward and training algorithm we use for our NCP task are described in the following sections.
Additionally, one could forgo the baseline and use the chosen likelihood-measue directly (e.g. Lng (x, a)) = - PPL(y) x, a)). Notably, when using GRPO's advantage estimation and simple reward equal to improvement (R(x, y,, a) (,)),, advantages are equivalent with and without the baseline (Shao et al., 2024b). However, we hypothesize that including a baseline provides a more interpretable and useful reward estimate, especially when using a more complicated reward definition. For example, across a group where all samples produce worse-than-basde
perplexities, it may be useful to set their rewards uniformly to zero (and not increase the likelihood of potentially damaging traces). We also test this in ablations in Appendix A.8 and found comparable results, and believe both formulations are worth exploring in future tasks. As all of these formulations retain the goal of increasing the cold-completion's 's likelihood via the reward, we view them as variants of the VR-CLI paradigm.
6

---

Published as a conference paper at COLM 2025
6 Reinforcement Learning for Next-Chapter Prediction
Reward Modeling for Next-Chapter Prediction We use the thresholded VR-CLI reward paradigm introduced in Equation (5) to define our reward for each (story-information, chapter, reasoning) tuple ($I,, C 1, p). Note that our 'answer' is the detailed plan & our policy model produces after reasoning over the story information. Example reasoning traces are in Appendix G. We use the reference model for our policy as our story-generator, π9, as we assume that the skills needed for story-reasonin those needed for engaging story-writing. For other tasks where the reasoning and answering skills are not so distinct it may make sense to use the policy model as the generator as well.
and summarizing are different than
Applying VR-CLI to our task implies that plans that induce higher likelihood of the true next chapter will also produce better generated chapters. We validate this assumption in Section 8, but also believe it aligns with our intuition concerning useful plans. For example, plans that correctly predict plot events, character details, or writing style in the true next chapter should increase its likelihood, and should also be a useful basis for generating a novel chapter. We provide example aligned excerpts between plans and next-chapters in Table 19.
We define our thresholded reward as:
0, I(S,; ((1,,)911P)) < 0.05
0.5, 0.05 S I(SI, C+1,p) <1
((1,))
0.9,
if G- <2
1, if (S(I,, 2
We find that this mild reward shaping helps ensure consistent training trajectories by encour- aging both 1) large perplexity improvements and 2) promising samples at the early stages of training when significant improvements in perplexity are rare. As we load the refer- ence model for calculating the Ki-divergence,
reward requires minimal computational overhead. Future work could explore lightweight reasoning for strong story generators (or vice-versa), but current setup significantly reduces training complexity.
GRPO Training Our task and reward setup relatively RI-algorithm agnostic; one could use an offline reasoning-reward dataset or any online policy learning algorithm that works with verifiable reward. We hypothesize that online methods are preferable for NCP as reasoning styles change over training, and preliminary tests show low reward prior to training: initially, average improvement is -0.06% with Qwen 2B.
We use GRPO for RL training as it has seen recent success in verifiable reward domains (Shao et al., 2024b; DeepSeek-At et al., 2025) and reduces computational overhead by removing the value model. GRPO generates multiple responses for each prompt and normalizes each reward by its group, before calculating the advantages and updating the policy parameters See Figure la for training overview and Appendix A.6 for more details.
Experimental Setup
7.1 Model Comparisons
Prior work has found success using the Qwen model series (Qwen et al., 2024), and Gandhi et al. (2025) showed they exhibit cognitive behaviours useful for reasoning. The majority of our experiments use Qwen-2.5 7B-Instruct-IM as our base model for training and story generation. We also report results with 3B, to examine trends across model sizes. Future work could further scale these experiments, but due to the long context size, large-model training is computationally expensive. For clarity, models Ni with noted parameters f are trained, and models π without noted parameters are frozen without further training. More training details are in Appendix A.6. We compare the following model variants.
7

---

Published as a conference paper at COLM 2025
B BR RL
Strength
Creativity
Develop
Language
Creativity
Develop
Language
(a) Bradley Terry Relative Strength
(b) 3B Bradley Terry Relative Strength
Figure 2: Bradley Terry relative strength parameters for each variant, by model size. RL- Trained (RL) has the highest relative strength in most dimensions, and the effect is more pronounced in 7B models. Table 2 shows win-probabilities for 7B models. Variant short- hands are introduced in Section 7.
RL-Trained (RL) Our proposed method, training policy model with VR-CLI and GRPO. Our story-generator not further trained, but our policy model is trained to produce useful plans via our VR-CLI reward (Section 6): f - n (SL); & f π (SL, p).
Base (B) We compare RL-Trained against a baseline model which does not train our story- generator produce any reasoning. This is our default NCP variant: è; 7 (S1:).
Base-Reasoning (BR) We also compare to a model which attempts to predict the next chapter by first generating reasoning trace; do not train our story-generator or our reasoning model (they are both set to the same): + π (SI:); è; + x (SI, p).
Supervised Fine- Tuning (SFT) We fine-tune our story-generator on the NCP task with the SFT objective next chapters. The generator predicts chapters like Base: ni (S1,).
7.2 Human Evaluation
We evaluate the generated story continuations by eliciting pairwise preferences following evidence suggesting that relative judgments can be more reliable than absolute ones (Lou- viere et al., 2015; Stewart et al., 2005; Liu et al., 2024). We draw on the criteria outlined in previous work (Huot et al., 2025; Chakrabarty et al., 2024; Chhun et al., 2022) to evaluate the continuations along the following dimensions: (1) Plot: Does it exhibit events and turns that move the plot forward logically? (2) Creativity: Does the continuation have engag- ing characters, themes, and imagery, and avoid overly cliched characters and storylines? (3) Development: Does introduce characters and settings with appropriate levels of detail and complexity? (4) Language Use: Is the language varied and rich, exhibiting rhetorical, linguistic, and literary devices? (5) Characters: Does it feature believable and conceptually consistent characters, including reasonable character arcs and development? (6) Overall Preference: Which of the two continuations did you prefer? Full instructions and recruitment details are in Appendix C. Annotators were recruited through Prolific their primary language was English and their employment role was in creative writing.
To validate our annotation procedure, we compute Fleiss' kappa across dataset of Base vs. Base- Aase-Reasoning comparisons. We find fair agreement across annotation dimensions, with the highest agreement Creativity and Overall Quality. More details are in Appendix D. Initial experiments showed colddcontinnutions were almost exclusively preferred to model generations, so we exclusively collect inter-variant comparisons.
We randomly sample 5 chapter indices per story in our test set and generate continuations for each model. Participants are shown the story information and two possible continuations and asked to provide preferences along the above dimensions. We collect one judgment per datapoint (20 datapoints per variant-coe arrian--omppriisom) and convert the preferences into relative model strengths using Bradley-Terry model (Bradley & Terry, 1952).
8

---

Published as a conference paper at COLM 2025
Dimension
SFT-B BR-B BR-SFT RL-SFT RL-B RL-BR
Genre
BR
RL
Diff
Plot
5.3
72.2
83.3
89.5
52.9
62.5
Character
26.3
68.8
84.2
89.5
46.7
61.5
Sci-Fi
0.68
1.68
1.00
Creativity
21.1
57.1
82.4
84.2
81.2
70.6
Fantasy
0.50
1.11
0.61
Development
22.2
68.8
88.2
89.5
52.9
66.7
Romance
0.27
0.91
0.64
Language
35.0
66.7
76.5
88.9
80.0
58.8
Historical
-0.33
0.60
0.93
Overall 15.0
66.7
83.3
90.0
76.5
64.7
Table Mean percent improvement by genre our test set. 'Diff' is
Table 2:
Bradley-Terry preference probabilities (%)
that A is preferred over B in A-B pairwise compar- isons. RL-TTainned is preferred across almost all di- mensions, and against Base has overall 76.5% pref- erence probability. See Section 7 for variant details.
the additional percent improvement gained by RL-training our reasoning. Model shorthands are in Section 7.
Model
# Words
Unique Words
Unseen Trigrams
Rouge-L F1
Rouge-L Prec
Base
1486.5
32.7%
77.8%
0.10
0.286
SFT
1278.1
23.7%
53.8%
0.09
0.308
Base-Reasoning
1568.0
31.8%
77.5%
0.11
0.282
RL-Trained
1479.5
33.2%
77.2%
0.11
0.306
Table 4: We calculate automated measures of length and diversity from the generated chapters across 7B model outputs: Words (average number of words per chapter), % Unique words within chapter, % Unseen Trigrams (% of all trigrams in chapter not present in SI). Rouge-L. (F1 and Precision) is computed against the gold standard, reference chapters. We find broadly similar results and uniqueness (Appendix B), indicating that the preferences come from higher quality storytelling instead of longer stories.
Results
We report both Bradley-Terry based probabilities (Table 2) and relative strengths (Figure 2), as well as true win-rates, including ties (see Table 15). We also investigate surface differences in predicted chapters (Table 4) and percent-improvement by genre (Table 3).
Training reasoning improves overall performance. We find our RL-Trained 7B model overwhelmingly preferred over all three baseline variants and has the highest relative strength value for all dimensions. Figure shows the relative strength scores for each comparison model. RL-trained yields the biggest improvements in creativity and overall preference, and the lowest improvement in characters. This model has 64.7% preference probability over Base-Reasoning in Overall Quality, indicating that optimizing our reward aligns with human judgments (see Table 2).
SFT model performs poorly on next chapter prediction. We find significant repetition issues in the chapters generated from the SFT model. For a fair comparison, we automatically truncate clear mode-collapse repetitions (Appendix B), but annotators still strongly prefer all other variants. We hypothesize that SFT performance would be improved by training on much larger dataset to reduce the effect of overfitting; doing starting point for reasoning models (DeepSeek-AT et al., 2025). We therefore belíeve that our method is more effective in low-resource settings, but can be used in concert with SFT approaches when more data is available.
may also be helpful
Differences between models go beyond surface-level variations. We investigate surface- level differences between the generated chapters by comparing their length, word/ word/trigram trigram diversity. While we do find the SFT-based chapters to be significantly shorter and less lexically diverse (Table 4), find few statistical differences between the other models. This indicates that annotators were not biased towards longer more varied stories and instead judged based on story content. Table 9 and Appendix B contain more details.

---

Published as a conference paper at COLM 2025
Perplexity improvement correlates with human judgments. To evaluate VR-CLI's Im- provement's utility as a proxy metric for human judgments, we compute the correlation between the pairwise judgment and the Improvement given reasoning trace and gold next-chapter (Appendix D.1). We find significant and positive Spearman's rank correlation of (p- 0.33, p = 0.01, N 60). While this improvement metric cannot be computed at true test time (without a gold continuation), it can be a useful proxy for further evaluating reasoning models on an unlabeled dataset without collecting expensive human judgments.
Reasoning has the most pronounced effect on the Sci-Fi genre. We compare pairwise pref- erences and average percent improvement across different genres. We find that reasoning has a particularly strong positive effect on Sci-Fi, and that training further improves the effect. Table 3 shows the percent improvement for Base- Reasoning and RL-Trained variants. See Tables 17 & 18 for pairwise preferences broken down by model size and genre.
Reasoning improves bigger models more. To test these trends are exclusive 7B models, we also perform the same experiments using Qwen 2.5 3B as a base. Relative strength scores are shown in Figure 2b and preference probabilities in Table 13; our RL-Trained setting still outperforms the baselines across most dimensions, but the effect is smaller and annotators preferred Baseline- Reaiinnnn in 'Creativity' and 'Language' dimensions. We find similar genre effects (Sci-fi still has the highest percent-improvement and limited lexical differences between settings; more details are in Appendix F. Due to the greater relative strength of RL-Trained 7B variants compared to their non-trained variants, we hypothesize that larger models are better able to learn useful reasoning for this task.
Conclusion
We approach the challenge long-story generation by proposing an author-inspired story- generation task, Mei--hahptr--rediiiiin (NCP), that uses combination of high-level and locally relevant story information to predict the next chapter of a book. We evaluate performance this task via dataset of recently published books and their per-chapter summaries, from which we derive a useful dense collection of story information.
We introduce novel reward modeling framework, Verifiable Rewards via Completion Likelihood Improvement (VR-CLI), that allows us to optimize a reasoning model to predict plans that increase the likelihood of the next chapter. We train 7B and 3B Qwen models using this framework and GRPO, and show that the resulting story completions are strongly preferred by expert judges over baselines. In the future, we plan to apply our VR-CLI framework to generation tasks whose output cannot be easily verified, such as long-form summarization and question answering.
Acknowledgments
We thank the anonymous reviewers for their constructive feedback. We gratefully acknowl- edge the support of the UK Engineering and Physical Sciences Research Council (grant EP EP/W002876/ 1). We would also like to thank Kolya Malkin for their advice and support this work.
Ethics Statement
Story generation systems have the potential to perpetuate biases (e-g., stereotypical charac- ters) and create harmful content. As the training signal for our work comes from already- published books we hope our models do not learn any more harmful behaviour than is already present in their pretraining data. However, future work should explore ways to measure and mitigate the biases of LLMs in long-form story generation.
Research on text-generation systems also needs to be careful to avoid violating the intellec- tual property rights of the authors whose work is being used. As our dataset (and therefore the models trained on it) are created using copyrighted work we individually purchased,
10

---

Published as a conference paper at COLM 2025
unable to publicly release our dataset models. We will however release the code necessary to reproduce our work, and the books used are listed in Table 5.
References
OpenAT Josh Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, S. Balaji, Valerie Balcom, Paul Baltescu, Haim-ing Bao, Mo Bavarian, J. Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christo- pher Berner, Lenny Bogdonoff, Oleg Boiko, Made-laine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, An- drew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, B. Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunx- ing Dai, C. Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, L. Fedus, Niko Felix, Sim on Posada Fishman, Juston Forte, Is-abella Fulford, Leo Gao, Elie Georges, C. Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo- Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, S. Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Jo-hannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, B. Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, I. Kanitscheider, N. Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, J. Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstan- tinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, J. Leike, Jade Leung, Daniel Levy, Chak Li, Rachel Lim, Molly Lin, Stephanie Lin, Ma- teusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, A. Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, S. McKinney, C. McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, O. Murk, David M'ely, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O'Keefe, J. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack W. Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, M. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, S. Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, F. Such, Natalie Summers, I. Sutskever, Jie Tang, N. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer' on Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll L. Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, C. J. Weinmann, Akila Welihinda, P. Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Han- nah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qim-ing Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 Technical Report. March 2023. URL https: http:///wwwwwwwnneeeeeeeee semanticsshollrr.rgg//aaaa GPT-4-Techmical-Regg wewpriiaa--eeoo--cccieeeeee 1tt.r/hhiiiiaaeeeeeeeeeae3666/a0 oaa.a//55500a2a88000555555333333
Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longwriter: Unleashing 10,000+ word generation from long context
11

---

Published as a conference paper at COLM 2025
LLMs. In The Thirteenth International Conference Learning Representations, 2025. URL https://openrevaan wwwtteeuut/neeeeee (otfrrrunmmikkkksSyhh....
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS 21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393.
Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 4):3444)3 1952. ISSN 0006-3444. doi: 10.2307 /2334029. URL https://wwww jstor www.oogg//ssoo htrr/sttable/2340095 Publisher: [Oxford University Press, Biometrika Trust].
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien- Sheng Wu. Art or Artifice? Large Language Models and the False Promise of Creativity. Proceedings of the CHI Conference on Human Factors in Computing Systems, Pp. 1-34, May 2024. doi:
URL https://dd acm. org//oi/10.... 1145//.1.1111111111
3642731. Conference Name: CHI 24: CHI Conference on Human Factors in Computing Systems ISBN: 9798400703300
Place: Honolulu HI USA Publisher: ACM.
Haolin Chen, Yihao Feng, Zuxin Liu, Weiran Yao, Akshara Prabhakar, Shelby Heinecke, Ricky Ho, Phil Mui, Silvio Savarese, Caiming Xiong, and Huan Wang. Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding, 2024. URL https::/arxiv.or//bbaaaa 240.0/222...........
Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chloé Clavel. Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 5794-5836, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https://aclantholoogy org. 2rg/2022, coling- 1. 509/.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proceedings of the 31st Interna- tional Conference on Neural Information Processing Systems, NIPS'17, pp. 4302-4310, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: boosting language models with scaled ai feedback. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. ar Xiv preprint ar arXim:2502.01455. 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin,
12

---

Published as a conference paper at COLM 2025
Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DeepSeek-Ri: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. 2025. doi: 10.48550/ AR110/2211..2222121290 URL https:///axv.... wtt. 1 12948. Publisher: arXiv Version Number: 1.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In Proceedings of the 41st International Conference on Machine Learning, ICML/24. JMLR.org, 2024.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical Neural Story Generation. In Proceedings of the 56th Amnual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653 10.18653/v1/P1111000 1082.8-1082. URL htttp://aclwett org/ he///ntologgg/P/// (atpllooyyy18-11666
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Good- man. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025. URL https:://rxiiii org/ abs/2503. 01307.
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriel Synnaeve. RLEF: Grounding code LLMs in execution feedback with reinforcement learning. In Forty- second International Conference on Machine Learning, 2025. URL https: //tpnr/e///peeeiiieeee net/ forum?2dd=2255nkee1
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bob- bie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Ev Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yean Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke
13

---

Published as a conference paper at COLM 2025
de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Isimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Gird- har, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Ra- parthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Paapakioos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poultonn, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Chang- han Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, lean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lak- shya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manay Avalani, Manish Bhatt, Mar- tynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang,
14

---

Published as a conference paper at COLM 2025
Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tian- jun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wen- wen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Llama 3 Herd of Models. 2024. doi: 10. 000/5//AXXX..24.0..22 21783. URL httpttt//aaxii.. wwtp://ogg/bbb////egeoeee 21783. Publisher: arXiv Version Number: 3.
Alexander Gurung and Mirella Lapata. CHIRON: Rich Character Representations in Long-Form Narratives. Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 8523-8547, 2024. doi: 10.18653/ v1 /003///11222222//2/ wew.ffiiinng-ennll.44. //aclanthology. hothhologyoorg2222 findings-emmlp 499. Conference Name: Findings of the Asso- ciation for Computational Linguistics: EMNLP 2024 Place: Miami, Florida, USA Publisher: Association for Computational Linguistics.
URL https:
Edward] Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In The Twelfth International Conference on Learning Representations, 2024a. URL https: wtwprrvviee..nttoouuiiiiiiieedee
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Open- rlhf: An easy-to-use, scalable and high-performance rlhf framework. ar preprint ar XrXiv2245..1113, 2024b.
Fantine Huot, Reinald Kim Amplayo, Jennimaria Palomaki, Alice Shoshana Jakobovits, Elizabeth Clark, and Mirella Lapata. Agents' room: Narrative generation through multi- step collaboration. In The Thirteenth International Conference on Learning Representations, 2025. URL https::t:///e htttt:::///pppeevviiiii http:///pnnreeeeeeeeeeooorrrrrra hett/forummdd--=f/ffFF//
Mete Ismayilzada, Claire Stevenson, and Lonneke van der Plas. Evaluating Creative Short Story Generation in Humans and Large Language Models. 2024. doi: 10.48550/ (0.48550/ARXIV. ARXIV. 2411.02316. URL https:://arke http://axxvvvorrrbbsr///////
abrrg/bbs/2411 02316. Publisher: arXiv Version Number:
4.
Jennie Jarvis. Crafting the character ARC. Beating Windward Press, September 2014.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu,, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao,
15

---

Published as a conference paper at COLM 2025
Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling Reinforcement Learning with LLMs. 2025. doi: ARX00/2511..21111111 125999. URL
12599. Publisher: arXiv
Version Number: 2.
Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip H. Torr, Fahad Shahbaz Khan, and Salman Khan. Lim post-training- A deep dive into reasoning large language models, 2025. URL https: / //axxiv http:///axxvv.orgala// http://ww.rrrabb//2....1111 2522.
Barbara Kyle. Page-Turner: Your Path to Writing Novel That Publishers Want and Readers Buy. Rosethorn Books, October 2016.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafford, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing Frontiers in Open Language Model Post- Training. 2024. doi: 10.48550/ 008550/ARXI..241..10 ARX1V22411.15124 URL https://arxxvvorr// hearrr/abb//111/// 15124. Publisher: arXiv Version Number: 4.
Zhong-Zhu Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhijiang Guo, Le Song, and Cheng-Lin Liu. From system 1 to system 2: A survey of reasoning large language models, 2025. URL
17419.
Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. Aligning with human judgement: The role of pairwise preference in large language model ev evaluators. In Dipanjan Das, Danqi Chen, Yoav Artizi, and Angela Fan (eds.), First Conference on Language Modeling COLM 2024, 2024.
Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. Frontmatter. Cambridge University Press, 2015.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruy Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839-849, 2016. doi: 10.18653. (0653/vl//M/ //////111111////// 1098. URL http://accc/ lee.l..bb.. org/ anthology/Nie 198. Conference Name: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Place: San Diego, California Publisher: Association for Computational Linguistics.
Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. Laser: Learning to adaptively select reward models with multi-arm bandits. arXiv preprint arXiv:2410.01735, 2024.
Litu Ou and Mirella Lapata. Context-aware hierarchical merging for long document sum- marization. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 5534-5561, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. URL h hww.aannntooooooooo/o//////e org/ 2025. findings-acl. 289/.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhiní Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744. Cur- ran Associates, Inc., 2022. URL https: httttpe///proooeiiingg..... neurgggs..eeriiie meurips.co capppr files/paper/ 0022/File/Dleefde5bee364333 35511455558884444444444455583113 -Pper oop Conferenc pdf.
16

---

Published as a conference paper at COLM 2025
Xiangyu Peng, Kaige Xie, Amal Alabdulkarim, Harshith Kayam, Samihan Dani, and Mark Riedl. Guiding Neural Story Generation with Reader Models. Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 7087-7111, 2022. doi: 10.18653/v /v1/ 2022. www.ffindnggeennn eegsseemll..26... URL https:/ /acp/athaaaaaooy org/2822, findings-emme
Conference Name: Findings of the Association for Computational Linguistics: EMNLP 2022 Place: Abu Dhabi, United Arab Emirates Publisher: Association for Computational Linguistics.
Chau Minh Pham, Yapei Chang, and Mohit lyyer. Clipper: Compression enables long- context synthetic data generation, 2025. URL
org. 14854.
Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, and Kai Chen. HelloBench: Evaluating Long Text Generation Ca- pabilitiee of Large Language Models. 2024. doi: 10,48550/ ARXIV.2409 16191. URL https://arxiv. www.rriw..oog.r///..ee wtt./or////////////// 16191. Publisher: arXiv Version Number: 1.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report. 2024. doi: 10.48550/ ARXIV URL https://arxiv.ort
15115.
Publisher: arXiv Version Number:
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Yijia Shao, Yucheng Jiang, Theodore Kanell, Peter Xu, Omar Khattab, and Monica Lam. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6252- 6278, Mexico City, Mexico, 2024a. Association for Computational Linguistics. doi: 10. 18653/ 200033/11/222// www.022.aaac/llooogg..... 347. URL https: https::/aclaathhllog/ org/2024. naacl-long 347.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Math Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024b. doi: 10. 10.48550/ ARXIV.2402. 03300. URL
033000000000.... 03300. Publisher: arXiv Version Number: 3.
Toby Simonds and Akira Yoshiyama. Ladder: Self-improving Ilms through recursive problem decomposition, 2025. URL https://aaxiv...r http//aaiivvorraa/////aa/ /ttp://////23300355/
N Stewart, G Brown, and N. Chater. Absolute identification by relative judgement. Psycho- logical Review, 4(112):881-91,, 2005.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20 Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Yunhao Tang, Sid Wang, Lovish Madaan, and Rémi Munos. Beyond verifiable rewards: Scaling reinforcement learning for language models to unverifiable data, 2025. URL https://arxiv.ct wwwp:riwi..oo/bbbboa aa////bs//2/50 2503. 19618
Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu.
Offline reinforcement learning for LLM multi-step reasoning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Vienna, Austria, July
Association for Computational Linguistics: ACI. 2025, pp.
17

---

Published as a conference paper at COLM 2025
2025a. Association for Computational Linguistics. ISBN 979-8-89176-256-5 URL https: //aclanthology.oo hwwp://lloooooooooooooooooo org 2025. findings--ace 461..64/.
Qianyue Wang,
Jinwu Hu, Zhengping Li, Yufeng Wang, Daiyuan Li, Yu Hu, and Mingkui Tan. Generating long-form story using dynamic hierarchical outlining with memory- enhancement. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Techrologies (Volume 1: Long Papers), Pp. 1352-1391, Albu- querque, New Mexico, April 2025b. Association for Computational Linguistics. ISBN 179-8-89177-189-6 doi: 10. 10.18653/v1 htt.///22..aal-ooo...5. URL https: //tp///aaaaaanto..... org/2025. naacl-long. 63/.
Yichen Wang, Kevin Yang, Xiaoming Liu, and Dan Klein. Improving pacing in long- form story planning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10788- Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2023 findings-emnlp.722 7.227. URL
org/ 2023. findings-emnlp 723/.
Yuxin Wang, Jieru Lin, Zhiwei Yu, Wei Hu, and Börje F. Karlsson. Open-world Story Generation with Structured Knowledge Enhancement: A Comprehensive Survey. 2022. doi: 008850/ARXIV.21111 24.6444. URL htt.httss:/// httppii///axxv..orrr wwt.:orgg/bbb//////////// 04634. Publisher: arXiv Version Number: 3.
Zhihua Wen, Zhiliang Tian, Wei Wu, Yuxin Yang, Yanqi Shi, Zhen Huang, and Dongsheng Li. GROVE: A retrieval-augmented complex story generation framework with forest of evidence. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3980 3998, Singapore, December 2023. h0/5/v112222//// we0.22..iinings-eeee findings-emmll.2266. URL https: http://llccllattoooo....ooo .wg.l/oooooooooooooooeeee wrw.orgg///// 2023. f findings-emnlp. 262/.
Association for Computational Linguistics. doi:
Yuhao Wu, Ming Shan Hee, Zhiqing Hu, and Roy Ka-Wei Lee. LongGenBench: Benchmark- ing Long-Form Generation in Long Context LLMs. 2024. doi: 10.48550/ ARXIV.24090020 0.0066 URL httpptp:s/aa....o/////aaaaa/////
02076. Publisher: arXiv Version Number: 7.
Kaige Xie and Mark Riedl. Creating suspenseful stories: Iterative planning with large language models. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume Long Papers), pp. 2391-2407, St. Julian's, 's, Malta, March 2024. Association for Computational Linguistics. doi: 1118653/v1/2224..0 1447 eacl-long. 147/-
147. URL https: hatp:::///claathhologg/o org/ 2024.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-RL: Unleashing LLM Reasoning with Rule- Based Reinforcement Learning, February 2025. URL http://arxiv.. org/ abs/2502. 14768. arXiv:2502.14768 [cs] version: 1.
Zhuohan Xie, Trevor Cohn, and Jey Han Lau. The Next Chapter: A Study of Large Language Models in Storytelling. In Proceedings of the 16th International Natural Language Generation Conference, pp. 323-351, Prague, Czechia, 2023. Association for Computational Linguistics. doi: 10.18653/vo /11655/vl// 202../222--nnnn- main.2.....3. URL https: // http://aclannn /ataaaaaaannttyyy..aaeaa org/ 2023. inlg-main. 23.
Rui Xu, Xintao Wang, Jiangjie Chen, Siyu Yuan, Xinfeng Yuan, Jiaqing Liang, Zulong Chen, Xiaoqing Dong, and Yanghua Xiao. Character is destiny: Can large language models simulate persona-driven decisions role-playing? arXiv preprint arXiv:2404.11 121338, 2024.
Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Pp. 4393-4479 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v http//2eennlmmmmaii.... / //acl //aclanthology.com org.2022..c2....ooo.. wttp://22..eemmmmm....... 296/
URL https:
18

---

Published as a conference paper at COLM 2025
Kevin Yang, Dan Klein, Nanyun Peng, and Yuandong Tian. DOC: Improving Long Story Coherence With Detailed Outline Control. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3378-3465, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.18653 / 190. URL /tR..ttts///alllntholoyyy www.:ey..rrggr2....o/ wrtp://o///....loooooooo... acl-long. 190.
Taewoo Yoo and Yun-Gyung Cheong. Leveraging LLM--onstructed Graphs for
Effective Goal-Driven Storytelling. 2024. URL https://ww/ semanticscholar.
Xinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin, Xintao Wang, Rui Xu, Jiangjie Chen, and Deqing Yang. Evaluating character understanding of large language models via character profiling from fictional works. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8015-8036, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ v1 11/2222..emmit www..eemml-mmii...... l/aclanthology org/2024. emnlp-main 456/.
URL https:
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng, Wild- chat: im chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards, 2025. URL https://arxiv http://ww..rrg/b////////// 19590.
Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text. 2023. doi: 10.48550/ 00055//AXXX..205513300 URL https::/arxxi..rrr abs/2305. 1334.. Publisher: arXiv Version Number: 1.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers, 2025. URL wrg./bs//505..... 21493.
https://arxxv.orr/bbb/2222
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. ar Xiv preprint ar Xio::909.00899. 2019. URL https://arxiis www. harg/abs/ii 000//99999909.....
A Data Curation
A.1 Book Selection
We selected fiction books based on popularity, critical acclaim, and the availability of chapter summaries SuperSummary. We excluded all sequels and books based on previous stories (e-g., retellings of myths) to ensure that all relevant information was contained within the book itself. We also excluded collections of short stories, stories featuring existing characters (e.g,, a known detective character), and stories relying on visual information (e.g., graphic novels). We also aimed include variety of genres; our train/test vaai/daaliiionn set all satisfy the following conditions:
1. At least one Sci-Fi book (that is not also fantasy), one fantasy book (that is not also Sci-Fi), and one book that is neither
2. One historical-fiction book
3. One romance book
4. One young-adult book, and one adult book
These conditions meant to ensure that when test for generalization not overfitting to specific genre/ styre/type of book.
This process gave us the books listed in Table 5.
19

---

Published as a conference paper at COLM 2025
Title
Author
Release Date
Split
The Women
Kristen Hannah
February, 2024
Train
Funny Story
Emily Henry
April, 2024
Train
First Lie Wins Just for the Summer
Ashley Elson
January, 2024
Train
Abby Jimenez
April, 2024
Train
When the Moon Hatched
Sarah A. Parker
May, 2024
Train
The Familiar
Leigh Bardugo
April, 2024
Train
We Solve Murders
Richard Osman
September, 2024
Train
The Book of Doors
Gareth Brown
February, 2024
Train
I Cheerfully Refuse
Leif Enger
April, 2024
Train
You are Here
David Nicholls
April, 2024
Train
The Reappearance of Rachel Price Martyr
Holly Jackson
April, 2024
Train
Kaveh Akbar
January, 2024
Train
Deep End
Ali Hazelwood
February, 2024
Train
The Husbands
Holly Gramazio
April, 2024
Train
Blue Sisters
Coco Mellors
May, 2024
Train
Sandwich
Catherine Newman
June, 2024
Train
A Fate Inked in Blood
Danielle L. Jenson
Febuary, 2024
Train
Heartless Hunter
Kristen Ciccarelli
February, 2024
Train
Water Moon
Samantha Sotto Yambao
January, 2024
Train
Beautiful Ugly
Alice Feeney
January, 2024
Train
The Three Lives of Cate Kay
Kate Fagan
January, 2024
Train
All the Colours of the Dark Burn
Chris Whitaker
June, 2024
Train
Peter Heller
August, 2024
Val
The Warm Hands of Ghosts
Katherine Arden
February, 2024
Val
The Tainted Cup
Robert Jackson Bennett
February, 2024
Val
The Grandest Game The God of the Woods The Mercy of Gods
Jennifer Lynn Barnes
July, 2024
Val
Liz Moore
July, 2024
Test
James S. A. Corey
August, 2024
Test
Where the Dark Stands Still
A.B. Poranek
February, 2024
Test
Witchcraft for Wayward Girls
Grady Hendrix
January, 2024
Test
Table 5: Books compiled in our dataset, with author names, release dates and partition. We separate books into splits better test generalization.
A.2 Chapter Summarization
We generate our high-level story plans and prior-story summaries using the following hyper-parameters (mostly taken from the default generation config):
temperature= 0.6, top-p= 0.9, top_k= 50, max_tokens: min(4096, text-tokens] 0.8)
A.3 Character Sheets
We selected the top three characters based on SuperSummary's Character Analysis details, aiming to represent the main protagonists and antogonists where applicable. When two char- acters seemed equally important, we broke ties via name occurrences in the corresponding book.
We use the same generation Syper-parameters as the original CHIRON work (Gurung & Lapata, 2024), but adapted to the generation config Llama 3.3 70B (temperature= 0.6, top-p-0.99 0.9).
We also use the zero-shot entailment module setting with Llama 3.3 70B, relying on its implicit reasoning abilities for the entailment task instead of explicit CoT and ICL prompting. These changes save significant computational resources by removing the need to generate reasoning steps for each claim across every chapter.
20

---

Published as a conference paper at COLM 2025
Split
# Datapoints
Avg. N.C.
Max. # N.C.
Avg. Prompt
Max. Prompt
Train
1004
2453.1
6677
6404.9
11216
Val
162
2263.7
6732
6382.1
10433
Test
181
2962.7
6217
7018.2
11176
Table 6: Number of datapoints in our final dataset, and correspondingg mean/ meean/mane /maxmuum tokens for 1) the next chapter (N.C.) and 2) the prompt to generate reasoning (Prompt) based on the Qwen-2.5 5 3B tokenizer.
Story Information
Mean
STD
Min
Max
Next Chapter
2498.8
1458.9
265
6732
Previous Chapter
2465.7
1482.3
58
6732
Character Sheets
2072.6
234.3
776
2696
Global Story Sketch
787.0
112.8
615
1102
Prior Story Summaries
676.2
141.5
168
1114
Next Chapter Synopsis
154.21
89.12
14
665
Table 7: Size statistics for each element of our Story Information, reported in tokens by the Qwen-2.5 3B tokenizer.
For summarizing the character sheets the same hyper-parameters used for chapter summarization, but with fewer generated tokens: max max_tokens== 0.8). We summarize each character sheet individually, and concatenate them together in our final story-information prompt. This summarization step drastically reduces the context size needed and makes the character information much denser.
|text-tokens]
A.4 Filtering
We filter out datapoints based on the following criteria to reduce training complexity and to ensure our datapoints are meaningful and informative:
Chapters (c++1) must have 200 words and < 5000 words. We also apply the upper limit filter to the previous chapter to reduce the potential size of the prompt.
We filter out the first two and last two chapters, as they often heavily feature text, and are difficult apply reasoning (e.g., too little infor- mation or too few possibilities) For books at the beginning of a series, the end is also often used set up future installments.
A.5 Final Dataset Size
These steps combined give us a final dataset whose descriptive statistics are given in Table and Table 7.
A.6 Training
Table lists the hyper-parameters for GRPO (top) and SFT (bottom):
More details for reproducing both setups are available in our code. We base our GRPO- training on (Hu al., 2024b). In both cases we selected the best performing checkpoint via validation performance (percent improvement for GPRO and loss for SFT).
A.7 SFT Sweeps
We ran small sweeps to test different SFT hyper- parpmmeerreeters, selected from previous work comparing SFT to RL methods. We evaluated learning rates of (2e-5, 5e-7, 5e-6, 4e-5), and amongst largely similar results found 2e-5 to perform the best by validation loss. Future
21

---

Published as a conference paper at COLM 2025
GRPO Hyperparameter
Value
Learning Rate
5e-7
KI. Coefficient
1e-6
Maax-Geertttio-Leeee
2048
Samples per prompt
16
Rollout batch size
64
Train batch size
64
Epochs
20
SFT Hyperparameter
Value
Learning Rate
2.0e-5
Gradient Accumulation Steps Epochs
10
Table 8: payper-paraeetess and their values. After training, models were selected by the best-performing checkpoint validation loss/reward
work could likely optimize further, but we believe that the largest barrier high-quality SFT performance is dataset size.
A.8 Reward Formulation Ablations (VR-CLI Variants)
Recall that for this work, we defined our perplexity-based Improvement measure as:
Te y, = y (yl ()]x)=== P (y/x, x 100 = [1
PPL
100
PPL,s(y122
PPL (y(x)
and our reward as a piecewise function (although we propose other reward formulations):
0,
< 0.05
0.5, 0.05 < I(SI; <
R(S1;
0.9, if ((5T; <
1, ((S), 2
We also ran ablations to test different formulations of our reward, within the same VR-CLI paradigm. Instead of costly human annotations, we evaluate these different methods via their average percent (perplexity) improvement on our test set, using reasoning produced after RL-training We sample 5 plans to reduce variance. Our "RL-Trained" method, with the previously described improvement and reward formulation, produces an average percent-improvemenn of 0.884.
NLL for PPL Our first ablation keeps everything the same, but switches out perplexity for log-likelihood in the Improvement calculation. We find new percent-improvemen 0.485%, about 45% worse than our method. We noticed a lack of convergence with the same hyper-parameters, so we re-ran training for 30 epochs instead of 20, resulting in a percent improvement of 0.627%, still 30% worse.
of
Unbounded NLL Improvement Our second ablation tests a much simpler version of our reward calculation: the Improvement calculation is reduced to the log-likelihood (without baseline calculation, and the 'unbounded' reward is set equal to the improvement.
(x, log P (y|x, a) ((S1,,
R(SI, Ci+1, p) (SS,,c44)
We find slightly better performance with this simpler reward compared to our default default with an average percent-improvementt of 1.031% % (about 16% higher than our method).
22

---

Published as a conference paper at COLM 2025
This differs from our initial experiments with 3B models (which may have worse reward distributions), which led us to develop the piecewise reward formulation. However, as described in Section 5, one downside to this approach is the worse interpretability during training, as 1) it can be unclear whether performance is better without reasoning, and 2) some samples may dominate the logged metrics.
Unbounded PPL Improvement Interestingly, when we performed the same experiment with perplexity (Lzo (x, y, P (y/xx, ( instead of log-likelihood our performance
dropped to an average of 0.760.
An interesting avenue for future work is detailed empirical comparison of these VR-CLI formulations on creative writing tasks and beyond.
A.9 Lessons from Hyper-paarametr Tuning
Due to the high computational cost of our experiments, we were unable to do large hyper- parameter sweeps. We did, however, run many initial tests to gain insight on useful hyper-parameters, which we share here to help guide future research in the space.
We found the following results during our hyper-parametts tuning to be helpful:
Low KL-divergence coefficient Similar to recent results in verifiable reward domains (e.g., math; Lambert et al. 2024), we found that a low KL term (the default parameter in some libraries 0.05; we use 1e-6) improved performance. We hypothesize that lowering this term encourages more exploration by the policy model, and that our reward definition is robust enough that it doesn't 't require significant regularization. However, we do find some cases of significant repetitions in our reasoning traces, which may be alleviated by 1) training on a larger dataset and 2) increasing the KL-divergence coefficient.
Longer max-generation lengths While the majority of our reasoning traces are under 1024 tokens, we found that increasing the maximum generation length to 2048 stopped our reward from converging early low value. As we see significant fluctuations reasoning trace-length during training, we hypothesize that using a higher maximum length prevents potentially informative traces from being cut off and incurring poor reward (as the plan at the end would be malformed).
Number of samples We found that increasing the number of generations per sample im- proved performance even when accounting for the additional computational cost. As described earlier, the initial average improvement very low, so we hypothesize that in- creasing the number of generations increases the likelihood of a non-zero reward, allowing the model to learn from more examples at the beginning of training.
In similar vein, we found it fruitful to spend time tuning the prompt to increase the likelihood of positive reward from baseline models. We would also encourage future work in data selection during training to select informative data points (Cui et al., 2025; Xie et al., 2025).
B Chapter Generation
To make model generations more closely comparable to the original chapters and each other, bound the length of our next-chapter generations to be between half and 1.5 times the length of the original (in tokens): [0.5 11, |
We also apply some automatic truncating to the generated chapters to replicate a more realistic scenario. For example, we cut off text after "### End of Chapter" to avoid making judgments non-next-chapter text. These rules applied to all completions but induce changes almost exclusively in SFT completions, which often devolve into repeated and unrelated text.
Our automatic formatting rules are:
Cut off text after end-of-chapter signifiers
23

---

Published as a conference paper at COLM 2025
Model
# Words
Unique Words
Unseen Trigrams
Rouge-L
Rouge-L Prec
3B-B
1383.10
35.60
83.30
0.100
0.284
3B-SFT
1149.30
25.30
56.00
0.080
0.320
3B-BR
1441.20
34.10
82.80
0.100
0.277
3B-RL-TTriined
1353.20
34.90
82.50
0.100
0.283
7B-B
1486.50
32.70
77.80
0.100
0.286
7B-SFT
1278.10
23.70
53.80
0.090
0.308
7B-BR
1568.00
31.80
77.50
0.110
0.282
FB-RI-Trained
1479.50
33.20
77.20
0.110
0.306
3B-BR-REAS
845.80
29.80
62.20
0.090
0.380
-r-L--Trined-RRAS
721.60
25.50
48.80
0.080
0.405
7B-BR-REAS
777.00
30.50
60.70
0.080
0.373
BE-RL-Trained-RE.. 796.40
26.80
29.80
0.100
0.455
Table 9: We calculate automated measures of length and diversity from the generated chapters across our variants: Words (average number words per chapter), Unique Words (percentage of unique words within chapter), % Unseen Trigrams (percent of all trigrams in chapter that were not present in SI). We find broadly similar lengths and uniqueness, indicating that the preferences come from higher-quality storytelling instead of longer stories. We also report metrics for the reasoning traces 'REAS', which show greater variability.
Cut off text after lines of more than 10 words are repeated three times
Cut off text after a chunk of 20 words has been seen 10 times
Cut off text after a chunk of 20 words has been seen once before and contains or fewer unique words
For fairness, we do not apply these rules to reasoning traces (to simulate a hands-off using reasoning), although they do occasionally also suffer from mode-collapse and repetitions.
Automated Metrics for Generated Chapters/Reasoning Table contains automated mea- sures of length and diversity, across our chapter generation variants. The biggest differences come from the SFT-based chapters, which produce shorter and less diverse chapters. In na-
contrast, the differences between our other variants are fairly small pellorrefonnngg ANOVA test between each of our variants (one test per metric, e.g., Words) does not pro- duce any statistically significant results (p 0.05), with the exception of Rouge-L Precision. As our automated metrics largely imply lexically similar stories across variants, we believe that annotators were not swayed in their judgments by differences in length or lexical diversity.
We interpret the Rouge-L. Precision result (Base and Base- Base-Reasoning have lower precision than SFT and RL-Trained) as implying our RL-Trained and SFT settings pull more text directly from the Story-Information.
We further break down Rouge-L Precision in Table 10 by using different pieces of story information as reference. We find that both generated chapters and reasoning traces draw heavily from all pieces of story information, but reasoning traces seem to more heavily use the Next Chapter Synopsis and Character Sheets. Trained reasoning traces also tend to have higher Rouge-L Precision than untrained reasoning, across 3B and 7B variants and story-information categories.
C Human Evaluation: Collecting Annotations
Annotators were recruited through Prolific their primary language was English and their employment is in creative writing. Despite only showing the story information SI; (instead of the full story text), the task is very time-consuning, taking annotators an average of 30 minutes per comparison. We filtered for useful annotators by duration spent the task
24

---

Published as a conference paper at COLM 2025
Model
Prev. Chap.
Prior Sum.
Plot Sketch
CSheets
Next Chap. Syn.
3B-B
0.197
0.106
0.116
0.171
0.051
3B-SFT
0.238
0.110
0.121
0.178
0.070
3B-BR
0.191
0.101
0.112
0.166
0.048
3B-RL-Trained
0.197
0.106
0.116
0.171
0,053
7B-B
0.207
0.099
0.108
0.160
0.061
7B-SFT
0.229
0.105
0.117
0.169
0.066
7B-BR
0.200
0.097
0.107
0.158
0.056
7B-RI--Trained
0.214
0.105
0.115
0.167
0.065
3B-BR-REAS
0.235
0.167
0.183
0.257
0.113
3B-RI-TTraind--RA.
0.248
0.178
0.195
0.243
0.186
7B-BR-REAS
0.229
0.162
0.180
0.244
0.114
MERLLLriind-REAA 0.243
0.200
0.220
0.322
0.120
Table 10: Rouge-L Precision for the generated next-chapters and the reasoning traces, using different pieces of the story-information SI as reference. We find that the reasoning traces contain more overlap with all pieces of the story information, but that there doesn't t seem to be an obvious correlation between specific pieces of the story information and model variant success. 'rrvv. Chap.' refers to the previous chapter, Prior Sum.' to the summary of the previous story, 'Plo Sketch' to the global plot sketch, 'CSheets' to the character sheets, and 'Next Chap. Syn.' to the synopsis of the next-chapter.
Task Guidelines and Consent
Procedores: Thank you taking part exprimenti stady; you presented partial information a global plan story, the information currentl known to the reader, and two actental contmnttions ons tth next section of the stopf, Your task is to jidge the story co co-tinuat one slong the report
language
good),
brief
justifcation for your atting No expert knowledgs requied perform this
Guidelines: Please careluly read the information, and then cansicer the cont must provided.
were the Voluntary Participation: This study performed by reaearcher at the Univaratty Edinburgh- If you have any questom,, feel free to contect Alesandier Gurung Gurung-1 udiacutt Partidpation voluntary have any parsonal Information, resparman,
to udge continuntions
auther
infomattion represents globel
story.
currentyy
thd from the
time. collected be research to your amarymoua
purptnon only. experiment.
the purpose conducting
you give consent to take part please click agree' below.
agrne
diaggree
Figure Task guidelines and consent for Prolific annotation task
(at least 50 minutes per 3 datapoints) and thoughtful justifications (at least 10 words on average). We paid annotators f14 per three datapoints or an estimated E9.33 per hour.
Figure shows the initial task guidelines and consent form given to Prolific annotators. Figure 4 shows the instructions and pieces of story information shown annotators for the example page, as well as the example annotations shown.
Table 11 shows justifications given by annotators for each dimension. We don't use these justifications directly, but take these detailed responses as evidence that our annotators are engaging with the task.
D Annotator Agreement and Correlations with Improvement
Annotator Agreement To validate our annotation procedure, we collect an initial dataset 20 pairwise preference datapoints between our Base and Base-Reasoning variants using Qwen 2.5 7B-1M the base model. We expect this comparison to induce the smallest differences between the story continuations due to the untrained nature the reasoning and the higher quality of the generator. We evaluate annotator agreement using Fleiss's Kappa (N = 20, = 3). We find fair agreement for this difficult comparison, comparable to the agreement found in Yang et al. (2023) in their most difficult settings.
25

---

Published as a conference paper at COLM 2025
Dimension (Variants)
Sample Justification
Plot (BR-RL)
Continuation (RL) makes better sense in progressing the plot by focusing on the repercussions of Rose's departure and the girls' se- cret efforts to find a spell to help her, which follows the established rebellion and witchcraft themes. [...]
Character (B-RL)
[..] Continuation A (B) is more about immediate drama without enhancing our insight into the characters, so B (RL) story.
in line
with their development and the thematic concerns Continuation B (RIJ provides more understated and plausible devel- opment of setting and character, particularly through the secretive, emotional interactions between Fern, Holly, and Rose, and the unveil- ing the barn as a sanctuary. [...]
Development (BR-RL)
Creativity (BR-RL)
Continuation (RL) has more imaginative and engaging details, such as the secret rendezvous to find spell and the emotional stake of saving Rose's baby, which are aligned with the rebellious and em- powering themes of witcheraft in the narrative. [...]
Language (BR-RL)
Continuation (RL) uses more varied and richer language, including rhetorical devices like "turned into knife" and emotional depth in dialogue, which contributes more atmosphere to the story. [...]
Overall Quality (B-RL)
Continuation B (B) is preferred because it aligns more closely with the established character arcs and plot points, such as Liska's growing sense purpose and the Leszy's ' complex past, while introducing new, engaging elements like his pact with Weles. [...]
Table 11: Sample sentences from annotators' justifications, by dimension (7B variants).
The lowest agreement found was for our 'Language' dimension, while the highest agreement was found for 'Creativity' and 'Overall Quality' dimensions. As we find minimal surface- level lexical differences between chapter generations, we believe language judgments may be more prone subjectivity. Fleiss' Kappa values are shown in Table 12.
Initial experiments showed that generations from 7B-models are almost exclusively pre- ferred to 3B generations, and true continuations are always preferred model generations, so we do not collect such preference combinations.
Dimension
Fleiss' Kappa
Plot
0.180
Character
0.102
Creativity
0,218
Development
0.186
Language
0.060
Overall Quality
0.210
Table 12: Fleiss' Kappa values (N = 20, k = 3) across our annotation dimensions.
D.1 Correlation Between Human Judgments and Improvement
We use data from three (7B) variant-comparisons: the Base vs. Base-Reasoning, Base vs. RL-Trained, and Base-Reasoning vs. RL-Trained. The pairwise judment is converted to an ordinal value of 0 (preferred the Base continuation), (no preference), and 2 (preferred the continuation with reasoning). We find Spearman's 's rank correlation of (p = 0.33, p 0.01, N 60).
E Effect of Genre on Reasoning Performance
We explore the impact of genre on our pairwise annotations. These results are based on the books within our test, the number of annotations per genre, per variant-comparison is
26

---

Published as a conference paper at COLM 2025
Dimension
SFT-B BR-B BR-SFT RL-SFT RL-B RL-BR
Genre
BR
RL
Diff
Plot
5.6
58.8
83.3
83.3
66.7
57.1
Sci-Fi
-0.32
0.09
0.41
Character
17.6
52.9
80.0
94.1
52.9
69.2
Fantasy
-0.35
-0.09
0.26
Creativity
17.6
66.7
77.8
86.7
53.8
38.5
Romance
-0.5
-0.18
0.32
Development
22.2
61.1
73.7
92.9
66.7
57.1
Historical
-0.82
-0.14
0.68
Language
15.0
62.5
78.9
94.4
46.7
42.9
Overall
5.9
60.0
85.0
94.1
52.9
62.5
Table 14: Average percent improve-
ment by genre our test set, 3B model variants. 'Diff' is the addi- tional percent improvement gained by RL-training our reasoning. Model shorthands are in Section 7.
Table 13: Bradley-Terry preference probabilities (%) for 3B variants. RI-TTrinned is preferred across almost all dimensions, although the effect is smaller than in 7B variants. See Section for variant details and Table for 7B values.
very small (as low as in some cases). Table and Table respectively show win-rates and preference probabilities for overall quality, across model sizes and variants.
Table shows the impact of reasoning and trained-rease reaionire on average percent- improvement, broken down by genre for 7B models. Trained reasoning produces higher average improvements across genres, including moving the historical genre from negative average improvements to positive. The biggest shifts in average improvement occur in Sci-Fi (1.0) and Historical (0.93). Sci-Fi has the highest resulting average improvement, with an average perplexity improvement of 1.68%.
Investigations with 3B-based Variants
Figure 2b shows the relative strength scores, Table contains the Bradley-Terry preference probabilities, and Table 16 contains the true win-rates (including ties) for each variant- comparison. We also compute the average percent improvement by genre in the same manner the 7B models, and report it in Table 14. We also break down the by-genre win-rates and preference probabilities in Table 17 and Table 18 respectively, for both 3B and 7B model sizes. Finally, report the automated metrics for 3B models well in Table and Table 10.
Relative to the performance of 7B reasoning variants, 3B variants noticeably weaker and exhibit a less strong improvement over baselines. For example, the Bradley- Terry preference probability of RL-Trained over Base is 52.9% for 3B models, compared to 76.5% for 7B models. The broad trends are similar, however: RL-Trained still has the highest relative strength of the tested variants, Sci-Fi is still the genre with the highest average percent-improvemmen, and automated metrics do not show significant difference between 3B variants, with the exception of SFT models and Rouge-L Precision.
Dimension
SFT-B BR-B BR-SFT RL-SFT RL-B B RL- BR
Plot
5.0
65.0
75.0
85.0
45.0
50.0
Character
25.0
55.0
80.0
85.0
35.0
40.0
Creativity
20.0
40.0
70.0
80.0
65,0
60.0
Development
20.0
55.0
75.0
85.0
45.0
50.0
Language
35.0
40.0
65.0
80.0
60.0
50.0
Overall
15.0
60.0
75.0
90.0
65.0
55.0
Table 15:
The (%) win-rate ( (br+b++ x 100) for 7B models, broken down by dimension. Note the win-rate percentage includes 'same' annotations, while preference probabilities do not, and therefore settings with win-rates < 50% may still be preferred. We still find broadly positive results for including and training reasoning.
27

---

Published as a conference paper at COLM 2025
Dimension
SFI-B BR-B B BR-SFT RL-SFT
RL-B RL-
Plot
5.0
50.0
75.0
75.0
50.0
40.0
Character
15.0
45.0
80.0
80.0
45.0
45.0
Creativity
15.0
60.0
70.0
65.0
35.0
25.0
Development
20.0
55.0
70.0
65.0
50.0
40.0
Language
15.0
50.0
75,0
85.0
35,0
30.0
Overall
5.0
60.0
85.0
80.0
45.0
50.0
br|
Table 16:
The % win-rate (Jbr same 100) for 3B models, broken down by dimension.
Note the win-rate percentage includes 'same' annotations, while preference probabilities do not, and therefore settings with win-rates < 50% may still be preferred. We still find broadly positive results for including and training reasoning.
Genre-Model Size
SFT-B
BR-B
BR-SFT
RL-SFT
RL-B
RL-BR
Sci-Fi-3B
0.0
80.0
80.0
80.0
60.0
60.0
Sci-Fi-7B
0.0
60.0
100.0
100.0
100.0
60.0
Fantasy-3B
0.0
60.0
100.0
80.0
30.0
60.0
Fantasy-7B
30.0
60.0
70.0
80.0
40.0
60.0
Romance-3B
0.0
100.0
40.0
Romance-7B
0.0
40.0
100.0
100.0
40.0
60.0
Historical-3B
10.0
60.0
80.0
80.0
50.0
50.0
Historical-7B
30.0
70.0
50.0
80.0
60.0
50.0
Table 17: The win-rate % ( [bb+bb- same for each variant-comparisonn, by 3B and 7B models, and by genre. We find that including reasoning in Sci-Fi is preferred both 7B and 3B settings, but that it may decrease performance in Romance books. Note the win-rate percentage includes 'same' annotations, while preference probabilities do not. Across sizes and genres, we find broadly positive win-rates for our RL-Trained variants. Table contains the corresponding Bradley-Terry preference probabilities..
[br]
Genre-Model Size
SFT-B
BR-B
BR-SFT
RL-SFT
RL-B
RL-BR
Sci-Fi-3B
0.0
80.0
80.0
100.0
75.0
75.0
Sci-Fi-7B
0.0
60.0
100.0
100.0
100.0
75.0
Fantasy-3B
0.0
60.0
100.0
88.9
37.5
66.7
Fantasy-7B
30.0
75.0
77.8
80.0
50.0
60.0
Romance-3B
0.0
100.0
100.0
25.0
50.0
Romance-7B
0.0
50.0
100.0
100.0
50.0
60.0
Historical-3B
12.5
60.0
80.0
88.9
55.6
62.5
Historical-7B
30.0
77.8
62.5
80.0
75.0
62.5
Table 18: The Bradley-Terry preference probabilities for each variant-comparisot by 3B and 7B models and by genre. Table 17 contains the corresponding win-rates. Our RL-Trained variants are preferred in almost all settings, except for Fantasy and Romance genres with 3B models. However, we find that our RI-Traeeed variant is still preferred over DaseRReasnning in those settings.
28

---

Published as a conference paper at COLM 2025
Pertored
fith deall eup apperde for
seadio vie tire drsocins, eash one maining tho pourtte mtoy contuuctios
Cose intuciions
be partal into manton book global the
inforsatonn
caneeity trow to nurnt undt potmeill ror arreereee five yhu nent ynsters ring)
julde
yoer
phii jugige the contesution aud the and the dino all murice sepeaars Uree gloaal ple the man, ard storration care siy lnsvet to nudde.
Descriptions of Each Section:
gives
Continuations
intoraaioon that alruay are withan about at andl as dan alio to co then nuller pempante
Comtination
Continuation &
janewry of Aloeedy Wiltee Chaptens Juan may of suples that ahruch lover estions (e uaady riki mayy cllati chastess bokor and that
CrarrE Theh locca ohe deverts
ner Chapter The and titra
1902 Goaa bock lesoel people pork
of
Chesste shorks of intornal an alwaay the thes nat claraciaaia raiding
pnrith lewe in book.
Sha't decilo the papernyy the
micher's vigiis bndlrs
hareeeallri dlovdd servil
prevost pae for
junt
froms,
perplattee wellll sounninig ring snthore witting ryle Note therche coriii astion contr arder evert thes agiee.
the sarii Coentia fase
di gre
This Chapaers Syropes A ide wass ahoul huppor the upic iing chapler witvre me here noad bn founed expanded apon atttionn
Ood
Ars losks maity mise
compertyy aepancy "heae'
Detailed Annotation Instructions:
are aiilar conpicy aslad uee your aeeranc
right the ouderr chagge thil
Lovi bock poundiin
lacc snroo and lorpinn oalk mnsettal
to keos you pelyeee es cae fur
logie
whll insernmeees Euperiing
Chersten antere reveree iefervable ceverttally refert mection
ullerach thean chandter the and showigg are mnmomabir
Fro
Createitt These oh mmppging charecter, themes, and peryy The do shoule not you
vole aflec teit wer supliblly reneioed peangs
torgeent the munia wakk singe
Deokgeewss the an and unth wa chide lavire vu and bed rittiannnn vnlivel with verecia drall that the unde used thair plac rany Agarzor cnaalaa
Langaage owe tang. sge used dha ow marttoo, to ineeeding offckts The the avid of spetiine vas beries used eee wertowelly overt rumatee thevedd in engueiicetoon
wa ce chactins
Proferseos
uddtton to gerting yourt simnnn ofe divertions it sko the loow whether infpeed sectnng ng stays the gare stibu ansy not your caras plase
seenting
fanry cnnciiioo dip
The
Story Information:
Open: High Level Story Summary//lan
Opan: Summary Alreacy Written Chapters
Open: Character Sheets
Open: Previcus Chapter
Open: This Cheoter's Synopsis
Open: Detailed Annotation Instructions
(a) Example instructions
(b) Example annotations
Figure 4: Screenshots of example instructions and annotations given to Prolific annotators. The annotation task looks the same, but without filled-in answers.
29

---

Published as a conference paper at COLM 2025
Generated Plan Excerpt
Next-Chapter Excerpt
Book Name
Dafyd's fears and confessions to Campar veal his deep sense of responsibility and emo- tional vulnerability. L... This chapter will explore the moral and emotional dilemmas faced by the characters, particularly Dafyd [...]
Dafyd struggles. know it's the right thing do. know, but it's hard. [...]"I don't want to lose you people," Dafyd said.
The
Mercy of Gods
The Leszy's plea to killed adds deeply emotional and complex moment, as Liska struggles with the decision to save him or sacrifice him.
Killl me, whispers hoarsely. 'Before takes over. But Liska canmot bring herself to draw the knife.
Where
the Dark
Stands Still
Chapter will see the girls perform power- ful spell to exact revenge on those who have wronged them, particulariy Dr. Vincent and Miss Wellwood.
Arrange yourselves around the circle, stand- ing at the points of the compass as your coven leader lights the fire before the altar and in- vokes the Triple-Faced Goddess. [...] "I give it the name of Wellwood. Let the pain she visits on me returned threefold."
Witchcraft
for Way-
ward
Girls
Alice's emotional state will be central focus, she begins to question her role in the family and her own desires. The chapter will explore the dynamics between Alice and her family, particularly strained relationship with Peter (...]
Peter's 's reaction to her mistakes began to morph from amusement to annoyance. [...] In Albany, Alice would have been lonely, if not for her son. [..]] What she thought about most the world in those days was Bear, and her collsonsunigg love for him. [...] He felt to her as much like coach as husband [...]
The God of the Woods
Table 19: Excerpt sentences from generated plans (from the RL-trained 7B model), matched to relevant sentences in the true next chapters. The generated plans had positive percent- improvement scores (2.51%, ), and the existence of matching sentences in the next chapter indicates that our Improvement metric aligns with our conception of good plan. It is also notable that many of these plans highlight thematic and stylistic points that may not show up in traditional chapter summaries. Perhaps unexpectedly, we also found overlap between plot events referenced the generated plans and the synopsis provided. The selected excerpts were not present in the synopses.
30

---

Published as a conference paper at COLM 2025
G Example Reasoning Traces
Reasoning Trace Before Training, Qwen 2.5-7B-Instruct (1.50% Improvement)
### Detailed Plan for Chapter 10:
1. "Setting the Scene:" <citation>ssuure A says X</citation,,, therefore <reasoning>>eassoinnng ciiationnsssssss
says /</iiiation>,, therefore <nasoning? nessniigg neawonnng>>wwsmnn>>>>>> < <citation>>soune says XayysX////iiaa X</citation,, therefore reasoning reasonnngg
2. "Fen's Reaction the Book." <citafion>soure A X </iiatiin>, therefore
crasoningSmssoonnng <citation><soure says X</citatio>>, therefore mosomn A says X<//ci<ation therefore <reasoning>> reasoningg
3. " "Rose's Skepticism:* <citation>souue says X< /cilation>, therefore Tressnniggrresonnnnn--ooo
A says X< </iitation>, therefore <neasonnng reasuming? <citation> A says X</cittatin>>>> therefore <reasoming reasening?
4. "Zinnia's 's Doubts:" <citatio>soure> A says X< /citation>, therefore "aaaoonnnggreassniii
ciitation>souree
says X< /citation>, therefore <reasoning> reasoming <citation> source A says X< X</ciiatton>,, therefore
<rraasoning: g. geasoning2rresoonnng reassning
5. "The Spell and Its Execution:* <citation>> source A says X</citatio>>, therefore
<reasoning reasoning> <citation>ssuree A says X< ///aaaiion>,, therefore <reasoning craasniingrrsscninngg <citation>>soure
A says X< //ct /itatiin?, therefore <nassoinng> reasuning?
Chapter: In summary: Ferm, eager explure its contents, especially since the librarian demonstrated her expertise with her midwifery skills. The girls are drawn to the spells that promise to alleviate their daily struggles, such as Dr. Vincent's mistreatment and Zinnia's
eel Summary
receiving book "How
Gmovy Witch" from Miss Parcae,
vomiting, Rose, initially skeptical, begins to the potential in the book, but Zinnia remains doubtful. When they decide to cast spell to transfer Zinnia's ailment to Dr. Vincent, they gather the necessary ingredients, including an egg, a lemon, and believe the spell working. They hide the witch bag in Dr. Vincent's 's office, hoping the spell will take effect. However, they leave office, they hear Dr. Vincent vomiting uncontroilably. which they interpret as sign of success. ends with the girls feeling sense of empowerment and growing bond, despite the consequences of their actions. *** Detailed Unfolding of the Chapter: 1. "*Opening Scene: Fern enters the Home after receiving the book "How to Be Groovy Witch" from Miss Parcae. The Home feels oppressive, and Fem eager to explore the book's contents. She reads the first few pages, feeling sense of empow erment and hope. The other girls, Rose, Zinnia, and Holly, gather around her, intrigued by the buok's promise..
other items. they recite the incantation, the egg turns black, leading them
chapter
2. "Rose's 's Skepticism: Rose, initially skeptical, begins to see the potential in the book. She reasons that the spells are written in the language of the establishment, but she is intrigued by the idea of using them to their advantage She suggests they try spell that promises to alleviate their daily struggles, such Dr. Vincent's mistreatment and Zinnia's constant vomiting.
"Zinnia's 'ss D Zinnia, remains doubtful, initially resistant to the idea of witchcraft. However, she persuaded by Fern and Rese's enthusiasm. They decide to cast a spell to transfer Zinnia's 's ailment to Dr. Vincent, hoping alleviate her sutfering. They gather the necessary ingredients, including an egg, lemon, and various other items. 4. "The Spell and Its Execution:** As they recite the incantation, the egg turns black, leading them to believe the spell working, They hide witch bag Dr. Vincent's office, hoping spell will take effect. As they leave the office, they hear Dr. Vincent vomiting uncontrollably, which they interpret sign of success. The girls feel sense of empowerment and growing bond, despite the consequences of their actions.
5. "CConclusion"" The chapter ends with the girls feeling a sense of empowerment and growing bond. They excited about the possibilities that the book offers, but they are also aware of the consequences of their actions. The next chapter will explore aftermath of their actions and the challenges they face result.
31

---

Published as a conference paper at COLM 2025
Reasoning Trace After Training, Qwen 2.5-7B-Instruct (2.58% Improvement)
#A# Detailed Plan for Next Chaptenitt
1. "FFem's 's Character Development <cilationn> evolves significantly throughout the story, driven by her experiences, interactions, and growing self-awareness. She transforms from desperate and ushamed beenager tu more determined and resilient individual, soeking to survive her time at the Home and prepare for the future. < //reasoning>> <'reasoning'> Fern's experiences the Home, including interactions with the other girls and the authorities, shape her thoughts and feelings. She becomes more aware of the harsh realities of her situation and the lies she has been told about childbirth (snippet 6):< </reasoning""
Soulate Source says X< /citatiion> therefore <Treasoning'> Fern's character
Sitaation-SSuueee A says X X< </iitation>, therefore
2. "The Girls' Experimentation with Witchcraft.* <ciiationn SSource A says X< /citation>, therefore < reasoning The story continues with Fern's daily life at the home, including her struggles with the strict rules and her relationships with the other girls (Chapter 8-9). She bonds with Zinnia,
girl who arrives at the home, reading, and they come up with plan to check out books from the Bookmobile together (Chapter
their shared love of
< </reasoning>> <citaaiionSSour Source says X< /citation>, therefore <reasoning>> The girls forbidden from reading certain books, including those on childbirth, but Mrs. Parcae, the librarian, sympathizes with Fern's desire to significant moment, Mrs. Parcae adjusts the fetus inside Fern's belly, explaining that it has been pressing against her bladder, and gives her book that will help her understand how to be woman (Chapter 9)xx <)<</remsnnnngg reasoning"
learn more about pregnancy and childbirth (Chapter
3. "The Spell and Its Consequences"" <citation Sourtio says X< /citttion> therefore <reasoning>> The book that Mrs. Parcae gave Femn is titled How to Be Groovy Witch. It contains spells for various things, all which seem useful Ferm. Because Mrs. Parcae was adept moving fetus and ending her constant need urinate, Femn has high hopes for this book (Chapter 10). < </reasoning > <citation Source A says X< / ditation>, therefore < creasoning > As they recite the incantation, the egg turns black. They hope that this means
spell working. When they transfer the egg and the other items into witch bag and hide it in Dr. Vincent's office, he begins vomiting uncontrollably (Chapter < 10),<</masoonin massnnnng>
*a* In summary: In the next chapter, Fern's 's character development is portrayed through her experiences at the Home for unwed mothers. The chapter ends with the girls' experimentation with witchcraft, and the spell they cast to harm Dr. Vincent. The chapter concludes with girls' decision join the coven, and their future plans. The chapter ends with jump forward in time, and Neva's reflection on her experiences at the Home. The chapter ends with a sense of closure, as Neva reunites with daughter and finds closure her life. The chapter ends sense freedom, Neva is able to confront her past and move forward. The chapter ends with a sense of satisfaction, as Neva's is life is complete. The chapter ends with sense closure, the story comes to an end. chapter ends a sense satisfaction, as readers have enjoyed the story and have learned from it. The chapter ends with sense of clusure, the story has come an end. The chapter ends with sense satisfaction, as the readers have enjoyed the story and have learned from The chapter ends with sense of closure, as the story has come an end.
32

---

Published as a conference paper at COLM 2025
Reasoning Trace Before Training, Qwen 2.5-3B-Instruct (0.24% Improvement)
#A# Reasoning:
1. "PPrevious Chapter Recap and Next Chapter Synopsis. In the previous chapter, Ferm receives book titled "How to Be a Gronvy Witch" from Miss Parcae. The book contains spells for various purposes, including transferring ailments to others. - The next chapter synopsis indicates that Fern and Rose perform spell to transfer Zinnia's 's constant vomiting to Dr. Vincent, who begins vomiting uncontroilably.
2. "Charrccter Development and Relationships **ern Fern has become more resilient and determined since her arrival at the Home. She has learned to read and has developed a sense of independence and resourcefulness. - "RRose"* Rose shown more compassionate side by standing up for Zinnia and sharing liver shake with her. She remains rebellious and confident but has developed sense of empathy. "Zinnia:** Zinnia has been new arrival and has shown resilience and desire to be part of the group. Her constant vomiting suggests she may be experiencing health
issues.
3. "Miss Parcae's Role:*.. Miss Parcae has been teaching the girls about witchcraft and has been helpful to Fern. She seems to have deep understanding the girls' needs and supportive of their desires for empowerment.
4. "Miss Wellwood's Role:" Miss Wellwood remains strict and judgmental, often criticizing the girls and pushing them give up their babies for aduption. Her presence adds tension to the situation.
5. "Dr. Vincent's Role:" Dr. Vincent is source of pain and suffering for the girls. His treatment of them is abusive, and he has been causing them distress.
6. "Holly's ys Role:" Holly has been mute and often overlooked, but she has shown resourcefulness and has formed connections with other girls, particularly Fern.
7. ""Social Justice and Empowern Empowerent::** - The girls are interested social justice and have been exploring ways resist
the uppressive rules and treatment they receive. The spell they perfurm is step towards empowerment and resisfance. eAu Reasoning:
""Plot Development: . "FFrm and Rose's Spell: Fern and Rose perform the spell to transfer Zinnia's 's constant vomiting Dr. Vincent. The spell works as intended, and Dr. Vincent begins vomiting uncontrollably. This event will likely cause well as Dr. Vincent himself.
significant reaction from the other girls and Miss Wellwood,
"Character Develapment:" "Feen:" Fern will feel sense of acomplishment from successully performing the spell, further boosting her confidence and resilience. She will likely continue seek ways resist the Home's oppressive rules and find ways to empower herself and her friends. "Ruse"* Ruse will be proud of her actions and may become more vocal about standing up to authority figures. Her actions will also strengthen her bond with Fern-- ""iiniaa:* Zinnia will be relieved and grateful for the spell, and she will likely feel more empowered and connected to the group. "Miss Parcae:* Miss Parcae will be pleased with the girls' actions and will continue to support them in their quest for empuw erment and resistance. "Miss Wellwoud:* Miss Wellwood will be furious and likely punish the girls for their actions, further pushing them to resist. "Dr. Vincent:" Dr. Vincent will be traumatized by the spell and may seek revenge, leading more conffict and danger for the girls.
*TTension and Conflict:** - The spell's 's suceess will create power dynamic shift within the Home, with the girls feeling more empowered and the authorities feeling threatened. Miss Wellwood likely punish the girls, possibly by restricting their access to books or other resources, which will further fuel their desire for empowerment. The girls will need to devise a new plan to help Holly, who is still in danger and needs their support.
up with new plan to help Hally. They will also need to deal with Dr. Vincent's reaction to the spell. "Character Points:"* Fern vocal and assertive. Zinnia will be grateful for the spell and will become more involved in the group's plans Miss Parcae will be supportive but cautious. Miss Wellwood will be furious and will likely restrict the girls' access to books and resources. Dr. Vincent will traumatized and may seek revenge. "SSttingg "* The bookmobile will serve temporary sanctuary for the girls, providing them with a sense of freedom and connection to the oubide world.
"Next Chapter Details:** Details:*_**ppp ""Plot." The girls will face Miss Wellwoud's punishment and will need
will continue to be the driving force behind the girls' resistance. Rose will be
#A# In summary: In the next chapter, Fern and Rose will perform the spell transfer Zinnia's 's constant vomiting Dr. Vincent, causing him to vomit uncontrollably. This event will create a significant power shift within the Home, with the girls feeling more empowered and the authorities feeling threstened. The girls will need to devise new plan to help Holly, who still in danger and needs their support. Miss Wellwood will likely punish the girls, further pushing them to resist. The girls will need navigate the aftermath of the spell, dealing with Dr. Vincent's reaction and Miss Wellwood's punishment. The bookmobile will continue to serve temporary sanctuary, providing the girls with sense of freedom and connection to the outside world. This chapter will further develop the themes empowerment, resistance, and the girls' growing bond as they face new challenges together.
33

---

Published as a conference paper at COLM 2025
Reasoning Trace After Training, Qwen 2.5-3B-Instruct (-0.06% Improvement)
#*# Reasoning:
1. "Introduction of the Book "How to Be a Groovy Witch";" - The chapter will begin by introducing the book "How to a Groovy Witch" Mrs. Parcae gave to Fern. <citation CChapter 9-, /citatiin deseribes the book as a paperback that Fem receives from Mrs, Parcae, so this chapter will confirm the title and provide more information about the book. 2. "Fern's Reaction the Book." The chapter will include Fern's is reaction to the book, highlighting her hopes that it will be useful her in her current situation. <<iitation>> Chapter 9< /clation>>mention that Fern high hopes for the book, so this chapter will expand on her thoughts about it.
book, particularly focusing on Rose's 's initial reluctance to read anything described in book. <ciitttion > Chapter 9</ /citation > mentions anything that is described in book, so this chapter will explore her reasoning and the girls' discussion about the book.
"Rallying the Girls Read Book The chapter will describe the girls' reactions
Rose initially doesn't want
4. "Zinnia's 's Doubts about Witches.* The chapter will include Zinnia's 's doubts about the existence witches, based on her previous reactions the idea of witchcraft. <cilation"> Chapter 9-C iitationnddscribii Zinnia's doubts about witches, so this chapter will expand on her thoughts and the girls' discussion about the topic.
"Performing the Spell Transfer Vomiting to Dr. Vincent:** The chapter will deseribe the spell that is described the book, particularly the one that promises to transfer an ailment to another person. <citation > Chapter 9- /citation mentions that the book contains spells for various things, so this chapter will focus on one these spells. 6. "Preppriig for the Spell:* The chapter will describe the materials needed for the spell, including the egg. lemon, and other items mentioned in the book. <<iitation>> Chapter 9-2 /ciaaiion describes the materials needed for the spell, this chapter will expand them and prepare the girls for the spell.
7. "Performing the Spell>" - The chapter will include the process of perfurming the spell, including the recitation of the incantation and the transformation
the egg. <ciitation> Chapter ><//ciationn>de ddescribes process of performing the spell, so this chapter will expand on it and include the transformation of the egg,
8. "Hiding the Spell
Vincent's Office:* The chapter will describe the girls' decision hide the spell in Dr.
Vincent's is office, including the preparation of the witch bag and the hiding of the items. <citation>> -Chapter
9-2 / citatiinn describes the girls' decision to hide the spell in Dr. Vincent's s office, so this chapter will expand on and include the preparation the witch bag and the hiding of the items.
9. "Dr. Vincent's 's Reaction to the Spell." The chapter will conclude with Dr. Vincent's reaction tu the spell, including his uncontrollable vomiting. <citation>> Chapter < /citation describes Dr. Vincent's 's reaction the spell, so this chapter will expand on it and include his reaction.
**# summary: In the next chapter, the chapter will introduce the book "How to Be Groovy Witch" that Mrs. Parcae gives to Femn, describe Fern's reaction to the book, rally the girls to read the book, include the girls' reactions to the idea witchcraft, describe the spell described in the book, prepare for the spell, perform the spell, hide the spell in Dr. Vincent's 's office, and conclude with Dr. Vincent's reaction to the spell.
34


Reviews:
[{"review_id": "96oiSj2kcE", "reviewer": "Reviewer_jQ4x", "summary": "This paper focuses on the long-form story generation via LLMs, which requires the LLMs to track plot and character arcs consistently and coherently in a long context. Different from previous methods, which rely heavily on the hand-designed prompts, it proposes the next-chapter prediction as the finetuning task and uses the completion likelihood as the reward model for RL training. Pairwise human judgments show that the proposed method is preferred across all metrics.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This proposes the next chapter prediction task for long-form story generation. It uses the story information, such as the global story sketch, previous summary, and character sheet, to generate the next chapter. Instead of supervised fine-tuning, it defines Completion Likelihood Improvement to measure the PPL decrease when given the plan. The reward is used in GRPO to optimize the plan generation model to generate a more informative story plan that can help the generation model.\n2. The pairwise human judgements should show that with the story plan, the generated stories are more preferred by humans, and further RL training on the plan generator can make the story better. \n3. This paper also provides a story generation dataset, including 30 new books and an average of 130k long contexts. The information, such as a high-level story sketch and story summary, is helpful for long-form story generation.", "reasons_to_reject": "1. The description of RL for Next Chapter Prediction is confusing. In Section 3, the final output of the Next Chapter Prediction is the generated chapter. However, the objective of RL in Next Chapter Prediction is to optimize the story plan while keeping the chapter generation model πG unchanged. There is a misalignment between the task outcome and the optimization model: it says RL for NCP, but VR-CLI is optimizing the plan instead of directly optimizing the next chapter prediction. It may be better to define the task as the next chapter plan prediction, and the quality of the plan is evaluated by the CLI defined on the next chapter. \n2. The lack of a comparison between the supervised finetuning with the RL via VR-CLI in optimizing the plan generation model. To verify the effectiveness of the proposed reward VR-CLI, there should be a supervised fine-tuning baseline. For example, take the chapter summaries from SuperSummary as the plan and fine-tune the story plan model πR. \n3. It is difficult to interpret the result in Table 4. The proposed method does not outperform in any metric here. It lacks another qualitative or quantitative analysis on why the story generated by the proposed method is better than the other baselines.", "questions_to_authors": "1. Why does Eqn 4 use PPL instead of the log-likelihood? \n2. How do you choose the threshold in Eqn 7? How sensitive the RL training is towards these hyperparameters?", "extra_scores": {}}, {"review_id": "Ri6tRHh1Y9", "reviewer": "Reviewer_oa9L", "summary": "The authors present an RL based training method for long-form story generation. They fine-tune a Qwen2.5 model on a newly proposed task of next chapter prediction (NCP), where, instead of direct prediction of book-length generations, they generate one chapter after the other (with chapters predicted so far being provided as context). The intuition is that the reasoning generated by the model will improve the overall story compared to an SFT that quickly overfits. They use GRPO for training and for the reward they use improvement in per-token perplexity (PPL). Specifically, the reward model is thresholded piece-wise function with the actual reward values depending on the improvement in PPL. This reward paradigm is referred to as VR-CLI in the paper, or, verifiable rewards via completion likelihood improvement, and it is inspired by prior work RLVR (RL via verifiable rewards).\n\nThey curate a dataset of 30 books covering four genres: scifi, fantasy, romance, and historical; and they use 22 of them to train their models. They conduct quantitative and qualitative evaluation, where they compare their RL trained model with multiple baselines including an SFT and no RL baseline. Both quantitative and qualitative results show that the proposed VR-CLI framework leads to better stories compared to the baselines and is favored by humans (although human written stories are still preferred almost exclusively over all).\n\nOverall, I like the experimental setup and this work has some merit. It shows that RL can work for a creative task while being more effective compared to SFT. My main concerns revolve around using PPL as a target for a creative task and potential overtuning of reward values (meaning that if someone were to do this for their own dataset for the same task they would likely need to spend a lot of time figuring the rewards values out) as the authors cannot release the dataset nor the models due to copyright issues.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents a novel take at long-form story generation using the task of next chapter prediction with RL.\n- The presented framework is promising as it makes RL work for long text generation task (whereas RL has traditionally known to be notoriously difficult to stabilize for NLP tasks)\n- The reward modelling is novel albeit potentially too specific for the presented setup\n- Comprehensive experimental and evaluation setup where the authors curate a dataset of new books (not public due to copyright issues) and compare against multiple reasonable baselines with A/B style human evaluation\n- Meaningful insights about the suitability of 3B and 7B models (3B models not suited for the task, while 7B models are)\n- The authors say they will release the codebase, which can be helpful, and this work will potentially be used by others in the community", "reasons_to_reject": "- The fact that it uses perplexity as target for a creative task doesn't sit well with me as lowering entropy or predictability can be considered arguably orthogonal to the intent of creative writing. I understand that it is difficult to come up with a really good measure, but I wouldn't have expected PPL to be the metric of choosing. Using PPL definitely makes sense for a more objective task, where there's less(er) room for creativity, but in this case, I am not sure it \"works\", as neither the test set is out-of-distribution (doesn't contain articles outside training set genres), and human written stories are almost always exclusively preferred over the proposed model (forcing an inter-variant comparison instead). Maybe you can strengthen the message to be revolve around improving an autonomous model (there are bits and pieces at the moment but it doesn't come off super strong right now).\n- Additionally, the exact thresholds in reward function feel quite adhoc to me (I am sure a lot of time was spent on this part) and am concerned that it won't transfer well to another dataset for the same task and someone trying to make it work on their dataset would again be spending a lot of time in figuring these threshold values.\n- The above two reasons makes me feel that RL might be shoe-horned for this task and may make a case that an SFT baseline could've worked with enough care, or maybe it just needed a larger dataset or both.", "questions_to_authors": "- Please add Table 6 and 7 (or a compressed version of them) to the main paper. It's an important detail showcasing how long is long really in the context of this work. I know you mention it in Section 4 but a table is always a good redundancy for such key details.\n- What other rewards have you tried (that \"didn't work\")?\n- Have you tried to use anything other than PPL for your reward? Coming up with those threshold values would've been surely painful, and since you already have a good sense of what metrics to look for (as you ask during human eval), have you considered using something like G-Eval scores (with an open-source base model of course) to provide you with the rewards?\n- Table 3 shows mean percent improvement of what exactly? And what is the range there, 0-1 or 0-100? And what is the base model (the 0 model)?\n- Why specifically chapter-level? Have you tried more finer or coarser-level of planning?\n- Re 3B v/s 7B models, I think it is important to consider not just the number of parameters, but also the data the base model has been exposed to. For instance, your finding may not hold for Qwen3-4b, which is better than even Qwen2.5-72b on some tasks (but since I know Qwen3 has been exposed to more data, this 4b v/s 72b comparison is not really fair).", "extra_scores": {}}, {"review_id": "wdP1og3PF0", "reviewer": "Reviewer_RQpG", "summary": "This paper tackles the challenging task of long-form story generation with large language models (LLMs), which requires maintaining coherence, character development, and stylistic consistency across thousands of tokens. The authors introduce a novel reinforcement learning (RL) framework—Verifiable Rewards via Completion Likelihood Improvement (VR-CLI)—to guide a reasoning model in generating structured plans for upcoming story chapters. The experiments validate the superiority of the proposed model.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Long-form story generation remains one of the most demanding applications for LLMs, where challenges like maintaining global coherence, avoiding repetition, and character consistency are magnified. This paper addresses these directly by proposing a task (NCP) that explicitly integrates story structure.\n2. The use of verifiable rewards via completion likelihood improvement (VR-CLI) is an original contribution. It avoids the need for hand-labeled rewards or hard-to-define heuristics by instead relying on a measurable and model-driven improvement signal. This strikes a good balance between tractability and expressiveness for creative writing.\n3. Leveraging a dataset of real-world books (with chapter summaries, character sheets, etc.) makes the method practical and scalable. The method for constructing structured story information from raw text is well-motivated and carefully documented. The experiments show the effectiveness of the proposed model.", "reasons_to_reject": "1. While the authors compare their method against several baselines (e.g., SFT, non-trained reasoning), more state-of-the-art RL-based methods could strengthen the comparison. \n2. The reward thresholding in Eq. 7 is not well-justified. The current formulation uses manually selected cutoffs (e.g., 0.05, 1, 2), which could significantly affect training dynamics. The paper would benefit from an ablation or sensitivity analysis exploring how these thresholds influence performance. Moreover, it would help to ground the thresholds in empirical observation or provide theoretical rationale.", "questions_to_authors": "1. How sensitive is the model performance to the threshold values in Eq. 7? Have you tried treating this as a continuous (rather than piecewise) reward signal? Have you conducted any ablation studies to assess how variations in these thresholds affect model convergence or final story quality?\n2. Have you considered evaluating against stronger RL baselines beyond non-trained or SFT models? If not, what are the practical challenges in integrating, for example, preference-based RL methods like DPO?", "extra_scores": {}}, {"review_id": "f3Q7gOpONo", "reviewer": "Reviewer_uAHM", "summary": "This paper proposes a reasoning-enhanced framework for long-form story generation, where the generation process is divided into three phases: reasoning, planning, and writing. The authors introduce a task called Next-Chapter Prediction (NCP) and design a verifiable reward function (VR-CLI) based on perplexity improvement to train the reasoning component using reinforcement learning. Experiments across multiple models and genres demonstrate the effectiveness of this approach in improving story quality.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper frames story generation as a structured process of “reasoning → planning → writing,” which aligns more closely with how human authors compose stories. This formulation is more principled than traditional end-to-end generation approaches.\n\n2. The proposed reward signal, comparing generation likelihood with and without reasoning traces, does not require any labeled data and is intuitively sound.\n\n3. The paper is clearly written and easy to follow.", "reasons_to_reject": "The main concern lies in whether perplexity improvement is truly a suitable proxy for story quality in the VR-CLI reward formulation. The paper currently lacks an intuitive explanation for why a decrease in PPL should correlate with a better story. It would help to include more analysis or concrete examples to justify the reasonableness of this proxy reward.\n\n+++++ update+++++\n\nFurther, I examined the section titled “Perplexity improvement correlates with human judgements.” The reported Spearman correlation (ρ = 0.33, p = 0.01) suggests a moderate positive relationship, indicating that higher-quality reasoning traces tend to yield lower PPL. However, I couldn’t find a clear definition of what constitutes “generation quality” in this context. Does a higher Improvement reflect better fluency, coherence, or something else? This should be clarified explicitly.\n\nDespite my concern about the use of perplexity as a reward signal, I still find the idea of leveraging unsupervised signals from the story generation process itself for RL both promising and scalable, as such data can be generated in large quantities.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors uses a clever way to formulate reward function (Verified Rewards via Completion Likelihood Improvement) that allows them to use an unlabeled book dataset as a learning signal for reasoning. They fine-tune a Qwen2.5 model on a next chapter prediction task, where, instead of generating full text, they generate one chapter after the other (with chapters predicted so far being provided as context). The intuition is that model will learn better reasoning and consistency as length increases which is a common pitfall in otherwise long form generation.The authors employ GRPO for training with improvement in per-token perplexity (PPL) as the reward signal. They authors use 30 books for experiment covering four genres: scifi, fantasy, romance, and historical and conduct thorough evaluation. Both quantitative and qualitative results show that the proposed VR-CLI framework leads to better stories compared to the baselines and is favored by humans (although human written stories are still preferred almost exclusively over all).

As several reviewers have noted this is a relatively early and successful adaptation of using RL for open domain challenging creative task while being more effective compared to SFT.  Reviewer oa9L raised several good concerns but authors have addressed them. Overall if the authors address all these in the camera ready and add the clarifications in the extra 1 page, this would make for a strong submission.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept