What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluation
Authors: Dingyi Yang, Qin Jin
Accepted at: acl25

What Matters in Evaluating Book-Length Stories?
A Systematic Study of Long Story Evaluation

Dingyi Yang, Qin Jin*
Renmin University of China
{yangdingyi,qjin}@ruc.edu.cn

Abstract
In this work, we conduct systematic research in a challenging area: the automatic evaluation of book-length stories (>100K tokens). Our study focuses on two key questions: (1) understanding which evaluation aspects matter most to readers, and (2) exploring effective methods for evaluating lengthy stories. We introduce the first large-scale benchmark, LongStoryEval, comprising 600 newly published books with an average length of 121K tokens (maximum 397K). Each book includes its average rating and multiple reader reviews, presented as critiques organized by evaluation aspects. By analyzing all user-mentioned aspects, we propose an evaluation criteria structure and conduct experiments to identify the most significant aspects among the 8 top-level criteria. For evaluation methods, we compare the effectiveness of three types: aggregation-based, incremental-updated, and summary-based evaluations. Our findings reveal that aggregation- and summary-based evaluations perform better, with the former excelling in detail assessment and the latter offering greater efficiency. Building on these insights, we further propose NovelCritique, an 8B model that leverages the efficient summary-based framework to review and score stories across specified aspects. NovelCritique outperforms commercial models like GPT-4o in aligning with human evaluations. Our datasets and codes are available at https://github.com/DingyiYang/LongStoryEval.

1 Introduction
Automatic Story Evaluation involves providing critiques and ratings to assess the quality of human-written or machine-generated stories. This process is crucial for recommendation systems or offering constructive feedback for improvement. Unlike simpler evaluation tasks that focus on fluency and accuracy (e.g., machine translation), story evaluation demands a comprehensive assessment, based on diverse human-centered criteria (Chhun et al., 2022). While recent advances have improved the evaluation of short stories (Guan and Huang, 2020; Guan et al., 2021; Chhun et al., 2022), particularly with the aid of large language models (LLMs) (Jiang et al., 2024b; Xie et al., 2023b), the evaluation of book-length stories (exceeding 100K tokens) remains significantly underexplored.

Evaluating book-length stories poses three major challenges: (1) Data Annotation Constraints: Human evaluation, while the gold standard, is time-intensive and cognitively demanding. As shown in Table 1, existing story evaluation benchmarks focus on shorter texts (100-2,000 tokens). Scaling human annotations for stories exceeding 100K tokens is impractical. (2) Inconsistent Evaluation Criteria: Most prior works rely on predefined criteria for evaluation, but there is no universal standard. Evaluation criteria vary across studies and often fail to

![Our proposed evaluation criteria structure and the reading process: A reader approaches a book with initial expectations based on its genres and premise. The story unfolds through character-driven plots, revealing its themes and world-building through the author’s writing. Through reading, the reader experiences enjoyment, engagement, and emotional impact, and determines whether this book meets the expectations.](page_1012_355_420_312.png)

Figure 1: Our proposed evaluation criteria structure and the reading process: A reader approaches a book with initial expectations based on its genres and premise. The story unfolds through character-driven plots, revealing its themes and world-building through the author’s writing. Through reading, the reader experiences enjoyment, engagement, and emotional impact, and determines whether this book meets the expectations.
reflect actual reader preferences. Our work aims to explore what real readers value in lengthy stories.

(3) Long Story Processing: Book-length stories often exceed the 128K-token context limit of most LLMs, posing challenges for effective evaluation. Even within this limit, processing such long contexts remains challenging for models. Identifying efficient evaluation strategies for lengthy stories is therefore critical.

To address these challenges, we collect ratings and reviews for 600 newly published lengthy novels from online readers. To completely avoid data contamination issues (Chang et al., 2024) that might affect our experimental analysis, none of these books were included in the training data of evaluated LLMs. The raw review data is sourced from GoodReads1, the largest book review platform. Using LLMs, we extract over 1000 reader-mentioned evaluation aspects, analyze the most frequent ones, and organize them into a hierarchical criteria structure (Figure 1). We further compare three types of processing methods for lengthy story evaluation: aggregation-based, incremental-updated, and summary-based. Additionally, we introduce NovelCritique, a specialized model for reviewing and scoring lengthy stories across specified aspects, which demonstrates superior alignment with human evaluations compared to commercial models.

Our contributions are summarized as follows:

• LongStoryEval: A benchmark for lengthy story evaluation. We introduce a large-scale benchmark comprising 600 books (published between 2024 and January 2025), with average rating scores and 340K reader reviews. Raw reviews are converted into structured critiques, overall assessments, and ratings, as shown in Figure 2 (d). Metadata, including book details (e.g., title, genres, premise) and reviewer profiles, is provided to facilitate future research.

• A hierarchical structure of evaluation criteria, and analysis of significant aspects. By analyzing all aspects raised by real readers, we develop a hierarchical evaluation criteria structure with 8 main aspects and 20 sub-aspects (Figure 1; Table 9). Our experiments reveal that plot and characters are the most influential objective aspects, while subjective aspects — emotional impact, overall enjoyment & engagement, and expectation fulfillment — are also critical to overall ratings.

• Explorations on effective methods for lengthy story evaluation. Among the three types of lengthy story processing methods, aggregation- and summary-based evaluations perform best. Our findings suggest that the most cost-efficient method involves generating a concise summary and averaging multiple summary-based evaluation results. Further experimental analysis is provided in §5.4.

• NovelCritique: A specialized model for lengthy story evaluation. We propose NovelCritique, an 8B model capable of reviewing and scoring lengthy stories across specified aspects. It outperforms commercial LLMs such as GPT-4o in aligning with human ratings.

2 Related Works

Story Evaluation. Story generation is a creative and open-ended task, making it more appropriate to explore metrics based on specific human standards. Traditional lexical-based metrics such as BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005) correlate poorly with human judgments. More recent metrics based on pre-trained neural networks, like BERTScore (Zhang et al., 2020) and BARTScore (Yuan et al., 2021), achieve better semantic comprehension. However, they still struggle to align well with human standards in story evaluation. To address this, several works (Guan and Huang, 2020; Ghazarian et al., 2021; Chen et al., 2022; Maimon and Tsarfaty, 2023) have conducted further training on story evaluation datasets or explored methods based on detailed analysis (Jiang et al., 2024b; Xie et al., 2023b) to improve performance. However, these explorations remain limited to short stories generated from ROC and WP datasets. The criteria used might also be restricted to predefined ones (Chhun et al., 2022; Xie et al., 2023a; Wang et al., 2024). These evaluation standards are inconsistent (Yang and Jin, 2024), and how well they align with actual readers’ preferences remains unclear.

LLM-Based Evaluation. The development of large language models also boosts LLM-based evaluations (Li et al., 2024; Gao et al., 2024). Through carefully designed prompts (Chen et al., 2023; Kim et al., 2023) and helpful strategies (Chan et al., 2024; Saha et al., 2024; Lee et al., 2024), existing methods can achieve good correlation with humans. However, methods based on closed-source models

1https://www.goodreads.com
can face problems of bias and inconsistency (Stureborg et al., 2024). Open-source LLM evaluators (Li et al., 2023; Kim et al., 2024), on the other hand, include only a small portion of creative story evaluation data in their pretraining. Considering the important role of lengthy stories in people’s daily lives, we attempt to explore how current LLMs handle lengthy story evaluation, compare different evaluation strategies, and propose a specialized evaluation model.

3 LongStoryEval Dataset

Data Collection. Considering the high cost and time constraints of human annotations, we leverage large-scale online reviews from real readers. Our dataset comprises 600 newly published novels. Due to copyright restrictions, we release only plot and character summaries rather than the full book content. To ensure fairness in our experimental analysis, these books are verified to be absent from the pretraining dataset of our evaluated LLMs, avoiding data contamination issues2 (Chang et al., 2024). For each book, we collect its average rating score and multiple reviews from Goodreads, the largest book review platform. Each raw review consists of the reader’s written critique along with a rating on a 1-5 scale.

Review Processing. As illustrated in Figure 8, raw reviews are often unstructured and lack clarity. Prior works (Gong and Mao, 2023; Lee et al., 2024) have shown that aspect-guided critiques enhance both readability and evaluation accuracy compared to direct scoring or relying on an unstructured overall review. Building on this insight, we reformat raw reviews by identifying user-mentioned aspects, extracting viewpoints for each aspect, and summarizing these viewpoints into a concise overall assessment. This process also involves refining the original language for clarity and brevity. The detailed processing prompt is provided in Table 10, with the temperature set to zero to prevent the introduction of new information.

We first apply DeepSeek-v2.5 (Liu et al., 2024) to process the raw reviews. If a raw review is too ambiguous and the reformatted version has less than 40% word overlap with the original text, we apply GPT-4o to process this raw review. If the overlap remains below the threshold after this second pass, the sample is filtered out.

Criteria Analysis. Through our review process, we extract over 1000 user-mentioned aspects and analyze the most frequently referred ones. We organize these aspects into a hierarchical criteria structure, referring to existing evaluation works (Guan et al., 2021; Chhun et al., 2022) and literary studies (Halliwell, 1998; Herman, 2011). Specifically, we begin by analyzing the eight top-level aspects and use LLMs to identify potential sub-aspects. After further analysis and refinement, we establish our criteria structure (Table 9). Figure 4 shows the distributions of these aspects. Additional discussion can be found in §A. Among the top-level aspects, some focus on objective qualities of the novel (i.e., plot & structure, characters, writing & language, world-building & setting, and themes), while others capture more subjective reader experiences (i.e., emotional impact, overall enjoyment & engagement, and expectation fulfillment).

Review Organization. After criteria analysis, we organize the extracted viewpoints by grouping them under the same criteria as their corresponding critiques. For example, as shown in Figure 2, the separate viewpoints for “plot development” and “plot coherence” are listed under “Plot & Structure”, forming the corresponding critique.

![Our data construction process (§3).](page_172_120_1097_312.png)

2We also propose an anonymized test set for evaluating future LLMs, which can significantly mitigate data contamination concerns (Wang et al., 2024). Details are in §B.3.
<table>
  <tr>
    <th>Dataset</th>
    <th># Stories</th>
    <th># Samples</th>
    <th># AVG Length</th>
    <th>Review</th>
    <th>Criteria</th>
  </tr>
  <tr>
    <td>OpenMEVA (Guan et al., 2021)</td>
    <td>2,000</td>
    <td>2.0K</td>
    <td>143 tokens</td>
    <td>-</td>
    <td>PLOT(CHA, WRI, EMOT(EMO), ENJ(ENG), EXP(REL))</td>
  </tr>
  <tr>
    <td>HANNA (Chitran et al., 2022)</td>
    <td>1,956</td>
    <td>19.0K</td>
    <td>375 tokens</td>
    <td>-</td>
    <td>PLOT(COH,SUR), WOR(COMM, EMO(EMO), ENJ(ENG), EXP(REL))</td>
  </tr>
  <tr>
    <td>StoryER-Rate (Chen et al., 2022)</td>
    <td>12,669</td>
    <td>45.9K</td>
    <td>495 tokens</td>
    <td>Overall</td>
    <td>PLOT(STR, CHA(CHAR), WRI(STRY), EXP(GENRE))</td>
  </tr>
  <tr>
    <td>Xie (Xie et al., 2023)</td>
    <td>200</td>
    <td>2.1K</td>
    <td>79 tokens</td>
    <td>-</td>
    <td>PLOT(COH, WRI(EL), WOR(COMM), ENJ(INT), EXP(REL))</td>
  </tr>
  <tr>
    <td>Per-DOC (Wang et al., 2024)</td>
    <td>596</td>
    <td>8.9K</td>
    <td>2.5K tokens</td>
    <td>Overall</td>
    <td>PLOT(ADAP,SUR,END), CHA, ENJ(INT)</td>
  </tr>
  <tr>
    <td><b>LongStoryEval</b></td>
    <td>600</td>
    <td>340K</td>
    <td>121K tokens</td>
    <td>Aspect-Guided</td>
    <td><b>PLOT</b>, CHA, WRI, THE, WOR, EMO, ENJ, EXP</td>
  </tr>
</table>

Table 1: LongStoryEval and existing story evaluation datasets. “Criteria” denotes the considered aspects – PLOT: plot & structure, CHA: characters, WRI: writing & language, THE: themes, WOR: world-building & setting, EMO: emotional impact, ENJ: enjoyment & engagement, EXP: expectation fulfillment. Abbreviations of existing datasets’ aspects are detailed in Table 3. Our criteria structure encompasses the previous inconsistent criteria, with overlapping top-level aspects shown in bold and covered sub-aspects underlined.

![Average score distribution and book length distribution in LongStoryEval.](page_184_624_377_377.png)

Figure 3: Average score distribution and book length distribution in LongStoryEval.

Statistics and Comparison. Our benchmark dataset includes: (1) 600 newly published books with their metadata, including titles, genres, and premises; (2) An average rating score for each book, along with its rating distribution from 1-5 stars; (3) Multiple reviews for each book, organized as aspect-guided critiques, an overall assessment, and a final rating score; (4) Reviewer metadata, including rating score distribution and self-introduction (if available).

Compared to existing story evaluation benchmarks (Table 1), LongStoryEval is the first to focus specifically on lengthy stories, with an average length of 121K tokens and a maximum of 397K. The length distribution is shown in Figure 3 (b). Unlike previous benchmarks, which rely on annotators to evaluate stories based on limited predefined criteria3, we collect real-world reader reviews and derive evaluation criteria through systematic analysis. This data-driven approach ensures that the criteria structure better reflects actual reader standards. As shown in Table 1, our evaluation criteria structure covers all key aspects identified in prior works. Additionally, our organized reviews demonstrate a multi-aspect-guided reasoning process for the evaluation score, enhancing interpretability and providing greater granularity for training story evaluation models.

![The distribution of the evaluation aspects in readers’ reviews.](page_1012_624_377_377.png)

Figure 4: The distribution 4 of the evaluation aspects in readers’ reviews.

structure covers all key aspects identified in prior works. Additionally, our organized reviews demonstrate a multi-aspect-guided reasoning process for the evaluation score, enhancing interpretability and providing greater granularity for training story evaluation models.

4 Method

Given a book-length story consisting of several chapters \( \{c_1, ..., c_n\} \) and an evaluation criteria list \( \{a_1, ..., a_m\} \), aspect-specific critique/review \( r_i \) and aspect-specific score \( s_i \) will be generated for each \( a_i \). All critiques will then be summarized into an overall assessment \( R \), accompanied by an overall rating score \( S \). In this work, we explore and compare three methods for evaluating lengthy stories

3While some benchmarks allow user-defined criteria, this is typically on a very small scale.

4To avoid genre bias, this distribution includes equal books from each of the genres: Romance, Fantasy, Thriller, Mystery, Historical Fiction, Science Fiction, and Young Adult.
(§4.1; Figure 5): aggregation-based, incremental-updated, and summary-based evaluations. We then propose a specialized model that uses the efficient summary-based strategy, as detailed in §4.2.

4.1 Lengthy Story Evaluation Methods

Aggregation-Based Evaluation. As illustrated in Figure 5 (a), each chapter is evaluated individually, and the chapter-level scores are subsequently averaged as the book-level score. These chapter-level scores can refer to either aspect-specific scores or the overall score. For each chapter’s evaluation, we provide the LLMs with the book’s metadata, the current chapter, and a plot summary of previous chapters to ensure contextual awareness.

Incremental-Updated Evaluation. This method assumes that a reader’s opinion of a book evolves during the reading process. As illustrated in Figure 5 (b), the model updates evaluations (both reviews and scores) progressively as it processes each chapter. At each step, the model receives the summary and evaluations from the previous chapters, processes the current chapter, and updates the reviews and scores. This process continues iteratively until the final chapter is reached.

Summary-Based Evaluation. A more intuitive approach involves reading the entire book first to form an overall impression before evaluation. A comprehensive overview of a lengthy story should include key aspects such as plot, characters, and writing style, similar to Wikipedia-style novel introductions. As illustrated in Figure 4, these elements are also frequently mentioned by real readers. Therefore, we condense the story into: plot summary, character analysis, and writing excerpts (selected paragraphs to reflect the writing style). These elements can effectively capture additional aspects such as themes and overall enjoyment.

Our summary is generated through incremental summarization, which aligns better with human preferences (Chang et al., 2024). As shown in Figure 5 (c), at each summarization step, we provide the current chapter and the previous summary (plot and character), generate a summary of the current chapter, and update the overall summary. Detailed explanations and prompts are displayed in §C.

4.2 Proposed Model: NovelCritique

As mentioned in §2 and confirmed by our experiments in §5.3, closed-source methods, while outperforming open-source alternatives, still lack consistency and exhibit poor alignment with human evaluations. To address these limitations, we introduce NovelCritique, a specialized model for evaluating long-form stories. NovelCritique follows the summary-based structure, which can achieve comparable results to aggregation-based evaluations (Table 2) while being much more efficient. The only deviation from the framework in Figure 5 (c) is that our real-reader reviews lack aspect-specific scores. For each training sample with a criteria list \( \{a_1, ..., a_m\} \), the model outputs consist of: aspect-specific critiques, an overall assessment, and an overall score. During inference, when an aspect-specific score is needed, the model applies the generated critiques of this aspect, summarizes them into an overall assessment, and produces a score that becomes the aspect-specific score.

Review Bias Mitigation. We address the selection bias in providing reviews, as users who give moderate ratings are less likely to write reviews (§B.2). Training on all collected reviews would create bias in model predictions. To counter this,

![Overview of three evaluation methods (§4.1). Here we illustrate the complete inputs and outputs in the summary-based structure. The other two methods similarly incorporate metadata and specified aspects as inputs to generate aspect-specific and overall evaluations (reviews and scores). For these two types, each chapter’s evaluation includes the current content and previous summaries as input, ensuring contextual awareness. Detailed prompt appears in Table 14.](page_820_120_601_496.png)

Figure 5: Overview of three evaluation methods (§4.1). Here we illustrate the complete inputs and outputs in the summary-based structure. The other two methods similarly incorporate metadata and specified aspects as inputs to generate aspect-specific and overall evaluations (reviews and scores). For these two types, each chapter’s evaluation includes the current content and previous summaries as input, ensuring contextual awareness. Detailed prompt appears in Table 14.
we filter training reviews across all rating levels (1-5) to match each book’s rating distribution. For instance, if a book has mostly 3-star ratings, but 5-star reviews are disproportionately overrepresented due to this bias, we filter out extra 5-star reviews based on the book’s rating distribution.

Rating Score Normalization\footnote{This normalization is applied only to training samples, while for evaluation, we use the average of all original ratings.}. Users’ rating standards vary significantly, with some being strict and others more moderate. To normalize the ratings (Shalabi and Shaaban, 2006), we adjust each rating score \( S \) as follows:

\[
S' = \frac{S - \mu_u}{\sigma_u} \times \sigma_{plat} + \mu_{plat},
\]

where \( \mu_u, \sigma_u \) denote the mean and standard deviation of the current user’s rating distribution, and \( \mu_{plat}, \sigma_{plat} \) represent platform-wide statistics.

Training. We train NovelCritique via instruction tuning (Ouyang et al., 2022) using cross-entropy loss. The instruction details are provided in Table 13. We select Llama 3.1-8B (Dubey et al., 2024) as the base model due to its strong performance among open-source alternatives. The training loss of each sample is calculated as:

\[
-\log P(r_{i \leq m}, R, S'|X_{Instruct.Metadata.Sum.Excerpts}, a_{i \leq m}).
\]

5 Experiments

We conduct experiments using the LongStoryEval dataset. From the 600 books, we designate 150 books as the test set (Tables 15-16), ensuring diversity across genres and score distributions.

5.1 Training Setup of NovelCritique

Our training set consists of the remaining 450 books and 176K filtered reviews (after mitigating review bias). Input summaries are generated via incremental summarization (\S C) using the GPT-4o model. We finetune the base Llama 3.1-8B model for three epochs with a learning rate of \( 1e^{-5} \) and a batch size of 32. The LoRA parameters (Hu et al., 2022) are configured as \( r = 64 \) and \( alpha = 16 \). The training was conducted on four A6000 GPUs, taking approximately 125 hours.

5.2 Baselines

LLM-Based Lengthy Story Evaluation. We conduct experiments to compare the effectiveness of three evaluation methods (detailed in \S 4.1): aggregation-based, incremental-updated, and summary-based evaluations. Building on research showing that LLM-based evaluation benefits from detailed criteria definitions (Chhun et al., 2024), we establish evaluation criteria for eight top-level aspects based on literature standards, as detailed in \S A.2. The evaluation prompts are provided in Table 14.

Backbone LLMs. We experiment with five backbone models: GPT-4o, GPT-4o-mini, DeepSeek-v2.5 (Liu et al., 2024), Mixtral 8x7B-Instruct (Jiang et al., 2024a), Llama 3.1-70B-Instruct, and Llama 3.1-8B-Instruct (Dubey et al., 2024). As shown in Table 5, all these models were trained on data pre-dating year 2024 to ensure fairness in evaluation. To improve stability, we apply greedy decoding for open-source models and set the temperature to zero for closed-source models. Since closed-source LLMs still produce variations (they do not use pure greedy decoding), we report the average rating score across five generations.

5.3 Main Results

Following prior evaluation works, we use Kendall-Tau correlations (Kendall, 1938) to measure agreement between human evaluations and model-predicted scores. For each evaluation method, we generate aspect-specific scores and the overall scores, then compute their correlation with human-assigned ratings (i.e., the average rating of each book). As shown in Table 2, NovelCritique demonstrates the highest correlation with human ratings across both overall scores and most evaluation aspects, with the exception of word-building and setting. This exception is understandable, as this aspect often requires a holistic understanding of the entire book, making it particularly challenging for summary-based models. Nevertheless, NovelCritique still outperforms other summary-based methods in this aspect.

The primary issue with closed-source LLMs is their inconsistency. Even with temperature=0 and low top-p settings, the results exhibit significant variability. This inconsistency is likely due to the long context windows, which increase the likelihood for models to focus on uncertain or less relevant story elements. Notably, this inconsistency issue is less pronounced in short story evaluation tasks (Chhun et al., 2024). To mitigate this, we average scores across five evaluation runs. While this improves stability, it also significantly increases
<table>
  <tr>
    <th></th>
    <th>PLOT</th>
    <th>CHA</th>
    <th>WRI</th>
    <th>WOR</th>
    <th>THE</th>
    <th>EMO</th>
    <th>ENJ</th>
    <th>EXP</th>
    <th>Overall</th>
  </tr>
  <tr>
    <th rowspan="2">One-Pass (Subset)</th>
    <td>GPT-4o</td>
    <td>3.3</td>
    <td>4.1</td>
    <td>7.9</td>
    <td>0.8</td>
    <td>3.3</td>
    <td>-1.2</td>
    <td>-3.2</td>
    <td>8.4</td>
    <td>5.5</td>
  </tr>
  <tr>
    <td>DeepSeek-v2.5</td>
    <td>4.4</td>
    <td>3.5</td>
    <td>4.8</td>
    <td>-0.9</td>
    <td>3.3</td>
    <td>-1.1</td>
    <td>-1.3</td>
    <td>9.4</td>
    <td>4.8</td>
  </tr>
  <tr>
    <th rowspan="6">Aggregation-Based</th>
    <td>GPT-4o</td>
    <td>14.3</td>
    <td>16.7</td>
    <td>10.2</td>
    <td>7.9</td>
    <td>10.4</td>
    <td>9.7</td>
    <td>9.1</td>
    <td>14.1</td>
    <td>15.2</td>
  </tr>
  <tr>
    <td>DeepSeek-v2.5</td>
    <td>17.2</td>
    <td>15.8</td>
    <td>7.0</td>
    <td>7.1</td>
    <td>11.0</td>
    <td>14.2</td>
    <td>11.1</td>
    <td>16.7</td>
    <td>15.1</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>14.2</td>
    <td>17.2</td>
    <td>7.2</td>
    <td>4.4</td>
    <td>9.5</td>
    <td>8.9</td>
    <td>8.1</td>
    <td>15.1</td>
    <td>12.3</td>
  </tr>
  <tr>
    <td>Llama 3.1-70B</td>
    <td>19.6</td>
    <td>13.8</td>
    <td>2.3</td>
    <td>13.8</td>
    <td>13.4</td>
    <td>7.7</td>
    <td>11.5</td>
    <td>18.9</td>
    <td>13.8</td>
  </tr>
  <tr>
    <td>Llama 3.1-8B</td>
    <td>15.5</td>
    <td>8.5</td>
    <td>-1.4</td>
    <td>2.8</td>
    <td>12.3</td>
    <td>7.5</td>
    <td>7.0</td>
    <td>13.7</td>
    <td>11.6</td>
  </tr>
  <tr>
    <td>Mixtral 8×7B</td>
    <td>9.5</td>
    <td>4.0</td>
    <td>2.5</td>
    <td>-0.2</td>
    <td>8.9</td>
    <td>9.5</td>
    <td>10.2</td>
    <td>6.8</td>
    <td>9.0</td>
  </tr>
  <tr>
    <th rowspan="6">Incremental-Updated</th>
    <td>GPT-4o</td>
    <td>8.0</td>
    <td>9.1</td>
    <td>9.1</td>
    <td>11.7</td>
    <td>10.5</td>
    <td>12.3</td>
    <td>12.1</td>
    <td>11.5</td>
    <td>10.9</td>
  </tr>
  <tr>
    <td>DeepSeek-v2.5</td>
    <td>8.9</td>
    <td>12.2</td>
    <td>9.0</td>
    <td>8.6</td>
    <td>12.5</td>
    <td>12.3</td>
    <td>6.6</td>
    <td>12.2</td>
    <td>11.6</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>7.9</td>
    <td>10.8</td>
    <td>6.7</td>
    <td>7.4</td>
    <td>8.5</td>
    <td>11.6</td>
    <td>8.5</td>
    <td>10.7</td>
    <td>9.3</td>
  </tr>
  <tr>
    <td>Llama 3.1-70B</td>
    <td>9.3</td>
    <td>13.3</td>
    <td>4.1</td>
    <td>1.7</td>
    <td>8.7</td>
    <td>4.9</td>
    <td>4.6</td>
    <td>6.1</td>
    <td>9.9</td>
  </tr>
  <tr>
    <td>Llama 3.1-8B</td>
    <td>7.0</td>
    <td>7.1</td>
    <td>4.4</td>
    <td>2.5</td>
    <td>1.9</td>
    <td>8.0</td>
    <td>7.8</td>
    <td>5.1</td>
    <td>6.7</td>
  </tr>
  <tr>
    <td>Mixtral 8×7B</td>
    <td>4.2</td>
    <td>10.8</td>
    <td>4.4</td>
    <td>6.6</td>
    <td>5.8</td>
    <td>2.3</td>
    <td>5.8</td>
    <td>2.6</td>
    <td>4.2</td>
  </tr>
  <tr>
    <th rowspan="6">Summary-Based</th>
    <td>GPT-4o</td>
    <td>15.3</td>
    <td>17.8</td>
    <td>4.5</td>
    <td>5.0</td>
    <td>7.2</td>
    <td>12.6</td>
    <td>11.8</td>
    <td>14.0</td>
    <td>13.4</td>
  </tr>
  <tr>
    <td>DeepSeek-v2.5</td>
    <td>13.4</td>
    <td>12.2</td>
    <td>1.8</td>
    <td>-3.8</td>
    <td>7.1</td>
    <td>8.9</td>
    <td>13.2</td>
    <td>15.1</td>
    <td>14.4</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>8.7</td>
    <td>7.5</td>
    <td>5.4</td>
    <td>4.8</td>
    <td>11.1</td>
    <td>11.6</td>
    <td>8.3</td>
    <td>7.9</td>
    <td>9.7</td>
  </tr>
  <tr>
    <td>Llama 3.1-70B</td>
    <td>11.2</td>
    <td>10.8</td>
    <td>-1.6</td>
    <td>5.3</td>
    <td>12.4</td>
    <td>9.2</td>
    <td>11.4</td>
    <td>14.5</td>
    <td>13.0</td>
  </tr>
  <tr>
    <td>Llama 3.1-8B</td>
    <td>10.4</td>
    <td>14.1</td>
    <td>4.9</td>
    <td>9.1</td>
    <td>9.6</td>
    <td>15.3</td>
    <td>14.5</td>
    <td>12.3</td>
    <td>12.4</td>
  </tr>
  <tr>
    <td>Mixtral 8×7B</td>
    <td>7.8</td>
    <td>7.4</td>
    <td>7.1</td>
    <td>-0.5</td>
    <td>-4.0</td>
    <td>5.6</td>
    <td>9.4</td>
    <td>6.7</td>
    <td>8.3</td>
  </tr>
  <tr>
    <td>NovelCritique-8B</td>
    <td><b>21.4</b></td>
    <td><b>20.8</b></td>
    <td><b>15.1</b></td>
    <td><b>11.2</b></td>
    <td><b>18.5</b></td>
    <td><b>21.1</b></td>
    <td><b>22.8</b></td>
    <td><b>20.5</b></td>
    <td><b>20.1</b></td>
  </tr>
</table>

Table 2: The system-level Kendall correlations between the human-assigned scores and model-generated evaluations. We report the correlation between aspect-specific scores and the overall score.

computational overhead. The cost is particularly high for incremental-updated and aggregation-based methods as they require processing the entire book context for each evaluation run.

5.4 Analysis

Which evaluation aspects mostly affect the final rating? As shown in Table 2, for objective aspects, plot and characters are the most influential. Themes have some impact but appear to be secondary considerations for most readers. World-building and writing quality are the least influential aspects, likely because most stories show similar levels in these areas (except for particularly exceptional books). For subjective aspects, emotional impact, enjoyment & engagement, and expectation fulfillment all play critical roles.

Which long story evaluation strategy is more effective? Prior to comparing our proposed methods, we assess whether existing models can effectively evaluate entire books in a single pass. We test a subset of books within the 128K token context. The one-pass results (lines 1-2 in Table 2) reveal a poor correlation with human ratings. Even when prompted to generate summaries first, these models often produce generic critiques that fail to capture the nuances of specific stories.

Regarding the three methods discussed in §4.1, aggregation-based and summary-based approaches demonstrate superior performance. While the incremental-updated method seems promising in theory, it faces two key limitations. First, it requires additional instructions, forcing models to both comprehend the current segment and consider its impact on previous evaluations. This problem is especially serious for less powerful models like Llama 3.1-8B and Mixtral 8×7B. Second, it suffers from inconsistency that accumulates over multiple updates. Given these constraints, we recommend using aggregation-based and summary-based methods until significant improvements in LLM capabilities emerge.

The main advantage of aggregation-based methods is their ability to access all details of a long story, enabling more thorough measurement and scoring. Summary-based models, on the other hand, offer two main benefits: (1) Efficiency—they require less time and resources per evaluation. By generating a single high-quality summary, we can reuse it across multiple evaluations, leading to more stable and robust results. (2) Potential for early evaluation—before completing the entire story, authors can receive reviews and scores based on their plot structure, character design, and writing samples. These early assessments can closely estimate judgments of their finished work.
<table>
  <tr>
    <th>GPT-4o</th>
    <th>Llama 3.1-8B</th>
    <th>NovelCritique-8B</th>
  </tr>
  <tr>
    <td><b>Plot and Structure:</b><br>The plot of “Remedial Magic” is intricate. The setting of Crenshaw, a town reliant on magic for survival amid environmental crises, provides a unique backdrop that is both engaging and relevant. The narrative is driven by the characters’ personal quests and the overarching mystery of the Congress of Witches. The story is well-paced and engaging, though with well-paced storytelling, though at times it may feel overly complex due to the numerous subplots and arcs. The twists, such as the manipulation of memories and the morally ambiguous actions of characters like Prospero, add depth and intrigue. However, the story is occasionally interrupted by marriages and memory alterations, may feel unsatisfying to some readers as it leaves certain moral questions unresolved and the characters’ autonomy compromised. (3.5)</td>
    <td><b>Plot and Structure:</b><br>The plot of “Remedial Magic” is engaging and complex, with multiple timelines that intersect and converge. The narrative effectively explores the consequences of magical abilities and the moral dilemmas faced by the characters. However, the pacing could be improved at times, with some sections feeling rushed or convenient. The plot twists are generally well-executed, but some may feel predictable to readers familiar with the fantasy genre. The resolutions are satisfying, but the story feels somewhat abrupt, leaving some questions unanswered. Overall, the plot is well-structured and engaging, but could benefit from further refinement. (3.5)</td>
    <td><b>Plot and Structure:</b><br>The story revolves around Ellie, a librarian who discovers she is a witch and falls in love with a powerful witch named Prospero. The plot is filled with magical college, mysterious disappearances, and a prophecy that Ellie is the key to saving Crenshaw, a magical land facing environmental crises. However, the pacing is somewhat uneven, which might deter some readers. The narrative jumps between multiple characters and their perspectives, which can be confusing at times. The story also includes a car accident that serves as a catalyst for Ellie’s discovery of her magical abilities, which feels somewhat contrived. (3.5)</td>
  </tr>
  <tr>
    <td><b>Characters:</b><br>The characters in the story are well-defined with distinctive arcs, particularly Ellie and Prospero. A redemption arc for the librarian is a key feature in a magical society with a compelling journey. Prospero presents a nuanced character whose motives are complex and sometimes ethically dubious. The diversity in characters, including the LGBT representation, is a strength. However, some secondary characters, like Dan and Maggie, who interact with the main plot, could benefit from more substantial development to enhance their impact with readers. (4.0)</td>
    <td><b>Characters:</b><br>The characters are well-developed and complex, with distinct personalities and motivations. Ellie, Maggie, and Prospero are particularly well-written, with nuanced and relatable character arcs. Dan and Sondre are also well-developed. Craig and Hestia are somewhat underdeveloped, but their relationships with each other are well-explored, and the characters’ relationships with each other are well-explored, and the romance between Ellie and Prospero is sweet and tender. (4.0)</td>
    <td><b>Characters:</b><br>Ellie’s character development is somewhat lacking. She transitions from a passive librarian to a powerful witch but fails to fully embrace her powers. Her innermost relationships, and her magic becomes crucial for survival, but she becomes a victim of Prospero’s manipulations. Ellie’s journey feels incomplete, and her growth is overshadowed by the story’s focus on Maggie. Maggie’s relationship with her son drives her actions. Her adaptability and magical prowess grow, but she is forced into morally complex decisions. Maggie’s sacrifice for her son is poignant, but her character feels somewhat one-dimensional compared to others. (3.0)</td>
  </tr>
  <tr>
    <td><b>Expectation Fulfillment:</b><br>The story largely fulfills expectations based on its premise and genres, offering an innovative mix of fantasy romance with LGBT representation. However, the promise of blending a magical university college with a central plot suggests which might leave some expecting this element disappointed. Overall, it delivers on suspense and magical intrigue. (3.0)</td>
    <td><b>Expectation Fulfillment:</b><br>The story meets the readers’ expectations based on the premise and genres. The fantasy and romance elements are well-executed, and the story raises important questions about the nature of power and mortality. However, the story could benefit from more complexity to meet reader expectations. (3.5)</td>
    <td><b>Expectation Fulfillment:</b><br>I was initially drawn by its premise. The idea of a librarian discovering she’s a witch and falling in love with a powerful witch from another world was intriguing. However, the execution fell short of my expectations. The story felt overly long and repetitive, with too much focus on the characters’ daily lives and not enough on the magical world and the central plot. (2.5)</td>
  </tr>
</table>

Figure 6: Critiques for “Remedial Magic” (AVG human rating: 2.9). The generated weaknesses are colored in red.

![Ablation studies on NovelCritique.](page_1017_1342_388_246.png)

Figure 7: Ablation studies on NovelCritique.

Do detailed summaries improve summary-based evaluation? As displayed in Table 7, replacing the overall summary with detailed chapter-guided summaries leads to a slight increase. We suggest that a more detailed summary can better reflect a story’s quality. However, longer summaries require more memory and involve more complex reasoning. It is important to find a balance between the level of detail and length, which could be explored in future works.

Is high-quality summaries necessary for summary-based evaluation? To assess the importance of summary quality, we replace GPT-4o-generated summaries with those produced by GPT-4o-mini (around 0.03% prices of GPT-4o).

The results (Table 7) show no significant decline in performance. This suggests a cost-efficient approach: generating summaries with GPT-4o-mini and then conducting evaluations using more advanced models like DeepSeek-v2.5 or NovelCritique.

Ablation Studies on NovelCritique. We verify the effectiveness of our designs in NovelCritique, including raw review organization, review bias mitigation, and rating score normalization. The results in Figure 7 demonstrate their effectiveness.

5.5 Qualitative Results

In Figure 6, we present evaluation results from different models. We find that many models tend to focus more on the story’s strengths, offering only limited commentary on its weaknesses. This tendency will also lead existing models to assign good scores for stories that humans consider poor. While we have tried to address this by asking models to provide advantages and disadvantages, this approach causes models to become excessively critical. Current models still struggle to generate nuanced critiques that closely align with human preferences, particularly in reflecting detailed evaluations. Critiques for a well-written story are displayed in §G.
6 Conclusion

This work explores the underexplored problem of evaluating book-length stories, addressing three core questions: (1) What evaluation aspects matter most to real readers? (2) What are the most effective methods for evaluating lengthy stories? (3) What challenges arise in LLM-based evaluation and how can they be addressed? To tackle these questions, we introduce LongStoryEval, a large-scale dataset comprising average rating scores and well-formatted reviews. Through analysis of these reviews, we propose a criteria structure that reflects human standards. Our experiments reveal the critical aspects influencing final ratings, and demonstrate the effectiveness of aggregation- and summary-based evaluations. While aggregation-based methods provide detailed and comprehensive evaluations, summary-based methods excel in efficiency and offer potential for early-stage evaluations. Acknowledging the limitations of existing LLMs, such as inconsistency and imperfect alignment with human preferences, we propose NovelCritique, an 8B model that exhibits improved correlation with human evaluations. We hope this work inspires further research into evaluating lengthy stories and fosters advancements in both lengthy story evaluation and generation.

Limitations

This work employs critiques and score generation for evaluation, which can be prone to inconsistencies. To mitigate this, we average results across multiple runs. Future work could explore alternative mitigation methods, such as employing pairwise comparison instead of direct scoring. Comparisons often yield more stable results but come with higher computational costs, underscoring the need for more efficient comparison strategies. Sampling-based approaches (Xu et al., 2024) also present a promising direction for generating more reliable scores.

Our current evaluation emphasizes general assessment over personalized preferences. However, since our dataset contains anonymized reviewer information, future studies could explore personalized evaluation approaches tailored to individual tastes and reading habits.

Ethical Problems

We acknowledge and strictly adhere to the Code of Ethics and Professional Conduct throughout this research. The potential ethical concerns are addressed as follows:

Data Source. Our review data comes from publicly available content on the Goodreads website 6, which is accessible to anyone. Following previous works (Wan et al., 2019; Wan and McAuley, 2018), we anonymize user IDs and review IDs to protect personal information. To mitigate the potential dissemination of harmful content in the raw reviews, we will only release our processed versions of reviews. For the books in our dataset, we collect all metadata from public Goodreads content and purchase electronic copies of the books. Considering copyright issues, only the book-level summaries will be released (Chang et al., 2024), while full content remains accessible through publicly available titles and author information (Tables 15-16).

Copyrights. As discussed before, we will only release processed versions to avoid potential ethical and copyright issues. Our data collection from public resources is for academic use only. To prevent commercial use, we will release our dataset under highly restrictive permissions that limit its use exclusively to academic research.

Acknowledgements

We thank all reviewers for their insightful comments and suggestions. This work was partially supported by the Beijing Natural Science Foundation (No. L233008).

References

Chimamanda Ngozi Adichie. 2009. The danger of a single story. TED.

Sara Ahmed. 2013. The cultural politics of emotion. Routledge.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005, pages 65–72. Association for Computational Linguistics.

James Scott Bell. 2004. Plot & structure: Techniques and exercises for crafting a plot that grips readers from start to finish. Writer’s Digest Books.

Christopher Booker. 2004. The seven basic plots: Why we tell stories. A&C Black.

6https://www.goodreads.com
Wayne C Booth. 1983. The rhetoric of fiction. University of Chicago Press.

Peter Brooks. 1992. Reading for the plot: Design and intention in narrative. Harvard University Press.

Janet Burroway, Elizabeth Stuckey-French, and Ned Stuckey-French. 2019. Writing fiction: A guide to narrative craft. University of Chicago Press.

Joseph Campbell. 2008. The hero with a thousand faces, volume 17. New World Library.

John G Cawelt. 2014. Adventure, mystery, and romance. University of Chicago Press.

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024. ChatEval: Towards better llm-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. Boookscore: A systematic exploration of book-length summarization in the era of llms. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Seymour Benjamin Chatman and Seymour Chatman. 1980. Story and discourse: Narrative structure in fiction and film. Cornell University Press.

Hong Chen, Duc Minh Vo, Hiroya Takamura, Yusuke Miyao, and Hideki Nakayama. 2022. Storyer: Automatic story evaluation via ranking, rating and reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1739–1753. Association for Computational Linguistics.

Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: An empirical study. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 - Findings, Nusa Dua, Bali, November 1-4, 2023, pages 361–374. Association for Computational Linguistics.

Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chloé Clavel. 2022. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 5794–5836. International Committee on Computational Linguistics.

Cyril Chhun, Fabian M. Suchanek, and Chloé Clavel. 2024. Do language models enjoy their own stories? prompting large language models for automatic story evaluation. Trans. Assoc. Comput. Linguistics, 12:1122–1142.

Paul Coulton, Joseph Lindley, Miriam Sturdee, and Michael Stead. 2017. Design fiction as world building. In Proceedings of the 3nd Biennial Research Through Design Conference, pages 1–16.

Mihaly Csikszentmihalyi. 1997. Flow and the psychology of discovery and invention. HarperPerennial, New York, 39:1–16.

Jerome De Groot. 2009. The historical novel. Routledge.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schellen, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Peter Elbow. 1998. Writing without teachers. Oxford University Press, USA.

Syd Field. 2005. Screenplay: The foundations of screenwriting. Delta.

Edward Morgan Forster. 1927. Aspects of the novel. Harcourt, Brace.

Gustav Freytag. 1895. Technique of the drama: An exposition of dramatic composition and art. S. Griggs.

Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2024. LLM-based nlg evaluation: Current status and challenges. Preprint, arXiv:2402.01383.

Gérard Genette. 1980. Narrative discourse: An essay in method. Cornell University Press.

Sarik Ghazarian, Zixi Liu, Akash SM, Ralph M. Weischedel, Aram Galstyan, and Nanyun Peng. 2021. Plot-guided adversarial example construction for evaluating open-domain story generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 4334–4344. Association for Computational Linguistics.

Peiyuan Gong and Jiaxin Mao. 2023. Coascore: Chain-of-aspects prompting for nlg evaluation.

Jian Guan and Minlie Huang. 2020. UNION: An unreferenced metric for evaluating open-ended story generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9157–9166. Association for Computational Linguistics.

Jian Guan, Zhixin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, and Minlie Huang. 2021. Openmeva: A benchmark for evaluating open-ended story generation metrics. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 6394–6407. Association for Computational Linguistics.
Stephen Halliwell. 1998. Aristotle’s poetics. University of Chicago Press.

David Herman. 2011. Basic elements of narrative. John Wiley & Sons.

Patrick Colm Hogan. 2003. The mind and its stories: Narrative universals and human emotion. Cambridge University Press.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

Henry James. 1884. The art of fiction. Longman’s magazine, 1882-1905, 4(23):502–521.

Fredric Jameson. 2013. The political unconscious: Narrative as a socially symbolic act. Routledge.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamberd, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024a. Mixtral of experts. arXiv preprint arXiv:2401.04088.

Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhui Chen. 2024b. Tigerscore: Towards building explainable metric for all text generation tasks. Trans. Mach. Learn. Res., 2024.

Suzanne Keen. 2007. Empathy and the novel. Oxford University Press.

Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93.

Joonghoon Kim, Sangmin Lee, Seung Hun Han, Saeran Park, Jiyoung Lee, Kiyoon Jeong, and Pilsung Kang. 2023. Which is better? exploring prompting strategy for llm-based metrics. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, Eval4NLP 2023, Bali, Indonesia, November 1, 2023, pages 164–183. Association for Computational Linguistics.

Seungone Kim, Jamin Shin, Yejin Choi, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, ICLR 2024. Vienna, Austria, May 7-11, 2024. OpenReview.net.

Stephen King. 2000. On writing: A memoir of the craft. Pocket Books.

Yukyung Lee, Joonghoon Kim, Jaehee Kim, Hyowon Cho, and Pilsung Kang. 2024. Checkeval: Robust evaluation framework using large language model via checklist. Preprint, arXiv:2403.18771.

Geoffrey N Leech and Mick Short. 2007. Style in fiction: A linguistic introduction to English fictional prose. Pearson Education.

Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023. Generative judge for evaluating alignment.

Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao, and Shuai Ma. 2024. Leveraging large language models for nlg evaluation: Advances and challenges. Preprint, arXiv:2401.07103.

Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, arXiv:2405.04434.

Aviya Maimon and Reut Tsarfaty. 2023. COHESENTIA: A novel benchmark of incremental versus holistic assessment of coherence in generated texts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5328–5343. Association for Computational Linguistics.

Nicole Matthews and Nickianne Moody. 2007. Judging a book by its cover: Fans, publishers, designers, and the marketing of fiction. Ashgate Publishing, Ltd.

Allyssa McCabe and Carole Peterson. 1984. What makes a good story. Journal of Psycholinguistic Research, 13(6):457–480.

Robert McKee. 1997. Substance, structure, style, and the principles of screenwriting. Alba Editorial.

Victor Nell. 1988. Lost in a book: The psychology of reading for pleasure. Yale University Press.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311–318. ACL.

James Phelan. 1996. Narrative as rhetoric: Technique, audiences, ethics, ideology. Ohio State University Press.
Steven Pinker. 2003. The language instinct: How the mind creates language. Penguin uK.

Shlomith Rimmon-Kenan. 2003. Narrative fiction: Contemporary poetics. Routledge.

Hanna-Riikka Roine. 2016. Imaginative, immersive and interactive engagements, the rhetoric of worldbuilding in contemporary speculative fiction.

Swarmadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2024. Branch-solve-merge improves large language model evaluation and generation.

Luai A. Al Shalabi and Zyad Shaaban. 2006. Normalization as a preprocessing engine for data mining and the approach of preference matrix. In 2006 International Conference on Dependability of Computer Systems (DepCoS-RELCOMEX 2006), 24-28 May 2006, Szklarska Poreba, Poland, pages 207–214. IEEE Computer Society.

John Skorupski. 1976. Symbol and theory. Cambridge University Press.

Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Large language models are inconsistent and biased evaluators. Preprint, arXiv:2405.01724.

Mengting Wan and Julian J. McAuley. 2018. Item recommendation on monotonic behavior chains. In Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018, pages 86–94. ACM.

Mengting Wan, Rishabh Misra, Ndapa Nakashole, and Julian J. McAuley. 2019. Fine-grained spoiler detection from large-scale review corpora. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28–August 2, 2019, Volume 1: Long Papers, pages 2605–2610. Association for Computational Linguistics.

Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuandong Tian. 2024. Learning personalized story evaluation. Preprint, arXiv:2310.03304.

James Wood. 2008. How fiction works. Jonathan Cape.

Wendy Wyatt. 2016. The ethics of trigger warnings. Teaching Ethics, 16(1).

Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023a. The next chapter: A study of large language models in storytelling. Preprint, arXiv:2301.09790.

Zhuohan Xie, Miao Li, Trevor Cohn, and Jey Lau. 2023b. Deltascore: Fine-grained story evaluation with perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5317–5331, Singapore.

Cong Xu, Gayathri Saranathan, Mahammad Parwez Alam, Arpit Shah, James Lim, Soon Yee Wong, Foltin Martin, and Suparna Bhattacharya. 2024. Data efficient evaluation of large language models and text-to-image models via adaptive sampling. Preprint, arXiv:2406.15527.

Dingyi Yang and Qin Jin. 2024. What makes a good story and how can we measure it? a comprehensive survey of story evaluation.

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 27263–27277.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.

Lisa Zunshine. 2006. Why we read fiction: Theory of mind and the novel. Ohio State University Press.

A Evaluation Criteria

A.1 Human Criteria

Table 9 presents our detailed criteria structure with raw aspects mapped to standardized categories. Below, we detail our analyzed criteria structure, including the aspect names, definitions, and their corresponding references:

Plot and Structure:
• Plot Development: Assess the progression of events within a story through their pacing, twists, conflicts, and resolutions, as mentioned in Bell (2004); Forster (1927); Brooks (1992).
• Structure: Assess the organization of events through exposition, rising action, climax, falling action, and resolution, examining their coherence and logical flow, as mentioned in Freytag (1895); Genette (1980).
• Ending: How well the ending reflects the consequences of the story’s main events and themes. A well-crafted ending provides closure or shows attractive potential, as mentioned in Rimmon-Kenan (2003); Phelan (1996).

Characters:
• Character Development: Examine how well characters develop throughout a story by examining their backstories, motivations, and character growth
arcs, as mentioned in McKee (1997); James (1884); Campbell (2008).

• Characterization: Assess the representation of the characters, ensuring appeal, realism, and reliability, as mentioned in Wood (2008); Burroway et al. (2019).

• Relationships: Assess the interactions and chemistry between the characters, examining how their relationships drive the plot forward and engage readers, as mentioned in James (1884); Chatman and Chatman (1980).

• Diversity: How effectively the story portrays a diverse cast of characters, including not only the protagonist but also secondary and side characters, mentioned in Hogan (2003); Adichie (2009).

Writing and Language:
• Writing Style: Whether the author’s choice of words, sentence structure, and paragraph organization conveys meaning effectively and creates an engaging atmosphere for readers, as mentioned in Leech and Short (2007); Skorupski (1976).
• Language: Assess the quality of vocabulary and syntax, focusing on issues like grammar and fluency, mentioned in Pinker (2003); Elbow (1998).
• Readability: Assess the clarity and readability of the entire story, as mentioned in Burroway et al. (2019); King (2000).

Themes:
• Exploration: How well the author explores clear topics and themes through the narrative, as mentioned in Forster (1927); Booth (1983).
• Depth: Measure the depth of the themes, such as their educational value and social commentary, as mentioned in Booker (2004); Jameson (2013).

World-Building and Setting:
• World-Building: Assess the realism and authenticity of the built world by examining its consistency and descriptive detail, as mentioned in Roine (2016); James (1884); Coulton et al. (2017).
• Setting: Evaluate the accuracy of historical, cultural, geographic, and technical settings, as discussed in De Groot (2009); James (1884). This might be more considered in historical stories.

Emotional Impact:
• Empathy: Evaluate how well the stories evoke emotional connections between readers and characters, as mentioned in McCabe and Peterson (1984); Keen (2007); Hogan (2003).
• Depth: Evaluate the stories’ ability to create deep emotional connections, as discussed in Ahmed (2013); Keen (2007).

Enjoyment and Engagement:
• Enjoyment: Evaluate how well a story creates enjoyment and interests, and builds anticipation for upcoming events, as mentioned in Nell (1988); Zunshine (2006).
• Engagement: Evaluate how well the story engages the readers through the reading process, as mentioned in Nell (1988); Zunshine (2006).

Expectation Fulfillment (Or Relevance):
• Genre: How well the story fulfills genre expectations for elements like romance, suspense, and other genre-specific features. As mentioned in Cawelt (2014); Phelan (1996).
• Premise: How well the story fulfills expectations based on the premise, as mentioned in Forster (1927); Field (2005).

Others:
• Originality (Or Creativity): Assess the story’s uniqueness and imaginative ideas, as mentioned in Csikszentmihalyi (1997).
• Content Warnings: Whether the story contains rude, unreasonable, or disrespectful elements, as mentioned in Wyatt (2016).
• Designment: Assess the design of the book cover (Matthews and Moody, 2007).
• Personal Bias: Assess the feelings influenced by very subjective factors, such as individual preferences and personal experiences (Wang et al., 2024).

A.2 Condensed Definitions
We condense the definition of the top-level aspects as follows, which can be applied in our evaluation prompts:

1. Plot and Structure (PLOT): Evaluate the plot development by examining pace, twists, conflicts, and their resolutions. Evaluate the story structure for coherence, logic, and complexity, paying attention to key elements like climax and ending.

2. Characters (CHA): Evaluate how well the characters are drawn, considering their development, characterization (including realism,
appeal, and relatability), relationships, and diversity.

3. Writing and Language (WRI): Evaluate the writing style’s ability to engage and captivate readers. Assess language quality by examining descriptions and dialogue. Measure the story’s clarity and readability.

4. World-Building and Setting (WOR): Evaluate the world-building and setting by assessing how detailed and well-described they are. Consider their authenticity, accuracy, or realism.

5. Themes (THE): Evaluate how well the themes are explored throughout the story and assess their depth.

6. Emotional Impact (EMO): Evaluate the story’s ability to evoke strong and deep emotional impact.

7. Enjoyment and Engagement (ENJ): Evaluate how engaging and enjoyable the story is for readers.

8. Expectation Fulfillment (EXP\textsuperscript{7}): Evaluate how effectively the story meets the readers’ expectations based on the premise and genres.

A.3 Criteria of Existing Benchmarks
The considered aspects of existing benchmarks and their abbreviation are listed in Table 3.

B Dataset: LongStoryeval

B.1 Review Processing
Figure 8 displays an example of our raw review processing, while the processing prompt is detailed in Table 10. Our dataset sampling shows high accuracy (96%), which is sufficient for our model training. Our experimental improvements demonstrate its quality, though further human refinement could be considered in the future.

B.2 Statistics of Review Bias
Review bias refers to the selection bias in providing reviews, where users who give moderate ratings are less likely to leave reviews. We evaluate the average ratio of review numbers among ratings for each rating scale. The statistics show: 5 stars (22%), 4 stars (19%), 3 stars (17%), 2 stars (22%), and 1 star (31%).

B.3 Contamination Issues
Contamination problems mean the datasets are exposed to LLMs. For contaminated cases, LLMs might show unfairly high performance (Chang et al., 2024; Wang et al., 2024). However, anonymization and summarization can largely mitigate these issues (Wang et al., 2024). Since our dataset is already summarized, we conduct additional anonymization by replacing identifiable elements such as character names and locations. This anonymized test set helps ensure fair evaluation of future LLMs. Additionally, we will release our processing code to facilitate the collection of newly published books.

B.4 Approximate Exploration of Significant Aspects
Since individual aspect ratings are unavailable in the ground truth data, we attempt to approximate aspect-specific ratings by calculating the average of ratings from these reviews covering each specific aspect. As the results shown in Table 4, plot, characters, and overall enjoyment show clear significance. However, this approximation is imperfect, as the scores are influenced by multiple mentioned aspects rather than representing a single aspect. Therefore, these findings should be considered as a reference, with the results in Table 2 offering more reliable evidence.

<table>
  <tr>
    <th>Dataset</th>
    <th>Aspects</th>
  </tr>
  <tr>
    <td>OpenMEVA (Guan et al., 2021)</td>
    <td>COH: coherence [consistency, causal and temporal relationship], CHA: character behavior, FLU: fluency [semantic repetition, paraphrases]</td>
  </tr>
  <tr>
    <td>HANNA (Chhun et al., 2022)</td>
    <td>COH: coherence, SUR: surprise, COM: complexity, EMP: empathy, ENG: engagement, REL: relevance.</td>
  </tr>
  <tr>
    <td>StoryER-Rate (Chen et al., 2022)</td>
    <td>STR: structure, CHAR: characterization [character shaping], STY: writing style, GENRE: genre expectations.</td>
  </tr>
  <tr>
    <td>Xie (Xie et al., 2023a)</td>
    <td>FLU: fluency, COH: coherence, COMM: commonsense, INT: interestingness, REL: relevance.</td>
  </tr>
  <tr>
    <td>Per-DOC (Wang et al., 2024)</td>
    <td>ADAP: plot adaptability, SUR: surprise, END: ending, CHA: characters, INT: interestingness.</td>
  </tr>
</table>

Table 3: Existing story evaluation benchmarks and their considered aspects.

<table>
  <tr>
    <th>PLOT</th>
    <th>CHA</th>
    <th>WRI</th>
    <th>WOR</th>
    <th>THE</th>
    <th>EMO</th>
    <th>ENJ</th>
    <th>EXP</th>
  </tr>
  <tr>
    <td>79.9</td>
    <td>83.2</td>
    <td>72.0</td>
    <td>66.4</td>
    <td>72.3</td>
    <td>74.0</td>
    <td>79.1</td>
    <td>72.4</td>
  </tr>
</table>

Table 4: Approximate correlation scores (\S B.4).

\footnotetext{7This can also be considered as the relevance to the “Inputs”(premise and genres).}
Raw Review

Review: I was so so excited to read reckless and it did not disappoint!! this was a masterpiece and I absolutely devoured it while powerless is all high stakes adventure but shared with me so much angst, and sweet moments and I think that was amazing! The focus on the characters just grew my connection to characters I already adored, and I feel like I could literally SEE Pae and Kai’s connection changing and growing with every chapter and the battles I was giggling out loud and kicking my feet, and the last second when I was on the edge of my seat because Kai, Paedyn or both of them were in some type of pickle...it was the best reading experience.

And somehow there was this suspense that grew the whole book and I just had to read, the days that I read this book those nights I went to sleep thinking about this world, and the characters and how they are so well written, and how she describes things had my heart sobbing, these characters and this world seriously came off the page for me and that’s magical

and the ending I don’t even want to talk about it because what was perfect!! I seriously didn’t predict that and I absolutely started sobbing like a baby and that’s how I know a book is a favorite. When I think about how much I love this book I literally get heart palpitations. It was insanely good and this series WILL live rent free in my mind until the third one 🥰

—songs that remind me of this series-favorite crime by Olivia Rodrigo=sad

Rating Score: 5 ★★★★★

Processed Review

- Characters:
  (Character Development)
    - The focus on the characters deepened my connection to characters I already adored.
  (Character Connection)
    - I could literally see Pae and Kai’s connection changing and growing with them.
- Writing & Language:
  (Writing Style and Descriptions)
    - Lauren’s writing is so beautiful, and the way she describes things had my heart sobbing. The characters and the story felt like they came off the page, which was magical.
- Emotional Impact:
  (Emotional Impact)
    - I was moved by the ending. It had me sobbing like a baby, which is how I know a book is a favorite.
- Enjoyment & Engagement:
  (Engagement)
    - The banter had me giggling out loud and kicking my feet, but five seconds later, I was on the edge of my seat because Kai, Paedyn, or both were in some type of pickle.
    - The suspense grew throughout the book, and I couldn’t stop reading. On the days I read this book, I went to sleep thinking about this world.
    - When I think about how much I love this book, I literally get heart palpitations. This series will live rent-free in my mind until the third book comes out.

Overall Assessment:
“Reckless” is a masterpiece that delivered everything I hoped for and more. It’s a perfect blend of exploration, angst, and sweet moments. It shows incredible character development, suspense, and emotional depth. Lauren’s writing is stunning, and the ending left me in tears. This book and series have left an indelible mark on me, and I can’t wait for the next installment.

Rating Score: 5 ★★★★★

Figure 8: Example of raw review processing. We identify user-mentioned aspects (shown in brackets), then extract viewpoints (underlined texts) for each aspect, and summarize these viewpoints into an overall assessment. These viewpoints will be organized based on the criteria structure shown in Table 9.

C Summarization Method

As mentioned in Section 4.1, in our summary-based methods, we apply the summaries generated through an incremental-update process. Specifically, we update both plot and character summarizations. The detailed prompts are shown in Table 11 and 12. For characters, we summarize their profiles and overall experiences. For plots, considering that non-linear narrative structures can affect readers’ reading experience, we instruct the model to maintain the original plot structure.

D Experimental Details

D.1 Knowledge Cutoff of Baseline Models.

As shown in Table 5, all of the baseline models are trained with data before 2024, not containing our dataset.

<table>
  <tr>
    <th>Models</th>
    <th>Knowledge Cutoff</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>October, 2023</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>October, 2023</td>
  </tr>
  <tr>
    <td>DeepSeek-v2.5 (Liu et al., 2024)</td>
    <td>November, 2023</td>
  </tr>
  <tr>
    <td>Mixtral 8×7B (Jiang et al., 2024a)</td>
    <td>December, 2023</td>
  </tr>
  <tr>
    <td>Llama 3.1-8B (Dubey et al., 2024)</td>
    <td>December, 2023</td>
  </tr>
</table>

Table 5: Knowledge cutoff of baseline models.

D.2 Additional Discussions

We find that during the incremental evaluation, if the LLMs generate positive opinions for the earlier paragraphs, then they are likely to maintain this tendentiousness. Even if there are problems in the latter chapters, they will not significantly change their assessment. The same pattern occurs in reverse.

E Ablation Studies

Criteria Definitions. To investigate the impact of detailed criteria definitions, we conducted experiments both with and without them. As shown in Table 6, providing definitions did not significantly improve performance for powerful models like GPT-4o and DeepSeek-v2.5. However, less powerful models such as Llama 3.1-8B and Mixtral-8×7B show marked improvement with detailed definitions. We attribute this difference to powerful models having already acquired a comprehensive understanding of these aspects through their training, while others have not.

Summary Quality. To explore how input summaries affect the summary-based evaluations, we conducted experiments with three types: detailed chapter-level summaries, an overall summary, and a lower-quality overall summary (generated by
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="8">PLOT CHA WRI WOR THE EMO ENJ EXP</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>w/ definitions</th>
    <th>w/o definitions</th>
    <th>w/ definitions</th>
    <th>w/o definitions</th>
    <th>w/ definitions</th>
    <th>w/o definitions</th>
    <th>w/ definitions</th>
    <th>w/o definitions</th>
    <th>w/ definitions</th>
    <th>w/o definitions</th>
    <th>w/ definitions</th>
    <th>w/o definitions</th>
    <th>w/ definitions</th>
    <th>w/o definitions</th>
  </tr>
  <tr>
    <th>GPT-4o</th>
    <td>15.3</td>
    <td>11.1</td>
    <td>17.8</td>
    <td>15.0</td>
    <td>4.5</td>
    <td>2.2</td>
    <td>5.0</td>
    <td>4.5</td>
    <td>7.2</td>
    <td>6.0</td>
    <td>12.6</td>
    <td>10.7</td>
    <td>11.8</td>
    <td>8.5</td>
    <td>14.0</td>
    <td>9.4</td>
    <td>13.4</td>
    <td>11.5</td>
  </tr>
  <tr>
    <th>DeepSeek-v2.5</th>
    <td>13.4</td>
    <td>16.8</td>
    <td>12.2</td>
    <td>13.7</td>
    <td>1.8</td>
    <td>0.8</td>
    <td>-3.8</td>
    <td>5.0</td>
    <td>7.1</td>
    <td>8.1</td>
    <td>8.9</td>
    <td>13.8</td>
    <td>13.2</td>
    <td>10.9</td>
    <td>15.1</td>
    <td>11.9</td>
    <td>14.4</td>
    <td>10.4</td>
  </tr>
  <tr>
    <th>Llama 3.1-8B</th>
    <td>10.4</td>
    <td>6.4</td>
    <td>14.1</td>
    <td>6.1</td>
    <td>4.9</td>
    <td>6.9</td>
    <td>9.1</td>
    <td>4.1</td>
    <td>9.6</td>
    <td>5.5</td>
    <td>15.3</td>
    <td>4.0</td>
    <td>14.5</td>
    <td>4.2</td>
    <td>12.3</td>
    <td>5.9</td>
    <td>12.4</td>
    <td>7.1</td>
  </tr>
  <tr>
    <th>Mixtral 8×7B</th>
    <td>7.8</td>
    <td>0.0</td>
    <td>7.4</td>
    <td>-5.1</td>
    <td>7.1</td>
    <td>-2.1</td>
    <td>-0.5</td>
    <td>-7.1</td>
    <td>-4.0</td>
    <td>3.3</td>
    <td>5.6</td>
    <td>5.7</td>
    <td>9.4</td>
    <td>-3.2</td>
    <td>6.7</td>
    <td>1.0</td>
    <td>8.3</td>
    <td>1.0</td>
  </tr>
</table>

Table 6: The system-level Kendall correlations between human-assigned scores and summary-based evaluation results, comparing scenarios with and without provided criteria definitions.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="8">PLOT CHA WRI WOR THE EMO ENJ EXP</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>Chapter Summaries (4o)</th>
    <th>Overall Summary (4o)</th>
    <th>Overall Summary (4o-mini)</th>
    <th>Chapter Summaries (4o)</th>
    <th>Overall Summary (4o)</th>
    <th>Overall Summary (4o-mini)</th>
    <th>Chapter Summaries (4o)</th>
    <th>Overall Summary (4o)</th>
    <th>Overall Summary (4o-mini)</th>
  </tr>
  <tr>
    <th>GPT-4o</th>
    <td>14.8</td>
    <td>15.3</td>
    <td>9.7</td>
    <td>14.5</td>
    <td>17.8</td>
    <td>9.2</td>
    <td>8.3</td>
    <td>4.5</td>
    <td>5.0</td>
    <td>6.0</td>
    <td>10.7</td>
    <td>7.2</td>
    <td>8.7</td>
    <td>15.6</td>
    <td>12.6</td>
    <td>8.5</td>
    <td>11.8</td>
    <td>7.2</td>
    <td>16.6</td>
    <td>14.0</td>
    <td>14.3</td>
    <td>13.4</td>
    <td>10.1</td>
  </tr>
  <tr>
    <th>DeepSeek-v2.5</th>
    <td>14.4</td>
    <td>13.4</td>
    <td>12.3</td>
    <td>15.0</td>
    <td>12.2</td>
    <td>9.7</td>
    <td>2.9</td>
    <td>1.8</td>
    <td>-3.8</td>
    <td>1.4</td>
    <td>1.3</td>
    <td>7.1</td>
    <td>6.9</td>
    <td>10.8</td>
    <td>8.9</td>
    <td>6.9</td>
    <td>10.2</td>
    <td>10.6</td>
    <td>12.2</td>
    <td>13.2</td>
    <td>12.9</td>
    <td>15.1</td>
    <td>14.4</td>
    <td>10.8</td>
  </tr>
</table>

Table 7: The system-level Kendall correlations between human-assigned scores and summary-based evaluation results, comparing scenarios that offer different types of summaries.

GPT-4o-mini). As shown in Table 7, more detailed and higher-quality summaries can be helpful, though not significantly so.

F Efficiency Comparison of Different Evaluation Strategies

The cost comparison of different strategies (§4.1) follows this order: Incremental-Updated > Aggregation-Based > Summary-Based. As mentioned in §5.2, we mitigate inconsistencies in LLM-generated evaluations by averaging 5 results. Thus, the summary-based method is more efficient because it requires only one high-cost summary of the lengthy book, with the 5 evaluations processing only short summaries. In contrast, for aggregation-based evaluation, each of the 5 evaluations must process both the lengthy book and previous summaries. For the incremental-updated method, each evaluation additionally processes the previous evaluations. Taking GPT-4o-based evaluations as an example, the runtime and money cost of 5 evaluations on the test set are shown in Table 8. In conclusion, while aggregation-based methods perform better than summary-based methods (Table 2), their high cost makes summary-based methods a more cost-effective choice.

<table>
  <tr>
    <th>Methods</th>
    <th>Input Tokens</th>
    <th>Runtime</th>
    <th>Cost</th>
  </tr>
  <tr>
    <td>Summary-based</td>
    <td>3,940K</td>
    <td>770min</td>
    <td>$94</td>
  </tr>
  <tr>
    <td>Incremental-updated</td>
    <td>12,720K</td>
    <td>4,268min</td>
    <td>$499</td>
  </tr>
  <tr>
    <td>Aggregation-based</td>
    <td>11,480K</td>
    <td>3,056min</td>
    <td>$416</td>
  </tr>
</table>

Table 8: The runtime and money cost of different evaluation methods (taking GPT-4o-based evaluations as an example).

G Additional Cases

In Figure 6, we present an example of a poorly written story, while Figure 9 shows a well-written one. It can be found that NovelCritique offers more concrete and less formulaic comments.

H Prompts

H.1 Review Processing

The review processing prompt is shown in Table 10.

H.2 Instructions of NovelCritique

The instruction of NovelCritique is shown in Table 13.

H.3 LLM-based Evaluation

The prompt for evaluation is displayed in Table 14.
<table>
  <tr>
    <th>Aspects</th>
    <th>Sub-aspects</th>
    <th>Mapped to normalized criteria</th>
  </tr>
  <tr>
    <td rowspan="3">Plot & Structure</td>
    <td><b>Plot Development</b></td>
    <td>Plot Development [Story Development, Plot Progression, Development], Plot Complexity [Complexity, Plot Twists, Turns, Surprises, Surprise, Plot Predictability, Plot Intensity], Plot Conflict [Conflict, Resolution, Plot Resolution, Conflict Resolution], Pacing [Pace], Plot Clarity</td>
  </tr>
  <tr>
    <td><b>Structure</b></td>
    <td>Plot Structure [Story Structure, Subplots, Timelines, Flow, Narrative Perspective], Plot Elements [Beginning, Middle, Climax, Plot Completion], Plot Coherence [Coherence, Plot Continuity, Consistency, Cohesion], Logic</td>
  </tr>
  <tr>
    <td><b>Ending</b></td>
    <td>Ending [Epilogue, Endings, Closure, Cliffhanger, Conclusion]</td>
  </tr>
  <tr>
    <td rowspan="6">Characters</td>
    <td><b>Development</b></td>
    <td>Character Development [Character Progression], Backstories, Motivations [Character Dynamics], Character Growth, Character Arcs, Character Behavior</td>
  </tr>
  <tr>
    <td><b>Characterization</b></td>
    <td>Characterization [Character Representation], Character Appeal [Character Appreciation, Character Identification], Character Realism, Character Relatability [Character Engagement], Character Depth [Character Complexity], Protagonist [Main Characters, Main Character, Character Focus], Character Introduction</td>
  </tr>
  <tr>
    <td><b>Relationships</b></td>
    <td>Character Relationships [Character Connection, Interactions, Friendships], Character Interactions, Chemistry [Intimacy, Romance and Relationships], Relationship Development | Family Dynamics, Relationship Dynamics]</td>
  </tr>
  <tr>
    <td><b>Diversity</b></td>
    <td>Character Diversity [Supporting Characters, Secondary Characters, Side Characters, Character Perspectives, Character Perspective, Character Management]</td>
  </tr>
  <tr>
    <td rowspan="2"><b>Writing & Language</b></td>
    <td><b>Writing Style</b></td>
    <td>Writing Style [Style, Author’s Style, Author’s Writing Style, Narrative Style, Author’s Skill], Atmosphere, Tone [Author’s Tone], Trope [Tropes], Writing Elements [Descriptions, Prose, Imagery, Dialogue, Dialogues]</td>
  </tr>
  <tr>
    <td><b>Language</b></td>
    <td>Grammar [Language Usage, Vocabulary], Fluency [Repetition, Repetitiveness, Author’s Reputation]</td>
  </tr>
  <tr>
    <td rowspan="1"></td>
    <td><b>Readability</b></td>
    <td>Readability [Clarity, Accessibility, Length]</td>
  </tr>
  <tr>
    <td rowspan="2">Themes</td>
    <td><b>Exploration/Clarity</b></td>
    <td>Thematic Elements [Topics, Point of View, Thematic Content, Symbolism, Motifs, Messages], Thematic Exploration [Themes Exploration]</td>
  </tr>
  <tr>
    <td><b>Depth</b></td>
    <td>Thematic Depth [Social Commentary, Educational Value, Influences]</td>
  </tr>
  <tr>
    <td rowspan="2">World-Building & Setting</td>
    <td><b>World-Building</b></td>
    <td>World-Building [Worldbuilding, World Building], Realism [Cultural Realism, Magic System, Magic, Magic Systems]</td>
  </tr>
  <tr>
    <td><b>Setting</b></td>
    <td>Historical Accuracy [Historical Setting, Historical Context, Time Period], Cultural Elements [Cultural References, Cultural Context, Cultural Representation], Technological Elements...</td>
  </tr>
  <tr>
    <td rowspan="2">Emotional Impact</td>
    <td><b>Empathy</b></td>
    <td>Emotional Response [Personal Connection, Emotional Resonance]</td>
  </tr>
  <tr>
    <td><b>Depth</b></td>
    <td>Emotional Depth [Emotional Range]</td>
  </tr>
  <tr>
    <td rowspan="2">Enjoyment & Engagement</td>
    <td><b>Enjoyment</b></td>
    <td>Enjoyment [Overall Impression, Overall Impact, Excitement, Appeal, Story Enjoyment, Interest, Humor], Anticipation for Future Works [Future Interest, Intrigue, Anticipation for Future Book, Sequel Potential]</td>
  </tr>
  <tr>
    <td><b>Engagement</b></td>
    <td>Engagement [Reader Engagement, Personal Engagement, Immersion, Story Engagement, Reader Engagement, Overall Experience, Reading Experience]</td>
  </tr>
  <tr>
    <td rowspan="3">Expectation & Fulfillment</td>
    <td><b>Genre</b></td>
    <td>Genre Expectation, Genre Appeal, Genre Preference, Audience Appeal, Genre Suitability, Romantic Elements, Thrill, Horror Elements, Fantasy Elements, Historical, Suspense, Tension, Scare Factor, Genre Elements, Genre Classification, Mystery]</td>
  </tr>
  <tr>
    <td><b>Premise</b></td>
    <td>Premise Expectations [Premise Expectation, Story Premise]</td>
  </tr>
  <tr>
    <td><b>Title</b></td>
    <td>Title Relevance</td>
  </tr>
  <tr>
    <td rowspan="3">Others</td>
    <td><b>Originality</b></td>
    <td>Originality [Creativity, Uniqueness, Originality and Creativity, Unique Elements]</td>
  </tr>
  <tr>
    <td><b>Content Warnings</b></td>
    <td>Content Warnings [Warnings, Trigger Warnings]</td>
  </tr>
  <tr>
    <td><b>Designment</b></td>
    <td>Designment [Cover, Cover Design]</td>
  </tr>
  <tr>
    <td rowspan="1"></td>
    <td><b>Personal Bias</b></td>
    <td>Personal Preference, Personal Experience</td>
  </tr>
</table>

Table 9: The detailed criteria structure.
GPT-4o

Plot and Structure:
The plot of “Mind Games” weaves together a suspense-filled narrative with emotionally driven character arcs. The combination of psychological and love story adds originality to the plot, allowing for more nuanced quiet exchanges amidst the tension. The central conflict between Thea and Ray Riggs builds slow-burning suspense, culminating in a gripping confrontation that leaves the reader questioning their motivations, such as Thea’s game design career, feel undeveloped. The pacing is uneven at times, with stretches of inactivity that could have been better utilized for character development. Despite this, the structure follows a logical progression, building emotional beats with moments of mac suspense. (4.0)

Characters:
Thea is a well-developed protagonist whose journey from vulnerability to strength is both relatable and engaging. The supporting characters, including Ty, Lucy, and Rem, add depth to her world. Ty’s character complements Thea as both a romantic interest and ally, while Lucy provides a grounding presence. However, the antagonist Riggs could benefit from more nuanced development to enhance his role in the psychological conflict. (4.0)

Expectation Fulfillment:
As a work of romantic suspense, Mind Games largely fulfills its genre expectations. It delivers a mix of romance, tension, and character growth. The psychic element sets it apart from conventional narratives and provides an interesting angle that meets themes of the unknown in human relationships. (4.0)

Llama3.1-8B

Plot and Structure:
The plot of “Mind Games” is engaging and suspenseful, with a clear structure that effectively builds tension and conflict. The story begins with a strong premise, introducing the main characters and their abilities to see into minds and souls. The plot unfolds at a good pace, with a clear three-act structure: setup, confrontation, and resolution. The narrative jumps forward in time, showing Thea’s growth into a confident and capable woman, and the twists involving Ty, Ray, and Lucy, to add depth and complexity to the story. However, some plot points, like Thea’s decision to confront Ray Riggs in prison, felt a bit convenient and lack a clear motivation. The ending was satisfying, but the resolution was abrupt, and the resolution could be more satisfying. Overall, the plot is well-structured, but some elements could be improved. (4.0)

Characters:
The characters in “Mind Games” are well-developed and complex, with each one playing a pivotal role. The main characters, like Thea, Ty, Ray, and Lucy, are well-rounded and add depth to the story. However, some readers may find Rem and the villain, Ray, a bit one-dimensional. (4.0)

Expectation Fulfillment:
“Mind Games” generally meets the reader’s expectations, particularly in terms of the romance and suspense elements. The author effectively delivers on the premise, and the story is engaging and suspenseful. However, some readers may find the resolution abrupt and the character development could be more nuanced. (4.0)

NovelCritique-8B

Plot and Structure:
The story revolves around Thea Fox, a 12-year-old girl who witnesses her parents’ murder by Ray Riggs, a powerful psychic. Thea’s grandmother, Lucy, shares the same psychic ability, which she’s kept hidden from Thea. Thea’s journey is a personal and path of self-discovery, as she struggles to understand her gift and its implications. The narrative jumps forward in time, showing Thea’s growth into a confident and capable woman, and the twists involving Ty, Ray, and Lucy, to add depth and complexity to the story. She eventually moves to Redbud Hollow, Kentucky, to live with her grandmother, where she meets Tyler Brennan, a single father who has moved to the town to start anew. Thea’s connection to Ray Riggs, who is now in prison, continues to haunt her, and she must navigate her feelings for Tyler while confronting the evil that has shaped her life. (4.5)

Characters:
Thea Fox: Thea was a strong and resilient protagonist who faced numerous challenges, including the loss of her parent’s murder and the ongoing threat from Ray Riggs. Her ability to see into minds and souls added depth to her character, making her both relatable and intriguing.

Lucy: Lucy was a wonderful supporting character. Her wisdom, kindness, and unwavering support for Thea made her a standout. The bond between Thea and Lucy was touching and heartwarming.

Ray Riggs: Riggs was a well-crafted antagonist. His ability to see into minds and souls, combined with his twisted thoughts and evil acts, made him a formidable and unsettling character. His ongoing threat to Thea added tension to the story. (4.5)

Expectation Fulfillment:
The novel is a romantic suspense, which I found to be a perfect blend of genres. It was engaging and kept me hooked throughout.

The story revolves around Thea Fox, a psychic who can see into minds and souls, and her grandmother Lucy, who shares the same ability. Thea’s journey is a personal and path of self-discovery, as she has been living with the trauma of his actions. The novel explores themes of love, family, and the evil that exists in the world. (4.0)

Figure 9: Critiques for “Mind Games” (AVG human rating: 4.35).

Raw Review Processing

As an expert in storytelling, your task is to reformat a reader’s review of a full-length novel into a more structured one.

Objective:
- Identify the evaluation aspects discussed in this review.
- Extract the relevant contents that correspond to each identified aspect.
- Organize these contents into a structured format.

Guidelines:
- Maintain the same tone and perspective (e.g., first-person) in the reformatted review, as if you are the original reader.
- Do not introduce any new information; only reformat the existing review. You may polish the language and improve clarity.
- Ensure all important viewpoints in the existing review are preserved.

Output Format:
Aspects:
- List of evaluation aspects identified in the review and the corresponding viewpoints.
Conclusion:
Brief summary of the review and a final verdict.
Rating Scores:
Any rating scores mentioned in the review.

Now, proceed with your task.

Review:
[Raw Review]

Reformatted Review:

Table 10: The prompt for review processing.
Generate Initial Summary

Below is the beginning of a lengthy story:

- - -
{ }
- - -

Instructions:
Your task is to craft a comprehensive summary of the beginning of a lengthy story, including plot and characters.

Guidelines:
1. Plot:
- Summarize the vital information in the beginning part, such as key events, main conflicts, backgrounds, settings, and characters.
- The story may feature non-linear narratives, flashbacks, or switches between alternate worlds or viewpoints. In such cases, state the time and settings for these narrative shifts.

2. Characters:
- Introduce the major characters’ information, including their profiles and overall experiences.

Output Format:
- - -
### Plot Summary:
{ }

### Characters:
Name:
- Profile: Concisely summarize the character’s profile with natural language, including the role (e.g., protagonist), attributes, personality, and relationships within 50 words.
- Overall Experience: Briefly describe the character’s overall experience, including motivations, events, and emotional states within 100 words.
- - -

Now start your summarization:

Table 11: Prompt to generate the initial summary.
Update Summary

Below is a segment from a story:

- - -
{ }
- - -

Below is a summary of the story up until this point:

- - -

### Plot Summary: { }

### Characters: { }

- - -

Instructions:
Your task is to sequentially analyze segments of a lengthy book, updating a comprehensive summary of the entire story, including plot and characters.

Guidelines:
1. Plot:
- First summarize the current segment. You must briefly introduce characters, places, and other major elements if they are mentioned for the first time.
- Update the overall plot summary to include vital information in this segment, related to key events, main conflicts, backgrounds, settings, and characters.
- The story may feature non-linear narratives, flashbacks, or switches between alternate worlds or viewpoints. In such cases, state the time and settings for these narrative shifts.

2. Characters:
- Update the major characters’ information, including their profiles, current experiences, and overall experiences.
- Add new significant characters from this segment to the list.
- If the characters not mentioned in this segment are not crucial to the story’s development, remove them from the list of major characters .

Output Format:
- - -
### Summary of Current Segment:
{ }
### Overall Plot Summary (within 1000 words):
{ }
### Characters:
Name:
- Profile: Update the character’s profile, including the role (e.g., protagonist), attributes, personality, and relationships within 50 words.
- Current Experience: Briefly describe the character’s motivations, events, interactions, and emotional states in this segment within 50 words. If the character isn’t mentioned, write “Not mentioned”.
- Overall Experience: Update the character’s overall experience and development throughout the story within 100 words, incorporating the current experience.
- - -

Now start your summarization:

Table 12: Prompt to incrementally update the summary of plot and characters.
NovelCritique

Below are the title, genres, and premise of a book-length story.

---

Title: {}

Genres: {}

Premise:
{}

---

Below are the plot summary, character analysis, and excerpts from this book-length story.

---

Plot Summary:
{}

Character Analysis:
{}

Excerpts: (Demonstrating Writing and Language):
{}

---

Write a review of this story based on the criteria below. Then conclude with an overall assessment and assign a final score from 1.0 (lowest) to 5.0 (highest).

### Review:
/*Specific Aspects*/
Characters:
{Critiques of Characters.}

Emotional Impact:
{Critiques of Emotional Impact.}

### Overall Assessment:
{Summary of the review.}

### Score: X.X

Table 13: Instructions of NovelCritique. Here we display a training sample focusing on two specific aspects: Characters and Emotional Impact.
LLM-based Evaluation

Below are the title, genres, and premise of a book-length story.

- - -

Title: {}

Genres: {}

Premise:
{}

- - -

Below are the plot summary, character analysis, and excerpts from this book-length story.

- - -

Plot Summary:
{}

Character Analysis:
{}

Excerpts:
{}

- - -

Please evaluate the story based on the criteria listed below. Provide detailed reviews highlighting both strengths and weaknesses. Assign a score for each aspect on a continuous scale from 0 to 100, where 0 represents the poorest performance and 100 represents the best performance.

Evaluation Criteria:
### 1. Plot and Structure:
- Review:
- Score: X
- Guidelines: Evaluate the plot development by examining pace, twists, conflicts, and their resolutions. Evaluate the story structure for coherence, logic, and complexity, paying attention to key elements like climax and ending.

/*Definitions of other aspects*/

### Conclusion:
- Overall Assessment:
- Overall Score: X
- Guidelines: Conclude with an overall assessment of the story, summarizing key insights and assigning an overall score on a continuous scale from 0 to 100.

Now start your evaluation:

Table 14: Prompt for LLM-based Evaluation. Here we display the prompt for summary-based evaluation, while for others, the difference is to replace the input contents and instructions. For example, in incremental evaluation, we input the current segment, previous summary, and previous evaluations, replacing “evaluate the story” as “update the evaluation”.
<table>
  <tr>
    <th>Title</th>
    <th>Author</th>
    <th>Genres</th>
    <th>Published Date</th>
    <th>AVG. Score</th>
    <th>Length(Word)</th>
  </tr>
  <tr>
    <td>Remedial Magic</td>
    <td>Melissa Matt</td>
    <td>Fantasy, Romance, LGBT</td>
    <td>February 20, 2024</td>
    <td>2.90</td>
    <td>88407</td>
  </tr>
  <tr>
    <td>Sanctuary of the Shadow</td>
    <td>Aurora Ascher</td>
    <td>Fantasy, Romance, Romantic</td>
    <td>January 9, 2024</td>
    <td>2.90</td>
    <td>96569</td>
  </tr>
  <tr>
    <td>We Came to Welcome You</td>
    <td>Vincent Tirado</td>
    <td>Horror, Thriller, LGBT</td>
    <td>September 3, 2024</td>
    <td>3.12</td>
    <td>160662</td>
  </tr>
  <tr>
    <td>Witchcraft</td>
    <td>Thilo Fustmann</td>
    <td>Science Fiction, Fantasy, Dystopia</td>
    <td>January 23, 2024</td>
    <td>3.20</td>
    <td>122665</td>
  </tr>
  <tr>
    <td>Memory Piece</td>
    <td>Lisa K.</td>
    <td>Historical Fiction, Literary Fiction, Contemporary</td>
    <td>March 19, 2024</td>
    <td>3.20</td>
    <td>89054</td>
  </tr>
  <tr>
    <td>The Longest Autumn</td>
    <td>Amy Avery</td>
    <td>Fantasy, Romance, Adult</td>
    <td>January 16, 2024</td>
    <td>3.21</td>
    <td>91973</td>
  </tr>
  <tr>
    <td>The Girl with No Reflection</td>
    <td>Keshe Chow</td>
    <td>Fantasy, Young Adult, Romance</td>
    <td>August 6, 2024</td>
    <td>3.28</td>
    <td>109194</td>
  </tr>
  <tr>
    <td>Angel</td>
    <td>EDY C.</td>
    <td>Mystery, Thriller, Mystery Thriller</td>
    <td>January 20, 2024</td>
    <td>3.28</td>
    <td>118321</td>
  </tr>
  <tr>
    <td>Castle of the Cursed</td>
    <td>Romina Garber</td>
    <td>Fantasy, Horror, Young Adult</td>
    <td>July 30, 2024</td>
    <td>3.30</td>
    <td>90165</td>
  </tr>
  <tr>
    <td>Beach Camp</td>
    <td>Beth Reekles</td>
    <td>Romance, Young Adult, Contemporary</td>
    <td>May 14, 2024</td>
    <td>3.34</td>
    <td>88344</td>
  </tr>
  <tr>
    <td>All That and More</td>
    <td>Peng Shuherd</td>
    <td>Science Fiction, Fantasy, Dystopia</td>
    <td>July 9, 2024</td>
    <td>3.34</td>
    <td>135468</td>
  </tr>
  <tr>
    <td>You Know What You Did</td>
    <td>K.T. Nguyen</td>
    <td>Thriller, Mystery, Mystery Thriller</td>
    <td>April 16, 2024</td>
    <td>3.36</td>
    <td>95189</td>
  </tr>
  <tr>
    <td>Draw Down the Moon</td>
    <td>P.C. Cast</td>
    <td>Fantasy, Young Adult, Romance</td>
    <td>April 2, 2024</td>
    <td>3.38</td>
    <td>80332</td>
  </tr>
  <tr>
    <td>Big Time</td>
    <td>Ben H Winters</td>
    <td>Science Fiction, Thriller, Mystery</td>
    <td>January 9, 2024</td>
    <td>3.41</td>
    <td>65522</td>
  </tr>
  <tr>
    <td>The Deadly Prophecies</td>
    <td>Andrew Tang</td>
    <td>Fantasy, Mystery, Young Adult</td>
    <td>January 21, 2024</td>
    <td>3.42</td>
    <td>70677</td>
  </tr>
  <tr>
    <td>Fish Out of Water</td>
    <td>Katie Ringle</td>
    <td>Romance, Contemporary, Adult</td>
    <td>February 13, 2024</td>
    <td>3.43</td>
    <td>72901</td>
  </tr>
  <tr>
    <td>Adam and Eve’s Matchmaking Tour</td>
    <td>Nora Nguyen</td>
    <td>Romance, Contemporary, Travel</td>
    <td>September 24, 2024</td>
    <td>3.44</td>
    <td>129425</td>
  </tr>
  <tr>
    <td>The Village Library Demon-Hunting Society</td>
    <td>C.M. Waggoner</td>
    <td>Fantasy, Mystery, Cozy Mystery</td>
    <td>September 24, 2024</td>
    <td>3.44</td>
    <td>93691</td>
  </tr>
  <tr>
    <td>This Ravenous Fate</td>
    <td>E.B. Smith</td>
    <td>Fantasy, Historical, Science Fiction</td>
    <td>October 12, 2024</td>
    <td>3.45</td>
    <td>149353</td>
  </tr>
  <tr>
    <td>I Need You to Read This</td>
    <td>Hayley Denning</td>
    <td>Fantasy, Lesbian, Vampires</td>
    <td>August 6, 2024</td>
    <td>3.45</td>
    <td>104419</td>
  </tr>
  <tr>
    <td>Lady Macbeth</td>
    <td>Jessa Maxwell</td>
    <td>Mystery, Thriller, Mystery Thriller</td>
    <td>August 13, 2024</td>
    <td>3.46</td>
    <td>71906</td>
  </tr>
  <tr>
    <td>Every Time I Go on Vacation, Someone Dies</td>
    <td>Ana Rios</td>
    <td>Fantasy, Historical Fiction, Readings</td>
    <td>August 13, 2024</td>
    <td>3.46</td>
    <td>69856</td>
  </tr>
  <tr>
    <td>Dark Restrain</td>
    <td>Caroline Mack</td>
    <td>Romance, Mystery, Magical Thriller</td>
    <td>August 13, 2024</td>
    <td>3.46</td>
    <td>70566</td>
  </tr>
  <tr>
    <td>Infinity Light</td>
    <td>Katee Robert</td>
    <td>Romance, Fantasy, Mythology</td>
    <td>August 6, 2024</td>
    <td>3.49</td>
    <td>78817</td>
  </tr>
  <tr>
    <td>An Academy for Lairs</td>
    <td>Kacen Callender</td>
    <td>Fantasy, Young Adult, LGBT</td>
    <td>February 6, 2024</td>
    <td>3.49</td>
    <td>113207</td>
  </tr>
  <tr>
    <td>Love, Lies and Cherry Pie</td>
    <td>Alexis Henderson</td>
    <td>Fantasy, High, Gothic</td>
    <td>September 17, 2024</td>
    <td>3.50</td>
    <td>121043</td>
  </tr>
  <tr>
    <td>One Big Happy Family</td>
    <td>Jackie J. Ray</td>
    <td>Fantasy, Contemporary, Adult</td>
    <td>May 1, 2024</td>
    <td>3.52</td>
    <td>82146</td>
  </tr>
  <tr>
    <td>Ghost Station</td>
    <td>S.A. Barnes</td>
    <td>Horror, Science Fiction, Thriller</td>
    <td>July 16, 2024</td>
    <td>3.54</td>
    <td>106628</td>
  </tr>
  <tr>
    <td>Grave Harbor</td>
    <td>Peter Nichols</td>
    <td>Mystery, Thriller, Horror</td>
    <td>April 9, 2024</td>
    <td>3.55</td>
    <td>105849</td>
  </tr>
  <tr>
    <td>As Still as Darkness</td>
    <td>Vera Kline</td>
    <td>Thriller, Mystery, Mystery Thriller</td>
    <td>April 30, 2024</td>
    <td>3.56</td>
    <td>81080</td>
  </tr>
  <tr>
    <td>Christ Comes Out of Her Shell</td>
    <td>Abbi Waxman</td>
    <td>Romance, Contemporary, Chick Lit</td>
    <td>February 20, 2024</td>
    <td>3.56</td>
    <td>13822</td>
  </tr>
  <tr>
    <td>The Getaway List</td>
    <td>Emma Lord</td>
    <td>Romance, Young Adult, Contemporary</td>
    <td>April 16, 2024</td>
    <td>3.57</td>
    <td>91578</td>
  </tr>
  <tr>
    <td>Last House</td>
    <td>Jessica Sharapov</td>
    <td>Historical Fiction, Historical, Family</td>
    <td>January 23, 2024</td>
    <td>3.57</td>
    <td>93259</td>
  </tr>
  <tr>
    <td>The Square Room</td>
    <td>Laura Ramsey</td>
    <td>Romance, Mystery, Chick Lit</td>
    <td>May 14, 2024</td>
    <td>3.59</td>
    <td>122656</td>
  </tr>
  <tr>
    <td>The Honey Witch</td>
    <td>Sydney J. Shields</td>
    <td>Fantasy, Romance, LGBT</td>
    <td>February 6, 2024</td>
    <td>3.59</td>
    <td>97241</td>
  </tr>
  <tr>
    <td>The Midnight Feast</td>
    <td>Lucy Foley</td>
    <td>Mystery, Thriller, Mystery Thriller</td>
    <td>May 14, 2024</td>
    <td>3.59</td>
    <td>98048</td>
  </tr>
  <tr>
    <td>The Unlikely</td>
    <td>Leah S. Bock</td>
    <td>Fantasy, Magical Realism, Historical Fiction</td>
    <td>June 18, 2024</td>
    <td>3.59</td>
    <td>90608</td>
  </tr>
  <tr>
    <td>The Minutiae of Time</td>
    <td>Kalame Bradley</td>
    <td>Science Fiction, Romance, Time Travel</td>
    <td>July 2, 2024</td>
    <td>3.61</td>
    <td>117109</td>
  </tr>
  <tr>
    <td>Sweet Nightmare</td>
    <td>Tracy Wolff</td>
    <td>Fantasy, Romance, Young Adult</td>
    <td>May 7, 2024</td>
    <td>3.61</td>
    <td>87170</td>
  </tr>
  <tr>
    <td>Hypnotized by Love</td>
    <td>Sarah Wilson</td>
    <td>Romance, Contemporary, Chick Lit</td>
    <td>March 1, 2024</td>
    <td>3.62</td>
    <td>139647</td>
  </tr>
  <tr>
    <td>The Glass Cheese</td>
    <td>Melissa Hubert</td>
    <td>Romance, Young Adult</td>
    <td>February 20, 2024</td>
    <td>3.62</td>
    <td>79768</td>
  </tr>
  <tr>
    <td>The Canterbury’s Guide to the Wastelands</td>
    <td>Sarah Brooks</td>
    <td>Fantasy, Military, Historical</td>
    <td>June 18, 2024</td>
    <td>3.63</td>
    <td>94373</td>
  </tr>
  <tr>
    <td>A Friend in the Dark</td>
    <td>Samantha M. Bailey</td>
    <td>Thriller, Mystery, Mystery Thriller</td>
    <td>March 1, 2024</td>
    <td>3.64</td>
    <td>67035</td>
  </tr>
  <tr>
    <td>The Framed Women of Ardmore House</td>
    <td>Brandy Schillace</td>
    <td>Mystery, Mystery Thriller, Cozy Mystery</td>
    <td>February 12, 2024</td>
    <td>3.65</td>
    <td>90841</td>
  </tr>
  <tr>
    <td>At the End of the World</td>
    <td>Nashelle Richards</td>
    <td>Thriller, Mystery, Historical Fiction</td>
    <td>July 2, 2024</td>
    <td>3.65</td>
    <td>65690</td>
  </tr>
  <tr>
    <td>The Cemetery of Untold Stories</td>
    <td>Julia Alvarez</td>
    <td>Fantasy, Magical Realism, Historical Fiction</td>
    <td>April 2, 2024</td>
    <td>3.66</td>
    <td>72967</td>
  </tr>
  <tr>
    <td>The Nature of Disappearing</td>
    <td>Kumi Cunningham Grant</td>
    <td>Thriller, Mystery, Mystery Thriller</td>
    <td>June 18, 2024</td>
    <td>3.66</td>
    <td>79555</td>
  </tr>
  <tr>
    <td>Perfect Little Monsters</td>
    <td>Cindy R.X. He</td>
    <td>Mystery, Young Adult, Thriller</td>
    <td>May 7, 2024</td>
    <td>3.66</td>
    <td>68630</td>
  </tr>
  <tr>
    <td>Brave New Island</td>
    <td>Grace O’Hara</td>
    <td>Science Fiction, Fantasy, Fantasy</td>
    <td>March 19, 2024</td>
    <td>3.67</td>
    <td>79030</td>
  </tr>
  <tr>
    <td>Ocean’s Godziri</td>
    <td>Elaine U. Cho</td>
    <td>Science Fiction, Science Fiction, Fantasy</td>
    <td>April 23, 2024</td>
    <td>3.67</td>
    <td>82458</td>
  </tr>
  <tr>
    <td>The Honeymoon Affair</td>
    <td>Sheila O’Flanagan</td>
    <td>Romance</td>
    <td>May 9, 2024</td>
    <td>3.67</td>
    <td>117284</td>
  </tr>
  <tr>
    <td>The Blue Biscuit Bay</td>
    <td>Cherie Cleaton</td>
    <td>Historical Fiction, Mystery, Gothic</td>
    <td>April 2, 2024</td>
    <td>3.68</td>
    <td>78574</td>
  </tr>
  <tr>
    <td>I’m Afraid You’ve Got Dragons</td>
    <td>Prue B. Anglin</td>
    <td>Fantasy, Dragons, Humor</td>
    <td>May 14, 2024</td>
    <td>3.69</td>
    <td>72237</td>
  </tr>
  <tr>
    <td>Under Loch and Key</td>
    <td>Lana Ferguson</td>
    <td>Romance, Fantasy, Paranormal</td>
    <td>December 3, 2024</td>
    <td>3.69</td>
    <td>105973</td>
  </tr>
  <tr>
    <td>Such a Lovely Family</td>
    <td>Aggie Blum Thompson</td>
    <td>Thriller, Mystery, Mystery Thriller</td>
    <td>March 12, 2024</td>
    <td>3.69</td>
    <td>94600</td>
  </tr>
  <tr>
    <td>Riley Sager</td>
    <td>Riley Sager</td>
    <td>Thriller, Mystery, Mystery Thriller</td>
    <td>June 18, 2024</td>
    <td>3.69</td>
    <td>97895</td>
  </tr>
  <tr>
    <td>The Things We Went Missing</td>
    <td>Chelsea Musso</td>
    <td>Romance, Historical, Chick Lit</td>
    <td>March 12, 2024</td>
    <td>3.69</td>
    <td>85368</td>
  </tr>
  <tr>
    <td>Danica Nova</td>
    <td>Romance, Contemporary, Indigenous</td>
    <td>August 6, 2024</td>
    <td>3.70</td>
    <td>91302</td>
  </tr>
  <tr>
    <td>The Starless Grait</td>
    <td>Yume Kitase</td>
    <td>Fantasy, Space, Adult</td>
    <td>June 11, 2024</td>
    <td>3.70</td>
    <td>93381</td>
  </tr>
  <tr>
    <td>Beauty and the Beastly</td>
    <td>Jennifer Donnelly</td>
    <td>Fantasy, Young Adult, Retellings</td>
    <td>May 7, 2024</td>
    <td>3.71</td>
    <td>91129</td>
  </tr>
  <tr>
    <td>Of Jades and Dragons</td>
    <td>Amber Lee</td>
    <td>Fantasy, Young Adult, Thriller</td>
    <td>June 14, 2024</td>
    <td>3.71</td>
    <td>101977</td>
  </tr>
  <tr>
    <td>Seven Summer Weekends</td>
    <td>Jane L. Rosen</td>
    <td>Romance, Contemporary, Women Fiction</td>
    <td>June 4, 2024</td>
    <td>3.72</td>
    <td>65185</td>
  </tr>
  <tr>
    <td>The Vacancy in Room 10</td>
    <td>Seraphina Nova Glass</td>
    <td>Thriller, Mystery, Mystery Thriller</td>
    <td>April 9, 2024</td>
    <td>3.73</td>
    <td>91666</td>
  </tr>
  <tr>
    <td>Mobsters</td>
    <td>Charlotte Becker</td>
    <td>Mystery, Young Adult, Thriller</td>
    <td>September 1, 2024</td>
    <td>3.74</td>
    <td>90696</td>
  </tr>
  <tr>
    <td>Beautiful Ugly</td>
    <td>Alice Harvey</td>
    <td>Mystery, Thriller, Mystery Thriller</td>
    <td>January 14, 2025</td>
    <td>3.74</td>
    <td>84800</td>
  </tr>
  <tr>
    <td>Such Charming Liars</td>
    <td>Karen M. McManus</td>
    <td>Mystery, Young Adult, Thriller</td>
    <td>July 30, 2024</td>
    <td>3.74</td>
    <td>91903</td>
  </tr>
  <tr>
    <td>The Stars Too Fondly</td>
    <td>Emily Hamilton</td>
    <td>Science Fiction, Romance, LGBT</td>
    <td>June 11, 2024</td>
    <td>3.76</td>
    <td>116399</td>
  </tr>
  <tr>
    <td>Letting Go of Things</td>
    <td>Kevin J. Anderson</td>
    <td>Romance, Historical, Chick Lit</td>
    <td>May 21, 2024</td>
    <td>3.76</td>
    <td>101959</td>
  </tr>
  <tr>
    <td>If Something Happens To Me</td>
    <td>Alex Hilay</td>
    <td>Thriller, Mystery, Mystery Thriller</td>
    <td>May 28, 2024</td>
    <td>3.76</td>
    <td>61908</td>
  </tr>
  <tr>
    <td>Long Island Compromise</td>
    <td>Taffy Brodesser-Akner</td>
    <td>Historical Fiction, Contemporary, Literary Fiction</td>
    <td>July 9, 2024</td>
    <td>3.77</td>
    <td>81230</td>
  </tr>
  <tr>
    <td>Death at Morning Hour</td>
    <td>Maureen Johnson</td>
    <td>Mystery, Young Adult, Mystery Thriller</td>
    <td>August 6, 2024</td>
    <td>3.77</td>
    <td>85578</td>
  </tr>
  <tr>
    <td>The Time Traveler’s Library</td>
    <td>K. A. L. Brown</td>
    <td>Fantasy, Young Adult, Thriller</td>
    <td>June 4, 2024</td>
    <td>3.78</td>
    <td>122553</td>
  </tr>
  <tr>
    <td>Say you’ll be mine : A novel</td>
    <td>Naina Kumar</td>
    <td>Romance, Contemporary, Adult</td>
    <td>January 16, 2024</td>
    <td>3.79</td>
    <td>86540</td>
  </tr>
  <tr>
    <td>How to Solve Your Own Murder</td>
    <td>Kristen Perrin</td>
    <td>Mystery, Thriller, Thriller</td>
    <td>March 26, 2024</td>
    <td>3.79</td>
    <td>99534</td>
  </tr>
  <tr>
    <td>Love Story</td>
    <td>Lindsey Kelk</td>
    <td>Romance, Contemporary, Chick Lit</td>
    <td>July 4, 2024</td>
    <td>3.81</td>
    <td>95533</td>
  </tr>
  <tr>
    <td>In the Heart of Crows</td>
    <td>Dana Stabenow</td>
    <td>Fantasy, Historical, Magical Realism</td>
    <td>June 4, 2024</td>
    <td>3.81</td>
    <td>78313</td>
  </tr>
  <tr>
    <td>One Last Breath</td>
    <td>Ganny Myers Sain</td>
    <td>Mystery, Young Adult, Thriller</td>
    <td>March 5, 2024</td>
    <td>3.81</td>
    <td>93658</td>
  </tr>
  <tr>
    <td>A Botanist’s Guide to Society and Secrets</td>
    <td>Kate Khavari</td>
    <td>Mystery, Historical Fiction, Cozy Mystery</td>
    <td>June 4, 2024</td>
    <td>3.81</td>
    <td>96785</td>
  </tr>
  <tr>
    <td>The School Run</td>
    <td>Alli Lowe</td>
    <td>Thriller, Mystery, Crime</td>
    <td>May 5, 2024</td>
    <td>3.82</td>
    <td>88185</td>
  </tr>
  <tr>
    <td>Five Little Angels</td>
    <td>Max Garland</td>
    <td>Romance, Young Adult, Adult</td>
    <td>May 7, 2024</td>
    <td>3.82</td>
    <td>81415</td>
  </tr>
  <tr>
    <td>The Broken Places</td>
    <td>Mia Sheridan</td>
    <td>Romance, Mystery, Romantic Suspense</td>
    <td>December 3, 2024</td>
    <td>3.84</td>
    <td>100338</td>
  </tr>
  <tr>
    <td>The Murder After the Night Before</td>
    <td>Katy Brett</td>
    <td>Mystery, Thriller, Mystery Thriller</td>
    <td>February 29, 2024</td>
    <td>3.85</td>
    <td>79183</td>
  </tr>
  <tr>
    <td>So Let Them Burn</td>
    <td>Kamiah Cole</td>
    <td>Fantasy, Young Adult, Dragons</td>
    <td>January 16, 2024</td>
    <td>3.85</td>
    <td>101796</td>
  </tr>
  <tr>
    <td>Shadowsound</td>
    <td>Lisa White</td>
    <td>Historical Fiction, Historical</td>
    <td>June 4, 2024</td>
    <td>3.86</td>
    <td>11866</td>
  </tr>
  <tr>
    <td>Dragonfruit</td>
    <td>Makia Lucier</td>
    <td>Fantasy, Young Adult, Dragons</td>
    <td>April 9, 2024</td>
    <td>3.87</td>
    <td>70344</td>
  </tr>
  <tr>
    <td>House of Glass</td>
    <td>Sarah Pekkanen</td>
    <td>Thriller, Mystery, Thriller</td>
    <td>August 6, 2024</td>
    <td>3.88</td>
    <td>93373</td>
  </tr>
  <tr>
    <td>The Grand Valley : A Novel</td>
    <td>Scott Alexander Howard</td>
    <td>Science Fiction, Fantasy, Time Travel</td>
    <td>February 17, 2024</td>
    <td>3.89</td>
    <td>92981</td>
  </tr>
  <tr>
    <td>Light See</td>
    <td>T.J. Alldredge</td>
    <td>Romance, LGBT</td>
    <td>June 4, 2024</td>
    <td>3.89</td>
    <td>91915</td>
  </tr>
  <tr>
    <td>House of Bone and Rain</td>
    <td>Gabino Iglesias</td>
    <td>Horror, Thriller, Mystery</td>
    <td>August 6, 2024</td>
    <td>3.89</td>
    <td>104032</td>
  </tr>
</table>

Table 15: Metadata of the test set. (Part 1/2)
<table>
  <tr>
    <th>Title</th>
    <th>Author</th>
    <th>Genres</th>
    <th>Published Date</th>
    <th>AVG. Score</th>
    <th>Length(Word)</th>
  </tr>
  <tr>
    <td>Voyage of the Damned</td>
    <td>Frances White</td>
    <td>Fantasy, Mystery, LGBT</td>
    <td>January 18, 2024</td>
    <td>3.90</td>
    <td>116476</td>
  </tr>
  <tr>
    <td>A Song to Drown Rivers</td>
    <td>Ann Liang</td>
    <td>Fantasy, Mystery, Historical</td>
    <td>October 1, 2024</td>
    <td>3.90</td>
    <td>93968</td>
  </tr>
  <tr>
    <td>The Prisoner's Throne</td>
    <td>Holly Black</td>
    <td>Fantasy, Young Adult, Romance</td>
    <td>March 5, 2024</td>
    <td>3.90</td>
    <td>92811</td>
  </tr>
  <tr>
    <td>Ada</td>
    <td>Alex Ransom</td>
    <td>Romance, Contemporary</td>
    <td>February 11, 2024</td>
    <td>3.91</td>
    <td>111244</td>
  </tr>
  <tr>
    <td>Three Kinds of Lucky</td>
    <td>Kim Harrison</td>
    <td>Fantasy, Urban Fantasy, Paranormal</td>
    <td>March 5, 2024</td>
    <td>3.91</td>
    <td>144316</td>
  </tr>
  <tr>
    <td>The Queen of Sugar Hill: A Novel of Hattie McDaniel</td>
    <td>ReShonda Tate</td>
    <td>Historical Fiction, Historical, Race</td>
    <td>January 30, 2024</td>
    <td>3.92</td>
    <td>160852</td>
  </tr>
  <tr>
    <td>Icon and Inferno</td>
    <td>Marie Lu</td>
    <td>Romance, Young Adult, Mystery</td>
    <td>June 11, 2024</td>
    <td>3.92</td>
    <td>82852</td>
  </tr>
  <tr>
    <td>The Twisted Ones</td>
    <td>Natalie Z. Jones</td>
    <td>Fantasy, Young Adult, Adult, Sci-Fi</td>
    <td>February 13, 2024</td>
    <td>3.92</td>
    <td>118521</td>
  </tr>
  <tr>
    <td>The Excitements</td>
    <td>C.J. Wray</td>
    <td>Historical Fiction, Mystery, Historical</td>
    <td>January 30, 2024</td>
    <td>3.93</td>
    <td>88857</td>
  </tr>
  <tr>
    <td>Hurt Mountain</td>
    <td>Angela Crook</td>
    <td>Mystery, Thriller, Mystery Thriller</td>
    <td>February 27, 2024</td>
    <td>3.93</td>
    <td>78349</td>
  </tr>
  <tr>
    <td>A Place for Vanishing</td>
    <td>Ann Frasier</td>
    <td>Horror, Young Adult, Gothic</td>
    <td>January 16, 2024</td>
    <td>3.96</td>
    <td>108387</td>
  </tr>
  <tr>
    <td>Experimondo</td>
    <td>Kate Rorick</td>
    <td>Romance, LGBTQ, Science Fiction</td>
    <td>June 4, 2024</td>
    <td>3.96</td>
    <td>85867</td>
  </tr>
  <tr>
    <td>Still the Sun</td>
    <td>Charlie N. Holmberg</td>
    <td>Fantasy, Romance, Science Fiction</td>
    <td>July 1, 2024</td>
    <td>4.00</td>
    <td>85867</td>
  </tr>
  <tr>
    <td>The Bright Sword</td>
    <td>Lev Grossman</td>
    <td>Fantasy, Historical Fiction, Retellings</td>
    <td>July 16, 2024</td>
    <td>4.01</td>
    <td>184430</td>
  </tr>
  <tr>
    <td>Love at First Book</td>
    <td>Jenn McKinlay</td>
    <td>Romance, Contemporary, Books About Books</td>
    <td>May 14, 2024</td>
    <td>4.01</td>
    <td>95564</td>
  </tr>
  <tr>
    <td>Imaginary</td>
    <td>Sue Bierley</td>
    <td>Fantasy, Romantic Suspense</td>
    <td>January 27, 2024</td>
    <td>4.03</td>
    <td>180877</td>
  </tr>
  <tr>
    <td>Where Sleeping Girls Lie</td>
    <td>Firdaş Anıtk-İlyası</td>
    <td>Mystery, Young Adult, Thriller</td>
    <td>March 14, 2024</td>
    <td>4.03</td>
    <td>133205</td>
  </tr>
  <tr>
    <td>I'm Starting to Worry About This Black Box of Doom</td>
    <td>Jason Pargin</td>
    <td>Science Fiction, Humor, Mystery</td>
    <td>September 24, 2024</td>
    <td>4.05</td>
    <td>128001</td>
  </tr>
  <tr>
    <td>The Door of Doors</td>
    <td>Gareth Owen</td>
    <td>Fantasy, Magical Realism, Travel</td>
    <td>February 11, 2024</td>
    <td>4.05</td>
    <td>173201</td>
  </tr>
  <tr>
    <td>Deeply Though Dead</td>
    <td>Diana Webb</td>
    <td>Mystery, Romantic Suspense, Thriller</td>
    <td>August 6, 2024</td>
    <td>4.05</td>
    <td>118024</td>
  </tr>
  <tr>
    <td>Summer Romance</td>
    <td>Anabel Monaghan</td>
    <td>Romance, Contemporary, Chick Lit</td>
    <td>June 4, 2024</td>
    <td>4.06</td>
    <td>77896</td>
  </tr>
  <tr>
    <td>Fire and Bones</td>
    <td>Kathy Reichs</td>
    <td>Mystery, Thriller, Crime</td>
    <td>August 6, 2024</td>
    <td>4.06</td>
    <td>60889</td>
  </tr>
  <tr>
    <td>Half One Moment</td>
    <td>Linda Moulton</td>
    <td>Mystery, Contemporary, Thriller</td>
    <td>September 10, 2024</td>
    <td>4.06</td>
    <td>122050</td>
  </tr>
  <tr>
    <td>Reckless</td>
    <td>Lauren Roberts</td>
    <td>Fantasy, Romantic Suspense</td>
    <td>July 2, 2024</td>
    <td>4.07</td>
    <td>97628</td>
  </tr>
  <tr>
    <td>Even If It Breaks Your Heart</td>
    <td>Erin Halin</td>
    <td>Romance, Young Adult, Contemporary</td>
    <td>February 6, 2024</td>
    <td>4.08</td>
    <td>79388</td>
  </tr>
  <tr>
    <td>The Hunter</td>
    <td>Tina French</td>
    <td>Mystery, Ireland, Mystery Thriller</td>
    <td>March 5, 2024</td>
    <td>4.09</td>
    <td>138237</td>
  </tr>
  <tr>
    <td>A Secretness Comes to Call</td>
    <td>T. K. Kessler</td>
    <td>Fantasy, Historical, Retellings</td>
    <td>August 15, 2024</td>
    <td>4.10</td>
    <td>95089</td>
  </tr>
  <tr>
    <td>The Missing Witness</td>
    <td>Allison Brennan</td>
    <td>Mystery, Suspense, Thriller</td>
    <td>January 23, 2024</td>
    <td>4.11</td>
    <td>98108</td>
  </tr>
  <tr>
    <td>Eleven</td>
    <td>Michelle Smelter</td>
    <td>Romance, Contemporary, Adult</td>
    <td>January 24, 2024</td>
    <td>4.12</td>
    <td>109791</td>
  </tr>
  <tr>
    <td>By Any Other Name</td>
    <td>Jodi Picoult</td>
    <td>Historical Fiction, Historical, Contemporary</td>
    <td>August 20, 2024</td>
    <td>4.13</td>
    <td>135244</td>
  </tr>
  <tr>
    <td>The Great Act of Juliette Willoughby</td>
    <td>Emily Lloyd</td>
    <td>Romance, Contemporary, Thriller</td>
    <td>June 1, 2024</td>
    <td>4.13</td>
    <td>110655</td>
  </tr>
  <tr>
    <td>I Hope This Doesn't Find You</td>
    <td>Ann Liang</td>
    <td>Romance, Young Adult, Contemporary</td>
    <td>February 6, 2024</td>
    <td>4.16</td>
    <td>71985</td>
  </tr>
  <tr>
    <td>The Day Shelley Woodhouse Wake Up</td>
    <td>Laura Pearson</td>
    <td>Romance, Contemporary, Mystery</td>
    <td>April 6, 2024</td>
    <td>4.18</td>
    <td>78800</td>
  </tr>
  <tr>
    <td>The God of Woods</td>
    <td>Liz Moore</td>
    <td>Mystery, Thriller, Mystery Thriller</td>
    <td>July 2, 2024</td>
    <td>4.18</td>
    <td>118863</td>
  </tr>
  <tr>
    <td>The Devil's Stones</td>
    <td>Lindsey Anne White</td>
    <td>Thriller, Mystery, Thriller</td>
    <td>March 5, 2024</td>
    <td>4.19</td>
    <td>91378</td>
  </tr>
  <tr>
    <td>The Mercy of Gods</td>
    <td>James S.A. Corey</td>
    <td>Science Fiction, Space Opera, Fantasy</td>
    <td>August 6, 2024</td>
    <td>4.20</td>
    <td>114209</td>
  </tr>
  <tr>
    <td>The Ghost Orchid</td>
    <td>Jonathan Kellerman</td>
    <td>Mystery, Crime, Thriller</td>
    <td>February 6, 2024</td>
    <td>4.20</td>
    <td>64841</td>
  </tr>
  <tr>
    <td>Embers in the London Sky</td>
    <td>Sarah Sundin</td>
    <td>Historical Fiction, Christian Fiction, Romance</td>
    <td>February 6, 2024</td>
    <td>4.21</td>
    <td>87676</td>
  </tr>
  <tr>
    <td>Of Elites and Embers</td>
    <td>Erica James</td>
    <td>Fantasy, Romantic, Contemporary</td>
    <td>February 6, 2024</td>
    <td>4.24</td>
    <td>108569</td>
  </tr>
  <tr>
    <td>Disturbing the Dead</td>
    <td>Kelley Armstrong</td>
    <td>Mystery, Time Travel, Historical Fiction</td>
    <td>May 7, 2024</td>
    <td>4.25</td>
    <td>114262</td>
  </tr>
  <tr>
    <td>Wisteria</td>
    <td>Adalya Grace</td>
    <td>Fantasy, Romance, Young Adult</td>
    <td>August 20, 2024</td>
    <td>4.27</td>
    <td>107341</td>
  </tr>
  <tr>
    <td>Think Twice</td>
    <td>Harlan Cohen</td>
    <td>Mystery, Thriller, Mystery Thriller</td>
    <td>May 14, 2024</td>
    <td>4.28</td>
    <td>79221</td>
  </tr>
  <tr>
    <td>Rescue Stars</td>
    <td>Carrie C. Sham</td>
    <td>Fantasy, Romantic, Contemporary</td>
    <td>April 13, 2024</td>
    <td>4.29</td>
    <td>80631</td>
  </tr>
  <tr>
    <td>The Tainted Cup</td>
    <td>Robert Jackson Bennett</td>
    <td>Fantasy, Mystery, Mystery Thriller</td>
    <td>February 6, 2024</td>
    <td>4.31</td>
    <td>110461</td>
  </tr>
  <tr>
    <td>The Briar Club</td>
    <td>Kate Quinn</td>
    <td>Historical Fiction, Mystery, Historical</td>
    <td>July 9, 2024</td>
    <td>4.32</td>
    <td>188275</td>
  </tr>
  <tr>
    <td>The Act of Disappearing</td>
    <td>Nathalia Gower</td>
    <td>Historical Fiction, Mystery, Historical</td>
    <td>May 22, 2024</td>
    <td>4.32</td>
    <td>108051</td>
  </tr>
  <tr>
    <td>Unhinged</td>
    <td>Nick Roberts</td>
    <td>Romance, Romantic Suspense, Suspense</td>
    <td>March 21, 2024</td>
    <td>4.35</td>
    <td>127819</td>
  </tr>
  <tr>
    <td>Exposed</td>
    <td>Brynn Asher</td>
    <td>Romance, Romantic Suspense, Suspense</td>
    <td>March 26, 2024</td>
    <td>4.36</td>
    <td>104163</td>
  </tr>
  <tr>
    <td>The Instruments of Darkness</td>
    <td>John Connolly</td>
    <td>Mystery, Thriller, Horror</td>
    <td>May 7, 2024</td>
    <td>4.41</td>
    <td>127973</td>
  </tr>
  <tr>
    <td>Time's Prey</td>
    <td>John Sanford</td>
    <td>Mystery, Thriller, Horror</td>
    <td>April 1, 2024</td>
    <td>4.42</td>
    <td>86512</td>
  </tr>
  <tr>
    <td>All My Secrets</td>
    <td>Lynn Austin</td>
    <td>Historical Fiction, Christian Fiction, Christian</td>
    <td>February 6, 2024</td>
    <td>4.43</td>
    <td>120629</td>
  </tr>
  <tr>
    <td>The Austrian Bride</td>
    <td>Helen Pursell</td>
    <td>Historical Fiction, Romance, World War II</td>
    <td>January 15, 2024</td>
    <td>4.43</td>
    <td>93443</td>
  </tr>
  <tr>
    <td>Quicksilver</td>
    <td>Callie Hart</td>
    <td>Fantasy, Romantic, Romance</td>
    <td>June 4, 2024</td>
    <td>4.43</td>
    <td>190189</td>
  </tr>
  <tr>
    <td>Heartbeat</td>
    <td>Sharon Sala</td>
    <td>Romance, Romantic Suspense, Mystery</td>
    <td>February 6, 2024</td>
    <td>4.45</td>
    <td>101930</td>
  </tr>
  <tr>
    <td>Random in Death</td>
    <td>J.D. Robb</td>
    <td>Mystery, Romance, Crime</td>
    <td>January 23, 2024</td>
    <td>4.47</td>
    <td>91807</td>
  </tr>
  <tr>
    <td>Heart of Frost and Scars</td>
    <td>Pam Godwin</td>
    <td>Reverse Harem, Dark, Romance</td>
    <td>August 26, 2024</td>
    <td>4.48</td>
    <td>155855</td>
  </tr>
  <tr>
    <td>A Calamity of Souls</td>
    <td>David Baldacci</td>
    <td>Mystery, Thriller, Crime</td>
    <td>April 16, 2024</td>
    <td>4.49</td>
    <td>122787</td>
  </tr>
  <tr>
    <td>Frank and Red</td>
    <td>Mark Haddon</td>
    <td>Contemporary, Family, Humor</td>
    <td>February 11, 2024</td>
    <td>4.52</td>
    <td>10822</td>
  </tr>
  <tr>
    <td>James</td>
    <td>Percival Everett</td>
    <td>Historical, Race, Retellings</td>
    <td>March 19, 2024</td>
    <td>4.53</td>
    <td>58104</td>
  </tr>
  <tr>
    <td>The Women</td>
    <td>Kristin Hannah</td>
    <td>Historical Fiction, Historical, Adult</td>
    <td>February 6, 2024</td>
    <td>4.62</td>
    <td>124703</td>
  </tr>
</table>

Table 16: Metadata of the test set. (Part 2/2)

Reviewer paper summaries: The paper has two core contributions: (a) a dataset (and a framework), called LongStoryEval, for training and testing long-story evaluators, and (b) an LLM-as-a-judge model, called NovelCritique, that is claimed to surpass current SOTA LLM models in long story evaluation w.r.t. human-judgment correlation.
Summary of strengths: The paper's contribution at the dataset front is commendable with for the first time bringing all the major 8 aspects as proposed in other datasets into one. The dataset also attempts to provide grouping of sub-aspects for finer performance analysis if needed. Also, the dataset have been created by curating real reader reviews from GoodReads.
Summary of weaknesses: While there is some merit to the paper, there are some key weaknesses that need to be addressed:

1. Abstract Line 14-20: At this point, it seems that the observation on the subjective aspects leads to some sort of truism. I did not get any new insight from that as it would be quite self-evident. However, as I read along it looks like this observation is not on the ground-truth data but rather on the performance of NovelCritique, which basically tells little as to what the ground-truth is. The results overall are at best weakly correlated.

2. Abstract Line 21: It is unclear at this point what the processing methods are for.

3. Line 190: What does it mean to say overall assessment has to "align" with rating score? How is it ensured? This requires clarity.

4. Line196: It is not clear what is meant by reprocessing.

5. General comment: An accuracy evaluation of the review processor needs to be done.

6. Lines 273-274: The argument demands corresponding ablation on NovelCritique. At least, the way the paper has been organized I could not make it out.

7. General comment: There should be an evaluation of the summarizer.

8. Line 326: Equation needs further elaboration and analysis as to why it achieves the desired adjustment needed because of inherent rating bias. In other words, how exactly is rating bias captured? What is the bound of C? What happens when C=0? It seems under such situation if we consider extreme bias then s_adjust = s_plat. Is that the intent?

9. Lines 337-338: It is unclear whether the Loss is only on the review score or also on the critique?

10. Line 416: It is unclear why these models are not evaluated within the same framework (i.e., plot summaries)? This analysis confuses me because I cannot now align this with the results of Table 2 (rows 2, 3 & 4) where the context length should not be the issue. 

11. Table 2: It seems that Kendall tau correlation is actually weak. Also, a statistical significance test should be computed, together with the significance test of the correlation difference between the models who all are within the same weak correlation band and NovelCritique.

12. General observation: The model architecture of NovelCritique has not been discussed. If the model uses what is mentioned as Backbone models then it is not clear which of the backbone variants performs the best. Also, in that case, NovelCritique is more of a framework than a model. This needs clarification.

13. According to me the prompt template is key to the understanding of a major part of the pipeline and should be part of the main body along with the ablation table, while the example critiques (Figure 4) can be part of Appendix.
Comments, suggestions and typos: 1. MINOR: Table 1 - Caption should be below as per ACL guidelines.

2. Line 326: MINOR: NovelCritique typo error

3. Line 341: MINOR: utilize in smallcap

4. Line 408: MINOR: Shouldn't it be one aspect?
Overall assessment scores: 2
Metareview score: 3
Metareview: 1. Introduced a long story evaluation data, LongStoryEval, that contains 512 book and their ratings and reviews.
2. Designed a hierarchical evaluation structure with 8 main aspects and 20 sub-aspects.
3. Conducted evaluation experiments with various models and various story processing methods (aggregation-based, incremental-updated, and summary-based)
4. Finetuned an 8b model, NovelCritique, using 412 examples, outperforming other out-of-the-box LLMs (including GPT-4o, Deepseek, etc.) by achieving higher correlations with human ratings.
Summary of reasons to publish: 1. A good data resource for studying book-length story evaluation.
2. An interesting and well-structured evaluation schema.
3. Nice experimental results.