A Tale of Evaluating Factual Consistency: Case Study on Long Document Summarization Evaluation
Authors: Yang Zhong, Diane Litman
Accepted at: acl25

From Information to Insight: Leveraging LLMs for Open Aspect-Based Educational Summarization

Yang Zhong
Department of Computer Science
University of Pittsburgh
yaz118@pitt.edu

Diane Litman
Department of Computer Science and LRDC
University of Pittsburgh
dlitman@pitt.edu

Abstract

This paper addresses the challenge of aspect-based summarization in education by introducing Reflective ASPect-based summarization (ReflectASP), a novel dataset that summarizes student reflections on STEM lectures. Despite the promising performance of large language models in general summarization, their application to nuanced aspect-based summaries remains under-explored. ReflectASP eases the exploration of open-aspect-based summarization (OABS), overcoming the limitations of current datasets and comes with ample human annotations. We benchmarked different types of zero-shot summarization methods and proposed two refinement methods to improve summaries, supported by both automatic and human manual evaluations. Additionally, we analyzed suggestions and revisions made during the refinement process, offering a fine-grained study of the editing strategies employed by these methods. We make our models, dataset, and all human evaluation results available at https://github.com/cs329yangzhong/ReflectASP.

1 Introduction

Real-world documents often contain various aspects (Titov and McDonald, 2008), necessitating summaries that respond to specific user interests. While aspect-based summarization (ABS) focuses on shareable subtopics across documents, such as summaries of customer reviews emphasizing the room and location aspects of hotels (Angelidis et al., 2021a), the emergence of open-aspect-based summarization (OABS) (Tan et al., 2020; Yang et al., 2023; Amar et al., 2023) enables the identification of unique aspects for each document and the generation of tailored summaries.

This paper argues for domain-specific aspect construction and appropriate evaluations, focusing on opinions in the educational domain in the form of student reflections. Student reflections provide valuable insights into students’ learning (Menekse et al., 2011; Menekse, 2020; Kim, 2024) and help instructors identify student misconceptions (Aslan et al., 2019; Alrajhi et al., 2021; Jacobs et al., 2022), thereby enabling them to strategize suitable follow-up actions. In the example in Figure 1, 26 students wrote reflections after a physics lecture, which covered different aspects such as Grounding, Current and Circuit, etc. While a prior corpus in this domain (REFLECTSUMM (Zhong et al., 2024)) provides human-written generic summaries, when focusing on a specific aspect such as Grounding, this generic summary merely notes that “The students also found the concept of grounding and current really interesting.” It thus fails to provide instructors with meaningful insights into how students engage with these aspects. For example, details like “they enjoyed how grounding simplifies calculation” could help instructors better prepare for future lectures and ensure that students are following the material effectively. Similarly, for reflections on confusing points, the aspect-based summaries can pinpoint the major sources of confusion. This allows instructors to revisit these topics or provide additional practice examples. Additional examples contrasting generic versus aspect-based reflection summaries are provided in Appendix A.

Student reflections offer a robust domain for OABS research due to their inherent complexity. The aspects can vary across disciplines, e.g., “sorting algorithms” is highly relevant to Computer Science while “in class activities” is relevant across course disciplines. This diversity tests the summarizing model’s capability to capture aspect-specific information within different contexts. Furthermore, students may articulate their reflections differently, even on the same aspect. The model must discern and synthesize both the shared underlying challenges and the distinctive insights unique to each student’s perspective. Additionally, evaluating and improving the capabilities of large language models
Section 2: ReflectASP

INPUT: All Reflections: (26 students)
• I found it interesting using the loop rule to solve the last problem because I made the problem more simple than I initially thought.
• I liked how the professor showed you how to find current in a circuit, it’s got the same value.
• I thought that the applications of grounding and how that could amplify complicated circuits was interesting.
• Grounding simplified & changes the calculation for current
• I found the part where the professor left the class as funny. It really gave us a bit a break in the middle.
• Learning the paths circuits take and what way the current flows I found to be very interesting.
• It’s cool to see how current flows in real wires and in real life.
• I found the concept of Kirchhoff’s concepts was really interesting.
• The grounding was very interesting.
• I thought grounding was interesting. I never understood what grounding meant before.
• Current analysis was interesting. I liked the concept of current points to a circuit.
• Batteries and circuits are intriguing. I like to learn about currents and how energy within batteries works.
• The circuit analysis about the voltage in loops summing to 0.
• I thought your explanation of ammeters and voltmeters was very well done ... how current runs through ammeters and how current doesn’t run through voltmeters and resistances.
• How you can use instruments to measure voltage and current and how these instruments can affect the circuit.
• I thought it was interesting that all grounded points are connected to each other.
• Voltmeters and ammeters are really interesting in that I think it’s cool that we can actually measure these concepts (voltage and current) pretty easily.

Aspect: Current Human-annotated Cluster marked w/ ⚪
Summary:
Many students found the topic of current to be interesting. Students specifically mention how current flows in real life and how grounding changes calculation for current to be interesting. Students also found using ammeters and voltmeters and calculating to measure current to be interesting, and how those instruments can affect current values.

Aspect: Grounding Human-annotated Cluster marked w/ ⚫
Summary:
Grounding was a topic that many students found interesting in today’s lecture. Students also enjoyed how grounding simplified calculations. Students enjoyed seeing the concept apply to circuit analysis. Finally, the idea that all grounded points are connected to each other was found to be interesting by several students.

Prior Work (ReflectSUMM) (general abstractive summary)
The students today found circuits and its components really interesting. They also found ammeters and voltmeters interesting. They also found the concept of grounding and current really interesting, and a couple of students found the loop rule interesting.

Figure 1: An example from our REFLECTASP dataset. For a given collection of reflections and a specified aspect, we provide human-annotated clusters of aspect-related reflections (highlighted in the input) and human-written aspect-based reference summaries (top right). Unlike prior work (REFLECTSUMM), which produces a single generic summary (bottom right), our annotated dataset includes individual summaries for each aspect, accounting for cases where multiple aspects exist within a single input. These aspect-based summaries capture more detailed elements of students’ reflections (italicized) such as enjoyed how grounding simplified calculations in the second summary, offering valuable insights that can help instructors enhance their teaching.

(LLMs) for generating aspect-based summaries in the educational domain is strategically critical. Unlike other text domains potentially compromised by training data leakage (Zhou et al., 2023), the educational data, collected from real-world scenarios and excluded from web sources, offer a more rigorous assessment of model performance. Once more capable LLMs are validated on the task, they can be deployed to real-world learning systems, helping instructors improve lectures and providing platforms for peer learning.

We introduce Reflective ASpect-based summarization (ReflectASP), a novel dataset containing 313 manually annotated data instances, including aspect, source student reflections, annotated aspect-based clusters of student reflections, and human-written abstractive aspect-based summaries. Our dataset addresses the lack of open aspect-based summarization resources in education. Built on real-world application data, it avoids potential data contamination and provides a fairer evaluation and is of similar scale when compared to the test split of similar manually curated datasets (Amplayo et al., 2021; Takeshita et al., 2024). It also features human-annotated abstractive summaries, ensuring natural and coherent text, as well as high-quality annotations of supporting aspect clusters, providing further validation of summary quality.

Using this corpus, we benchmark various zero-shot summarization approaches and propose two refinement methods that leverage LLMs’ capability to self-critic and improve (Madaan et al., 2023; Huang et al., 2023; Welleck et al., 2023). Our experiments, covering multiple LLM backbones, include diverse automatic evaluations and human evaluations to validate the benefits of each approach and provide suggestions for future work in this novel domain and related aspect-based summarization tasks. Finally, we conduct a data-driven analysis of both the refinement suggestions and pre-/post-refinement summaries to identify common strategies used by LLMs to improve aspect-based summary generation, shedding insights for future work in the aspect-based summarization task.

2 Constructing the REFLECTASP Corpus

Dataset Curation. The student reflections in REFLECTASP are a subset of those in REFLECTSUMM (Zhong et al., 2024), which comes with phrase-based, extractive and generic abstractive summaries. For each lecture, the dataset provides a collection of student reflections focusing on interesting or confusing points. Annotators are directed to extract five noun-phrases summarizing the reflections, and mark original student reflections as evidence for their annotated noun phrases. Out of the 782 reflections-summary pairs in the dataset, we construct our dataset by treating all reflections as the multi-document summarization input and the annotated phrases as the aspects. We removed
<table>
  <tr>
    <th>Domain</th>
    <th>Collect.</th>
    <th>Sum. Rew.</th>
    <th>Incl. Ext.</th>
    <th># Test Set</th>
    <th>Word<sub>input</sub></th>
    <th>Doc<sub>input</sub></th>
    <th>Aspect</th>
    <th>Word<sub>sum.</sub></th>
    <th>Novelty-n(1/2/3)</th>
    <th>Comp. Ratio</th>
  </tr>
  <tr>
    <td>FACETSUM</td>
    <td>Scientific News</td>
    <td>A</td>
    <td>No</td>
    <td>x</td>
    <td>6,000</td>
    <td>6,827</td>
    <td>-</td>
    <td>4</td>
    <td>290</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ASPECTNEWS</td>
    <td>News</td>
    <td>M</td>
    <td>No</td>
    <td>x</td>
    <td>400</td>
    <td>248</td>
    <td>-</td>
    <td>4</td>
    <td>115</td>
    <td>0.02/0.23/0.50</td>
    <td>704:1</td>
  </tr>
  <tr>
    <td>SPACE</td>
    <td>Reviews</td>
    <td>M</td>
    <td>Yes</td>
    <td>✓</td>
    <td>150</td>
    <td>14,335</td>
    <td>100</td>
    <td>6</td>
    <td>26</td>
    <td>0.11/0.49/0.69</td>
    <td>30:1</td>
  </tr>
  <tr>
    <td>REFLECTSUM+</td>
    <td>Reviews</td>
    <td>M</td>
    <td>Yes</td>
    <td>✓</td>
    <td>150</td>
    <td>1,002</td>
    <td>10</td>
    <td>18</td>
    <td>30</td>
    <td>0.16/0.58/0.76</td>
    <td>41:1</td>
  </tr>
  <tr>
    <td>ACL-SUM</td>
    <td>Scientific</td>
    <td>M</td>
    <td>Yes</td>
    <td>✓</td>
    <td>300</td>
    <td>915</td>
    <td>-</td>
    <td>3</td>
    <td>22</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>OASUM</td>
    <td>Wikipedia</td>
    <td>A</td>
    <td>No</td>
    <td>x</td>
    <td>112,005</td>
    <td>1,612</td>
    <td>-</td>
    <td>40</td>
    <td>82</td>
    <td>0.11/0.49/0.74</td>
    <td>68:1</td>
  </tr>
  <tr>
    <td>OPENASP</td>
    <td>News</td>
    <td>M</td>
    <td>No</td>
    <td>x</td>
    <td>596</td>
    <td>6,860</td>
    <td>26</td>
    <td>576</td>
    <td>82</td>
    <td>0.11/0.49/0.74</td>
    <td>68:1</td>
  </tr>
  <tr>
    <td>LEXABSUMM</td>
    <td>Legal</td>
    <td>A</td>
    <td>No</td>
    <td>x</td>
    <td>148</td>
    <td>14,357</td>
    <td>-</td>
    <td>50</td>
    <td>251</td>
    <td>0.07/0.49/0.70</td>
    <td>66:1</td>
  </tr>
  <tr>
    <td>REFLECTASP (ours)</td>
    <td>Education</td>
    <td>M</td>
    <td>Yes</td>
    <td>✓</td>
    <td>313</td>
    <td>817</td>
    <td>43</td>
    <td>280</td>
    <td>69</td>
    <td>0.19/0.63/0.84</td>
    <td>12:1</td>
  </tr>
</table>

Table 1: Descriptive statistics comparing prior datasets (top) to REFLECTASP with their test split. The first five are on ABS, and the others belong to OABS. For the Collection method of aspect-based summaries, A denotes “Automatic” and M denotes “Manual.” We distinguish human-written aspect-based summaries (Sum. Rew.) from those extracted from the input or generic summaries. Incl. Ext. refers to human-annotated content extracted from the input to support the abstractive summary. # Test Set is the number of instances in the test split (document set + aspect + summary). Doc<sub>input</sub> measures the average number of input reflections/documents/articles. We also report the number of unique aspects (Aspect), input length (Word<sub>input</sub>) and summary length (Word<sub>sum.</sub>) in words. We also report the proportion of novel n-grams absent from the input (Novelty-n) and the input-to-summary length ratio (Comp. Ratio). The dash (-) indicates that the metric is not applicable. Gray rows are comparable to our corpus given the bold features in the last row.

lectures where the number of students was small (fewer than ten students, so summarization isn’t needed) and selected aspect-reflection pairs where at least five students mentioned the phrase. This reduced the total amount of data points from 3908 to 1096, which was further reduced to 1064 by removing phrases on “No Confusion”. There exist 767 distinct aspects, highlighting the open-aspect nature of the dataset compared to other corpora. Our further analysis reveals several distinct groups of phrases. The primary group consists of course-specific terminologies, which vary across different courses and are dependent on the lecture and subject matter (i.e., Newton’s Laws in a Physics course). There are also multiple clusters of phrases that are shareable across different lectures, such as “Assignment related problems”, “Quiz and examination”, along with “Other Statements” and “No Confusions”. We include details in Appendix B.2.

Gold Reference Summaries. We recruit two in-house annotators to annotate a subset of 313 unique aspect-lecture pairs. Annotators were first trained in two batches to understand and grasp the tasks before beginning assigned real jobs. We explored two approaches in constructing the summaries: (1) clustering all reflections and drafting the summary from scratch and (2) extracting aspect-related input and revising on top of a GPT-4 generated summary. A pilot study on ten samples suggested that the second option retained good quality through manual inspection and significantly reduced the time needed to write the summary (from 40 mins to 15 mins per data point). We thus apply the second option to produce the full corpus, with full details in Appendix C. We measure inter-annotator performance in ROUGE (Lin, 2004) (R-1/R-2/R-L), which are 48.1/21.9/35.1 among 90 doubly-annotated instances (of the 313 instances).

Human Edits Analysis. Following the methodology outlined in DCR (Wadhwa et al., 2024), we utilize the Levenshtein distance to quantify the number of edits between the original GPT-4-generated summaries and human-edited versions in Table 2. Specifically, we analyze the edit distance by categorizing operations into deletions, insertions, and substitutions at the word level and report the average normalized proportion of edits across all summary pairs. We divide the number of words that are deleted/inserted/substituted by the original length of the GPT-4 generated summary. Our analysis reveals that deletion is the most frequent edit operation, which aligns with the annotators’ feedback, indicating that GPT-4-generated summaries often include extraneous aspect-unrelated details. Additionally, we observe that approximately 28.27 percent of words are substituted, suggesting that human annotators engage in revisions to improve the quality and relevance of the summaries.

<table>
  <tr>
    <th>Category</th>
    <th>Total Edits</th>
    <th>Del.</th>
    <th>Insert.</th>
    <th>Sub.</th>
  </tr>
  <tr>
    <td>Percentage</td>
    <td>65.17%</td>
    <td>35.20%</td>
    <td>1.70%</td>
    <td>28.27%</td>
  </tr>
</table>

Table 2: Human revision analysis on REFLECTASP by the proportion of word-level edit operations: Deletion (Del.), Insertion (Insert.), and Substitution (Sub.).

Comparison to Other Datasets. Table 1 compares our REFLECTASP to existing ABS corpora, in-
cluding FACETSUM (Meng et al., 2021), ASPECT-NEWS (Ahuja et al., 2022), SPACE (Angelidis et al., 2021a), OPOSUM+ (Amplayo et al., 2021), and ACLSUM (Takeshita et al., 2024), as well as to OABS corpora, including OASUM (Yang et al., 2023), OPENASP (Amar et al., 2023), and LEXABSUMM (T.y.s.s. et al., 2024). We highlighted the datasets that are comparable to ours, focusing on the key features of manual collection, human-rewritten summaries, and including annotated extractive supporting sentences. Among ABS corpora with human-rewritten summaries (which guarantees reference coherence), our dataset’s test size is comparable or larger, making it sufficient for evaluation purposes. In the domain of OABS, our dataset is the only one that includes high-quality human-rewritten summaries along with supporting annotations on the source side. In contrast, OpenASP bypasses the source document, relying instead on manually extracting portions of generic summaries. This approach not only limits its quality to that of the original summaries but also risks compromising text coherence when extracting content from different locations. Additionally, unlike prior work that often relies on extreme compression (e.g., SPACE compressing 14k words into summaries averaging 26 words), our dataset strikes a balance between quality and abstractiveness. Our dataset uniquely focuses on the under-explored educational domain, offering potential real-world applications to enhance teaching performance.

Dataset Analysis. We discuss properties of REFLECTASP that emphasize its underlying diversity from several angles. The input document lengths vary from 39 to 2467 tokens (Figure 6 in the Appendix), averaging 817 words. The summary length ranges from 16 to 145 tokens, with a median input-to-output (compression) ratio of 12:1. The aspect label length ranges from 1 to 8 words, showcasing the diversity of aspects. Measuring novelty-n (See et al., 2017) further confirmed that summaries contain a certain level of abstractiveness by using new words not present in the input (0.19/0.63/0.84 for 1/2/3 grams, respectively). Overall, REFLECTASP requires models to perform well on abstractive forms of summarization. Details for these analyses are in Appendix D.

3 Aspect-based Summarization Task

Given course reflections from one lecture and an aspect such as “Integration”, we experiment with large language models and test how well they can pick up the salient reflections to generate an abstractive aspect-based summary of the reflections.

LLM Backbones. For LLM backbones, we selected different versions of powerful open-sourced models: LLAMA 3-8B (LLAMA3), LLAMA 3.1-8B (LLAMA3.1) and also LLAMA 3.1-70B (Dubey et al., 2024), all with instructed version. We further add proprietary LLMs GPT-3.5, GPT-4 and GPT-4o as strong baselines. Implementation details are in Appendix E.1

Methods. Given different backbone LLMs, we instruct them using combinations of different methods: (1) Baseline uses a basic prompt (see Table 9 in Appendix F). (2) Self-Refine uses a Generate-Suggest-Refine framework to use the model to improve its outputs (details and prompt in Appendix F.2). This design aligns with the prompt-chaining in Sun et al. (2024), which was proven to be effective. (3) DCR (Wadhwa et al., 2024) employs a Detect-Critique-Refine pipeline with models fine-tuned for each phase. We used their released models based on LLAMA3. (4) E2A (mimicking the human annotation instructions) uses an extract-then-abstract approach (Takeshita et al., 2024) by prompting the model to first extract relevant student reflections, then generate the abstractive summary. (5) E2A w/ MC-Refine harnesses a fact checker to help identify the salient errors in the generated summaries and refine accordingly. Given the initial summary generated from E2A, we apply MINICHECK (MC) (Tang et al., 2024) to evaluate the faithfulness of individual sentences, utilizing the system-extracted aspect-relevant reflections. Next, we generate sentence-level error detection and revision suggestions among those detected sentences. However, instead of relying on fine-tuned critique and feedback modules, we instruct the LLM to detect spans within the sentences and provide revision suggestions accordingly. In the end, the LLM is prompted to incorporate all sentence-level revision suggestions to generate the final refined version. We visualize these methods in Figure 2 and include prompts in Appendix F.

Evaluation Metrics. Given the gold references, we measure ROUGE F1s (Lin, 2004) (ROUGE-1 (R-1), ROUGE-2 (R-2), ROUGE-L (R-L)) and BERTScore F1 (Zhang* et al., 2020) (BS). To

1We include additional results for three weaker backbones (LLAMA 2-13B-chat (Touvron et al., 2023), MISTRAL (Mistral-Nemo) (Jiang et al., 2023)) and FLAN-T5 (Chung et al., 2022) in the Appendix.
Figure 2: Left: the exemplification of different approaches on leveraging LLMs in zero-shot open-aspect-based summarization task (§3 and §4). Top Right: the outputs of different refining approaches (DCRand Self-Refine), as well as our proposed approaches (E2A and E2A w/MC-Refine). Bottom Right: an analysis of the revision process, showing the evaluation discrepancies among reference-based evaluation, GPT-based evaluation, and human evaluation.

assess factual accuracy (faithfulness), we report the proportion of summary sentences supported by the documents using the SOTA fact-checker Minicheck (Tang et al., 2024). We report \( \mathbf{MC}_{\text{EXT}} \) and \( \mathbf{MC}_{\text{INPUT}} \), which evaluate faithfulness based on annotated aspect-related reflections and the full lecture reflections, respectively, as grounding documents. Metric details are in Appendix G.

4 Results

This section addresses two research questions:
RQ1. How well do LLMs generate aspect-based summaries of reflections in a zero-shot setting?
RQ2. How does different refinement help with the summarization? We conducted automatic and human evaluations to validate our findings.

4.1 Automatic Reference-based Evaluation

RQ1. Table 3 shows models’ performance with different LLM backbones. Comparing among the baseline prompt methods (lines 1, 7, 13), we observe that stronger and more advanced models generally obtained higher R1, R-L, and BS. This is evident in the faithfulness evaluation scores (\( \mathbf{MC}_{\text{EXT}} \) and \( \mathbf{MC}_{\text{INPUT}} \)). We also note that proprietary LLMs obtain lower automatic scores than the open-sourced LLMs, indicating that their wording might not align with human-written references (rows 19-21). Unlike the findings in ACLSUM (Takehita et al., 2024), we observe that *E2A can help consolidate the aspect-relevant information*. We posit that this is due to the differing compression ratios of the summaries: ours are longer and include aspect-specific details, making them sufficient to evaluate the behavior of different approaches, unlike ACLSUM’s extremely short, one-sentence summaries. Among different variants of LLAMA3 and LLAMA3.1 models (row 1 vs. 4, row 7 vs. 10, and row 13 vs. 16), the E2A approach obtained significant improvements across different metrics. Such gain is more salient in smaller models such as LLAMA3. Additional results are in Appendix H.

RQ2. We first assess the revision effects of the Self-Refine and DCR methods (the second and third rows in each block) across all three LLM backbones. Overall, both obtained worse ROUGE and BERTScore results compared to human-written references. However, while the performance gap for Self-Refine narrows as the model improves (rows 1/2 vs. rows 13/14), DCR exhibits greater performance degradation. The analysis of generated summary length (last two columns in Table 3) shows that DCR tends to aggressively shorten the original summary, likely due to the domain mismatch between its training data (meeting summarization) and our education dataset. 2 Our proposed *E2A w/ MC-Refine* overall generate better summaries compared to the baseline prompting on Llama3.1 and LLama3.1-70B (rows 12/18 vs. 7/13). Moreover, it obtained highest faithfulness scores on LLAMA3.1-70B (row 16 vs. 18) through revision. We attribute it to the strong extractive capability of the LLM

2The differences in content length impact \( \mathbf{MC}_{\text{EXT}} \) and \( \mathbf{MC}_{\text{INPUT}} \), so we omit DCR’s MC scores in the table.
<table>
  <tr>
    <th>ID</th>
    <th>Model</th>
    <th>R-1</th>
    <th>R-2</th>
    <th>R-L</th>
    <th>BS</th>
    <th>MC<sub>EXT</sub></th>
    <th>MC<sub>INPUT</sub></th>
    <th># Sents</th>
    <th># Words</th>
  </tr>
  <tr>
    <td>1</td>
    <td>LLAMA3</td>
    <td>45.99</td>
    <td>18.32</td>
    <td>41.29</td>
    <td>89.89</td>
    <td>39.31</td>
    <td>83.04</td>
    <td>4.15</td>
    <td>105.8</td>
  </tr>
  <tr>
    <td>2</td>
    <td>w/ Self-Refine</td>
    <td>44.16</td>
    <td>15.87</td>
    <td>39.03</td>
    <td>89.58</td>
    <td>36.87</td>
    <td>79.45</td>
    <td>3.28</td>
    <td>88.6</td>
  </tr>
  <tr>
    <td>3</td>
    <td>w/ DCR</td>
    <td>38.67</td>
    <td>13.66</td>
    <td>34.88</td>
    <td>89.65</td>
    <td>31.72</td>
    <td>77.72</td>
    <td>2.77</td>
    <td>47.2</td>
  </tr>
  <tr>
    <td>4</td>
    <td>E2A</td>
    <td><b>48.81*</b></td>
    <td><b>18.21*</b></td>
    <td><b>42.82*</b></td>
    <td><b>90.22*</b></td>
    <td><b>56.59*</b></td>
    <td><b>89.35*</b></td>
    <td>3.12</td>
    <td>74.3</td>
  </tr>
  <tr>
    <td>5</td>
    <td>w/ DCR</td>
    <td>35.02</td>
    <td>14.13</td>
    <td>31.53</td>
    <td>89.70</td>
    <td>31.11</td>
    <td>77.11</td>
    <td>1.86</td>
    <td>30.9</td>
  </tr>
  <tr>
    <td>6</td>
    <td>w/ MC-Refine</td>
    <td>44.69</td>
    <td>17.02</td>
    <td>41.88</td>
    <td>90.01</td>
    <td>53.87*</td>
    <td>87.59*</td>
    <td>2.48</td>
    <td>60.9</td>
  </tr>
  <tr>
    <td>7</td>
    <td>LLAMA3.1</td>
    <td>46.21</td>
    <td>17.57</td>
    <td>41.02</td>
    <td>89.89</td>
    <td>40.72</td>
    <td>85.11</td>
    <td>3.49</td>
    <td>94.8</td>
  </tr>
  <tr>
    <td>8</td>
    <td>w/ Self-Refine</td>
    <td>43.68</td>
    <td>15.29</td>
    <td>38.41</td>
    <td>89.53</td>
    <td>38.98</td>
    <td>79.10</td>
    <td>3.18</td>
    <td>90.1</td>
  </tr>
  <tr>
    <td>9</td>
    <td>w/ DCR</td>
    <td>37.95</td>
    <td>13.70</td>
    <td>34.09</td>
    <td>89.75</td>
    <td>31.72</td>
    <td>77.72</td>
    <td>2.39</td>
    <td>42.5</td>
  </tr>
  <tr>
    <td>10</td>
    <td>E2A</td>
    <td><b>48.81*</b></td>
    <td><b>18.21*</b></td>
    <td><b>42.82*</b></td>
    <td><b>90.22*</b></td>
    <td><b>56.59*</b></td>
    <td><b>89.35*</b></td>
    <td>3.31</td>
    <td>78.7</td>
  </tr>
  <tr>
    <td>11</td>
    <td>w/ DCR</td>
    <td>34.97</td>
    <td>13.78</td>
    <td>31.59</td>
    <td>89.70</td>
    <td>31.11</td>
    <td>77.11</td>
    <td>1.94</td>
    <td>32.4</td>
  </tr>
  <tr>
    <td>12</td>
    <td>w/ MC-Refine</td>
    <td>46.99*</td>
    <td>18.13*</td>
    <td>41.88*</td>
    <td>89.16</td>
    <td>57.35*</td>
    <td>88.85*</td>
    <td>3.25</td>
    <td>75.9</td>
  </tr>
  <tr>
    <td>13</td>
    <td>LLAMA3.1-70B</td>
    <td>47.54</td>
    <td>18.64</td>
    <td>42.12</td>
    <td>90.04</td>
    <td>53.97</td>
    <td>89.48</td>
    <td>3.55</td>
    <td>94.9</td>
  </tr>
  <tr>
    <td>14</td>
    <td>w/ Self-Refine</td>
    <td>46.05</td>
    <td>16.88</td>
    <td>40.37</td>
    <td>89.90</td>
    <td>55.74*</td>
    <td>86.58</td>
    <td>2.93</td>
    <td>82.6</td>
  </tr>
  <tr>
    <td>15</td>
    <td>w/ DCR</td>
    <td>38.25</td>
    <td>14.06</td>
    <td>34.89</td>
    <td>89.89</td>
    <td>31.72</td>
    <td>77.72</td>
    <td>2.45</td>
    <td>42.2</td>
  </tr>
  <tr>
    <td>16</td>
    <td>E2A</td>
    <td><b>48.81*</b></td>
    <td><b>18.21*</b></td>
    <td><b>42.82*</b></td>
    <td><b>90.22*</b></td>
    <td><b>56.59*</b></td>
    <td><b>89.35*</b></td>
    <td>3.51</td>
    <td>84.7</td>
  </tr>
  <tr>
    <td>17</td>
    <td>w/ DCR</td>
    <td>34.84</td>
    <td>13.47</td>
    <td>31.25</td>
    <td>89.83</td>
    <td>31.11</td>
    <td>77.11</td>
    <td>1.94</td>
    <td>32.1</td>
  </tr>
  <tr>
    <td>18</td>
    <td>w/ MC-Refine</td>
    <td>47.80</td>
    <td>18.44</td>
    <td>42.40</td>
    <td>90.34*</td>
    <td>71.79*</td>
    <td><b>90.50*</b></td>
    <td>3.43</td>
    <td>79.9</td>
  </tr>
  <tr>
    <th colspan="10">Proprietary LLMs</th>
  </tr>
  <tr>
    <td>19</td>
    <td>GPT3.5-turbo</td>
    <td>35.72</td>
    <td>9.95</td>
    <td>32.05</td>
    <td>88.04</td>
    <td>44.85</td>
    <td>91.25</td>
    <td>4.96</td>
    <td>100.8</td>
  </tr>
  <tr>
    <td>20</td>
    <td>GPT4</td>
    <td>33.51</td>
    <td>7.08</td>
    <td>29.06</td>
    <td>87.50</td>
    <td>51.30</td>
    <td>89.82</td>
    <td>4.05</td>
    <td>97.3</td>
  </tr>
  <tr>
    <td>21</td>
    <td>GPT4o</td>
    <td>34.15</td>
    <td>7.93</td>
    <td>30.48</td>
    <td>87.52</td>
    <td>58.07</td>
    <td>91.57</td>
    <td>4.89</td>
    <td>102.9</td>
  </tr>
  <tr>
    <td>32</td>
    <td>Human (Oracle)</td>
    <td>N/A</td>
    <td></td>
    <td></td>
    <td>76.97</td>
    <td>87.42</td>
    <td></td>
    <td>3.49</td>
    <td>69.2</td>
  </tr>
</table>

Table 3: Experimental results on ReflectASP. All results are averaged over three runs. Gray rows indicate the baseline models, and * means the score is significantly better than the baseline models within each block. The best score for each backbone is bold. Light colored cells are not directly comparable to other cells.

(LLAMA3.1-70B obtains 79.6 R-L in extracting the supporting aspect-based reflections, comparable to human extracting performance).3

Prior studies have shown that some models, despite achieving higher human ratings, may under-perform on reference-based metrics (Zhang et al., 2023; Wadhwa et al., 2024). This motivates us to pursue further evaluations, incorporating both human assessments and analyses assisted by large language models (LLMs) in the following section.

4.2 Automatic GPT-based Evaluation

To overcome the drawbacks of lexical-based metrics, following Wadhwa et al. (2024), we include multiple GPT-4 based evaluation metrics: GPT-4 Factuality Likert Scale Score (G), which uses GPT-4 to score generations when provided a well-defined rubric (Li et al., 2024). We scored the refined/new generation and the initial/baseline output individually and also report the score differences \( \Delta G \). Additionally, we compute the pairwise score difference (Pair \( \Delta G \)) between refined/new generations and initial/baseline outputs and use them to determine the fractions of Wins (\( W \)), Same Scores (\( S \)), and Losses (\( L \)). The order of responses is randomized during the evaluation. Scoring prompts and metric details are in Appendix G.

We report the GPT-based evaluation results for LLAMA3 and LLAMA3.1-70B in Table 4 to compare between the weakest and strongest open-sourced models.4 A more complete table can be found in Appendix H.2. Regarding RQ1, similar to the findings in §4.1, according to the scoring rubric, LLMs can generate summaries that are “overall factually consistent, with a few inconsistencies with the source materials” (rounded to 3), and E2A approaches can further improve. On RQ2, unlike the trend observed in reference-based methods, all refined approaches are found to bring performance gains when compared to the baseline with the simple prompt (as evidenced by the positive values of Pair \( \Delta G \) and Win rate \( W \)). These improvements are more profound in the smaller model. While DCR tends to truncate contents more aggressively, it enhances the faithfulness of generated summaries, as indicated by the significant value on Pair \( \Delta G \) and Win rate. However, it is also worth noting the larger Lose rate compared to other refinement approaches, suggesting that this approach can not constantly improve the summary’s quality. Our proposed E2A w/ MC-Refine approach obtained the highest GPT-4 rating, and pair-wise test (last block) indicated significant improvements.

4.3 Human Evaluation

We conduct human evaluations on the generated summaries. Using Amazon Mechanical Turk, we randomly selected 50 samples from ReflectASP and collected annotations on summaries generated

3We include detailed analysis in Appendix H.3.

4Due to the high cost of calling GPT-4o APIs (each model evaluation costs more than 5 dollars), we opt for one run and report the significance through paired bootstrap test.
<table>
  <tr>
    <th>Model</th>
    <th>G↑</th>
    <th>ΔG↑</th>
    <th>Pair ΔG↑</th>
    <th>W↑</th>
    <th>S</th>
    <th>L</th>
  </tr>
  <tr>
    <th colspan="7">Pairwise Comparison with Baseline Summary as the Original Input</th>
  </tr>
  <tr>
    <td>LLAMA3</td>
    <td>2.74</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>w/ Self-Refine</td>
    <td>2.74</td>
    <td>0.00</td>
    <td>0.10*</td>
    <td>0.19</td>
    <td>0.71</td>
    <td>0.10</td>
  </tr>
  <tr>
    <td>w/ DCR</td>
    <td>2.78</td>
    <td>0.02</td>
    <td>0.25*</td>
    <td>0.40</td>
    <td>0.46</td>
    <td>0.14</td>
  </tr>
  <tr>
    <td>E2A</td>
    <td>2.85*</td>
    <td>0.11*</td>
    <td>0.28*</td>
    <td>0.35</td>
    <td>0.57</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>E2A w/ MC-Refine</td>
    <td><b>2.87</b></td>
    <td><b>0.11*</b></td>
    <td><b>0.31*</b></td>
    <td>0.35</td>
    <td>0.59</td>
    <td>0.06</td>
  </tr>
  <tr>
    <th colspan="7">Pairwise Comparison with E2A Summary as the Original Input</th>
  </tr>
  <tr>
    <td>LLAMA3-E2A</td>
    <td>2.85</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>w/ MC-Refine</td>
    <td>2.87</td>
    <td>0.02</td>
    <td>0.14*</td>
    <td>0.24</td>
    <td>0.65</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>LLAMA3.1-70B E2A</td>
    <td>2.87</td>
    <td>0.02</td>
    <td>0.11*</td>
    <td>0.19</td>
    <td>0.73</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>w/ MC-Refine</td>
    <td><b>2.91*</b></td>
    <td><b>0.04*</b></td>
    <td><b>0.14*</b></td>
    <td>0.20</td>
    <td>0.73</td>
    <td>0.07</td>
  </tr>
</table>

Table 4: GPT-related evaluation results of different methods. Within each block, pairwise metrics compare the outputs of the given system to those in the highlighted rows. A ↑ indicates significant improvement over the previous row (p<0.05) based on a paired bootstrap test, while * denotes that the absolute value is significantly different from zero.

<table>
  <tr>
    <th>ID</th>
    <th>Model</th>
    <th>Rel. to A (1-3)</th>
    <th>Consis. (1-3)</th>
    <th>Asp. Faithful.<br>Fully<br>Part.<br>Not</th>
  </tr>
  <tr>
    <td>1</td>
    <td>LLAMA3</td>
    <td>2.57</td>
    <td>2.56</td>
    <td>75.7%<br>24.3%<br>N/A</td>
  </tr>
  <tr>
    <td>2</td>
    <td>w/ Self-Refine</td>
    <td>2.61</td>
    <td>2.56</td>
    <td>76.1%<br>23.9%<br>0.6%</td>
  </tr>
  <tr>
    <td>3</td>
    <td>E2A</td>
    <td>2.60</td>
    <td>2.58</td>
    <td>77.2%<br>22.1%<br>0.7%</td>
  </tr>
  <tr>
    <td>4</td>
    <td>w/ MC-Refine</td>
    <td>2.63</td>
    <td>2.69</td>
    <td>77.6%<br>22.4%<br>N/A</td>
  </tr>
  <tr>
    <td>5</td>
    <td>LLAMA3.1-70B</td>
    <td>2.52</td>
    <td>2.52</td>
    <td>73.3%<br>26.7%<br>N/A</td>
  </tr>
  <tr>
    <td>6</td>
    <td>w/ Self-Refine</td>
    <td>2.63</td>
    <td>2.56</td>
    <td>76.7%<br>23.3%<br>N/A</td>
  </tr>
  <tr>
    <td>7</td>
    <td>E2A</td>
    <td>2.59</td>
    <td>2.67</td>
    <td>77.5%<br>22.5%<br>N/A</td>
  </tr>
  <tr>
    <td>8</td>
    <td>w/ MC-Refine</td>
    <td>2.66</td>
    <td>2.67</td>
    <td>78.5%<br>21.5%<br>N/A</td>
  </tr>
  <tr>
    <td>9</td>
    <td>GPT3.5</td>
    <td>2.70</td>
    <td>2.72</td>
    <td>76.6%<br>22.6%<br>0.8%</td>
  </tr>
  <tr>
    <td>10</td>
    <td>GPT4</td>
    <td>2.61</td>
    <td>2.60</td>
    <td>84.5%<br>15.0%<br>0.5%</td>
  </tr>
</table>

Table 5: Human evaluation results. Relevance to Aspect (Rel. to A.) assesses whether the summary discusses the aspect exclusively (3), partially (2), or not at all (1). Consistency determines whether the facts in the summary are consistent with the facts in the original input from fully (3) to not supported (1). Additionally, we report the aspect-based sentence-level faithfulness (Asp. Faithful.), which measures the proportion of sentences that are Fully/Partially/Not supported by the annotated aspect-focused reflections.

by ten different systems. For each sample, three annotations were obtained (thus a total of 1500 annotations), with document-level metrics (Relevance to Aspect and Consistency) reported as averages and sentence-level annotations determined by majority vote. We selected two baseline models LLAMA3-8B and LLAMA3.1-70B. To investigate the effects of different approaches, we conducted a comparative analysis of summaries generated by the raw baseline, E2A, Self-Refine and E2A w/ MC-Refine outputs. We additionally include GPT3.5 and GPT4. All systems are anonymized. Annotation details and interface are in Appendix I.

Table 5 shows the performance of different approaches. Consistent with the results in Table 3 and Table 4, different approaches improved the initial summary. The E2A approach obtained performance gains on all three metrics (row 1/5 vs. row 3/7). Unlike the drastic reference-based performance gap between the original and self-refined version, human raters assigned higher relevance scores to self-refined summaries, suggesting that the revisions can help improve the aspect-relevance (rows 2 and 6). Our introduced E2A w/ MC-Refine (rows 4 and 8) obtained the best performances on both backbone LLMs, improving both relevance and consistency of the contents, which aligns with the observations from the faithfulness metrics (Table 3) and GPT-4 based evaluations (Table 4). We observed that sentence-level aspect-based faithfulness evaluations across different models show similar distributions. This differs from the automatic faithfulness scores in Table 3, where applying E2A on MCEXT (rows 1-3) improved scores by over 17 points, compared to a 1.5% gain in "Fully Supported" scores for LLAMA3. We attribute this discrepancy to the complexity of sentence-level annotations and the relatively small sample size used in human evaluations compared to the full test set.

5 Analysis of Summary Revisions

GPT-based and human evaluations suggested that revised summaries become more relevant to the aspect, but how do different refinement approaches help with it? In this section, we present a data-driven study on document-level revisions, aiming to understand what common strategies LLMs use in different refinement approaches. To examine the modifications made by different refinement approaches, we run an automated system (Jiang et al., 2022) to extract edits and determine their underlying intentions. This model (trained on scientific paper revisions) compromises sentence alignment, edit extraction, and intention classification modules. The taxonomy and model details are in Appendix J.1. Figure 3 provides an illustrative example.

We start by exploring the dynamics of sentence-level edit operations, aiming to understand how LLMs modify sentences during refinement. As shown in Figure 4, an individual LLM exhibits different behaviors with different refinement strategies. On Llama3.1-70B, DCR favors more proportion of deletions compared to the other approaches
Figure 3: An illustration of edits analysis. Summaries are also in bottom right of Figure 2.

Figure 4: Distribution of edit actions among revised sentences.

Figure 5: Distribution of span-level edit intentions during different refinements. We include the count of edits in parenthesis.

(more than 50% of edit operations), which explains its reduced performance on automatic metrics and the shorter content length. For the smaller LLAMA3, Self-Refine makes more adding edits compared to the other approaches, potentially introducing details from the original reflections that are not covered by the human reference summaries. When comparing LLAMA3.1-70B and LLAMA3, our proposed E2A w/ MC-Refine approach demonstrates differing proportions of deletion and addition edits, which we attribute to the varying capabilities of LLMs. We include additional analysis on the refinement suggestions and the linguistic features of summaries in Appendices J.2 and J.3.

We also analyzed the edit intentions on all revised sentences between the original and refined summaries. The distribution of the intentions is visualized in Figure 5. Most edits are categorized as content updates. Notably, DCR exhibits the fewest edits overall (i.e., for LLAMA3.1-70B, DCR contains 856 edits, way less than the other two approaches with around 1900 edits). This matches with previous findings that DCR tends to remove content. Additionally, DCR contains higher proportion of style improvements compared to the other two methods, which does not alter the meaning. Self-Refine instead contains the least proportion of content updates, and includes more grammar typo fixes. The E2A w/ MC-Refine approach has a smaller portion of edits for improving the simplicity (4.5% and 4.2%) and improve format (4.3% and 3.4%), which aligns with the goal of enhancing the quality of aspect-based summaries.

6 Related Work

Open Aspect-based Summarization. Recent work introduced multiple OABS datasets (Tan et al., 2020; Amar et al., 2023; Guo and Vosoughi, 2024), where aspects are document-based. Our corpus is the first to have the unique combination of ample human-annotated sentences from the source document with carefully crafted, human-rewritten aspect-based summaries. While proprietary LLM (GPT-3.5) started demonstrating zero-shot capability in performing OABS task (Amar et al., 2023; Guo and Vosoughi, 2024; Mukku et al., 2024), the capabilities of more accessible open-sourced LLMs remain under-explored. Additionally, much of the prior work focuses on domains like news and product reviews, which could potentially be influenced by contamination in the LLMs’ training process. Our study explores the use of open-source LLMs for OABS on a novel dataset featuring diverse, document-dependent aspects in the educational domain, evaluating their performance through comprehensive approaches beyond basic prompting..

LLM Feedback and Refinement. Generating feedback at inference-time is essential for LLMs to refine their answers (Madaan et al., 2023; Welleck et al., 2023; Zheng et al., 2023a). More recently, researchers (Huang et al., 2023; Kamoi et al., 2024; Palmeira Ferraz et al., 2024) noticed that LLMs may struggle with self-correction without external guidance. In summarization, prior work (Zhang et al., 2023) leverages GPT-3.5 to iteratively revise
summaries to improve the factuality and controllability in news articles. Wadhwa et al. (2024) proposed a specialized "Detect-Critique-Refine" pipeline, which incorporates fine-tuned critique and feedback models to enhance the factuality of refined summaries. We employ open-sourced LLMs to generate feedback based on minimal instructions, leveraging the extractive power of the model and an external fact-checker to better localize errors, to produce better summaries on a given aspect.

7 Discussion

Dataset Characteristics. We carefully compared our dataset to existing ABS and OABS corpora in §2 Table 1, showing that our dataset is larger than most manually curated corpora for evaluation purposes. Our annotations are based on publicly available real-world application data (Reflect-Summ) released in prior work (Zhong et al., 2024), which covers four distinct subjects and spans over 20 courses, offering a rich and diverse testbed. Regarding the size of collected reflections per lecture, our survey of undergraduate course sizes at three well-known universities in the Northeastern US—including the institution where the original Reflect-Summ dataset was collected (details in Appendix L)—indicates that approximately 84% of undergraduate courses have 50 or fewer students. Given the self-volunteering nature of data collection and variations in attendance throughout the semester, our dataset, with an average of 43 student reflections per lecture, represents the majority of real-world scenarios. Lastly, collecting annotations from in-house annotators is both time-consuming (15 mins per instance) and expensive (suppose the annotator is paid $15 per hour, each annotated summary costs around $4).

Out-of-Domain Testing and Insights. In addition to our REFLECTASP corpus, we further evaluated our proposed approaches on an out-of-domain dataset, SPACE (Angelidis et al., 2021a), by using their testing split, which contains reviews about 25 unique hotels and covers six popular aspects. We compare our approaches to the strong baseline systems released in prior work (Hosking et al., 2023). We find that the MC-Refine approach substantially outperformed other methods (details in Appendix K). A close examination of both SPACE and our datasets, together with manual verification of the generated summaries from different systems, provides us with valuable insights into the advantages of our datasets. Firstly, our dataset’s high-quality annotation resolves the inconsistency between reference-based metrics and the faithfulness evaluation. In contrast, older datasets suffer from their noisy annotation and heuristic-based construction, leading to inconsistent performance. Moreover, our dataset contains richer information in the summaries that well summarize the input. Lastly, our dataset motivates innovations in faithfulness-guided approaches by identifying two main challenges in aspect-based summarization: (a) the inclusion of aspect-unrelated details and (b) the oversimplification of aspect-specific content. We provide further justifications in Appendix K.4.

8 Conclusion

In this work, we contribute REFLECTASP, the first open-aspect-based summarization dataset in the educational domain, with 313 high-quality aspect-based summaries and annotated supporting clusters. This dataset offers rich coverage of various aspects, diverse inputs and outputs, and an abstractive nature to enhance human understanding. We extensively test open-sourced LLMs in zero-shot fashions. Our introduced extract-then-abstract (E2A) and refinement approaches improved LLMs’ capability in generating more focused summaries, of which results are verified through rigorous automatic and human evaluations. Lastly, our analysis of revisions across different text versions reveals the techniques used by various refinement approaches, offering insights for future innovations in aspect-based summarization.

Acknowledgement

The research reported here was supported in part by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A180477, and the National Science Foundation through Grants 2329273 and 2329274. This work is also supported by LRDC. This research was supported in part by the University of Pittsburgh Center for Research Computing and Data, RRID:SCR_022735, through the H2P cluster, which is supported by NSF award number OAC-2117681. We want to thank the members of the Pitt PETAL group, the Pitt NLP group, the CourseMIRROR group, and anonymous reviewers for their valuable comments in improving this work. We also thank Alfa Satya Putra and Ali Alhaddad from Purdue University for their annotation efforts.
Limitations

Methodologies. This study leverages existing student-written reflections and utilizes the generative power of LLMs to produce and refine aspect-based summaries. Although this approach was effective for the specific educational dataset we used, it may not be readily applicable to different datasets. Also, our reliance on conducting experiments in a zero-shot manner may hinder the model from understanding the meaning of the prompts and thus fail to produce higher-quality revisions.

Is GPT-based Evaluation Reliable? While there exists work on prompting GPT models to rate the summaries (Liu et al., 2023; Zhang et al., 2023; Dubois et al., 2023; Wadhwa et al., 2024), we acknowledge that leveraging LLM as an evaluator may carry biases and the model could favor its own output (Zheng et al., 2023b; Panickssery et al., 2024). In this work, we use the proprietary LLM, GPT4o, as the evaluator to measure the quality of open-sourced models like LLAMA to avoid the potential self-preference. The evaluation prompts are adopted from previous work (Wadhwa et al., 2024).

Manual Annotation Challenges. Curating human-annotated datasets presents challenges. As summarized in Table 1, an examination of existing datasets reveals a reliance on automatic collection and alignment methods, such as the heuristic selection of Wikipedia abstracts for OASUM (Yang et al., 2023). This approach often results in incoherent reference summaries. While some work, such as AspectNews (Ahuja et al., 2022) and OpenASP (Amar et al., 2023), repurposed the generic summary through human annotations on aspect labels of sentences, they acknowledged the high cost of such annotations. SPACE (Angelidis et al., 2021a) produced abstractive summarization, proposing a multi-stage pipeline to identify salient sentences and then produce generic/aspect summaries with the help of off-the-shelf classifiers. They reported low inter-annotator agreement. Instead, our dataset came with human-annotated aspect-based clusters with aspect values, which provide more grounding for our collected high-quality human-based abstractive summaries.

Dataset Scale. We acknowledge that our dataset only has a test set but no training split. To enhance the future work, we plan to include continuing annotation on the remainder of the unannotated portion of the 1064 aspect-lecture pairs curated in S2, as well as employing LLMs in synthesizing large-scale training data leveraging similar educational datasets. Extending our work to other types of classes such as MOOCs would be an exciting direction parallel to our current studies but with a focus on aiming to handle longer inputs (hundreds and even thousands of reflections) and much noisier input as the course takers typically come from different backgrounds.

Utilization of LLM Output as the Summary Reference Draft. LLMs have been proven to produce high-quality summaries from a human perspective. Goyal et al. (2022) found that summaries crafted by GPT-3 are preferred over those from state-of-the-art (SOTA) fine-tuned models, despite the latter achieving higher scores in reference-based evaluations against human-produced summaries. Furthermore, Pu et al. (2023) found that human evaluators significantly prefer summaries generated by GPT-4, outperforming human-generated summaries and summaries generated by fine-tuned models from multiple perspectives. Our dataset is created as a hybrid of LLM output and human revision, combining the precision of human judgment with the generative capabilities of LLMs to streamline the lengthy process of initial drafting, similar to Liu et al. (2024).

Other Perspectives of Human-Evaluated Text Quality. Recent advanced LLMs like GPT-4 are found to produce fluent texts. Besides the mentioned “Relevance to Aspect” annotation, in an older version of the annotation, we prompted the annotators to provide a binary label for fluency, observing that two in-house annotators annotated over 95% of summaries as fluent. This finding aligns with Zhang et al. (2024) and Amar et al. (2023), who found that LLMs generated fluent texts. In our reported human annotation, we introduced both Consistency and Aspect-based Factuality (Sentence-level) to account for the much more challenging aspect of assessing the factuality of AI-generated summaries. Recent work (Hosking et al., 2024) also demonstrated that human preference scores can under-represent aspects such as factuality, presenting the challenges in better evaluating text qualities in the era of LLMs.

Ethical Considerations

Abstractive summarization models have been found to contain hallucinated artifacts that do not faithfully represent the source texts. Regarding the user-sensitive information within the dataset, we do not
see concerns about applying our model, as user-specific information will not be included in the students’ reflections. The original ReflectSumm dataset was created with students’ consent, ensuring their responses could be collected and used for research purposes. We acknowledge the potential for bias in human annotation, particularly in the context of abstractive summaries and crowd-sourced summary evaluations. This is due to the majority of our crowd-sourcing annotators being based in the U.S. However, no private information is collected from the annotators. We only collect annotator’s input on the refined summary, as well as evaluations on the qualities. Lastly, the authors acknowledge the use of Grammarly and GPT-4o for correcting sentences that are less fluent but not for generating or drafting new content.

References

2024. The Role of Reflection-Informed Learning and Instruction in an Introductory Physics Course for Engineering and Science Students. Zenodo.

Ojas Ahuja, Jiacheng Xu, Akshay Gupta, Kevin Horecka, and Greg Durrett. 2022. ASPECTNEWS: Aspect-oriented summarization of news documents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6494–6506, Dublin, Ireland. Association for Computational Linguistics.

Laila Alrajhi, Ahmed Alamri, Filipe Dwan Pereira, and Alexandra Ioana Cristea. 2021. Urgency analysis of learners’ comments: An automated intervention priority model for mooc. In International Conference on Intelligent Tutoring Systems.

Shmuel Amar, Liat Schiff, Ori Ernst, Asi Shefer, Ori Shapira, and Ido Dagan. 2023. OpenAsp: A benchmark for multi-document open aspect-based summarization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1967–1991, Singapore. Association for Computational Linguistics.

Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021. Aspect-controllable opinion summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6578–6593, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2021a. Extractive Opinion Summarization in Quantized Transformer Spaces. Transactions of the Association for Computational Linguistics, 9:277–293.

Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2021b. Extractive opinion summarization in quantized transformer spaces. Transactions of the Association for Computational Linguistics, 9:277–293.

Stefanos Angelidis and Mirella Lapata. 2018. Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3675–3686, Brussels, Belgium. Association for Computational Linguistics.

Sinem Aslan, Nese Alyuz, Cagri Tanriover, Sinem E. Mete, Eda Okur, Sidney K. D’Mello, and Asli Arslan Esme. 2019. Investigating the impact of a real-time, multimodal student engagement analytics technology in authentic classrooms. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, page 1–12, New York, NY, USA. Association for Computing Machinery.

Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.".

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models.

Noam Dahan and Gabriel Stanovsky. 2025. The state and fate of summarization datasets: A survey. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7259–7278, Albuquerque, New Mexico. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. In Thirty-seventh Conference on Neural Information Processing Systems.

Sebastian Gehrmann, Tosin Adewumi, Karmanyaa Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu,
Dipanjian Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinene Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahmood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Nirajan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96–120. Online. Association for Computational Linguistics.

Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356.

Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708–719, New Orleans, Louisiana. Association for Computational Linguistics.

Xiaobo Guo and Soroush Vosoughi. 2024. Disordered-DABS: A benchmark for dynamic aspect-based summarization in disordered texts. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 416–431, Miami, Florida, USA. Association for Computational Linguistics.

Tom Hosking, Phil Blunsom, and Max Bartolo. 2024. Human feedback is not gold standard. In The Twelfth International Conference on Learning Representations.

Tom Hosking, Hao Tang, and Mirella Lapata. 2023. Attributable and scalable opinion summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8488–8505, Toronto, Canada. Association for Computational Linguistics.

Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. ArXiv, abs/2310.01798.

Jennifer Jacobs, Karla Scornavacco, Charis Harty, Abhijit Suresh, Vivian Jia Yin Lai, and Tamara R. Sumner. 2022. Promoting rich discussions in mathematics classrooms: Using personalized, automated feedback to support reflection and instructional change. Teaching and Teacher Education.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Chao Jiang, Wei Xu, and Samuel Stevens. 2022. arXivEdits: Understanding the human revision process in scientific writing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9420–9435, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When can llms actually correct their own mistakes? a critical survey of self-correction of llms. Transactions of the Association for Computational Linguistics, 12:1417–1440.

Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Linmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao, and Shuai Ma. 2024. Leveraging large language models for NLG evaluation: Advances and challenges. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16028–16045, Miami, Florida, USA. Association for Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Linguistics.

Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. 2024. Benchmarking generation and evaluation capabilities of large language models for instruction controllable summarization. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4481–4501, Mexico City, Mexico. Association for Computational Linguistics.
Wencan Luo and Diane Litman. 2015. Summarizing student responses to reflection prompts. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1955–1960, Lisbon, Portugal. Association for Computational Linguistics.

Wencan Luo, Fei Liu, and Diane Litman. 2016. An improved phrase-based approach to annotating and summarizing student course responses. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 53–63, Osaka, Japan. The COLING 2016 Organizing Committee.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. Neurips.

Muhsin Menekse. 2020. The reflection-informed learning and instruction to improve students’ academic success in undergraduate classrooms. The Journal of Experimental Education, 88(2):183–199.

Muhsin Menekse, Glenda Stump, Stephen Krause, and Micheline Chi. 2011. The effectiveness of students’ daily reflections on learning in an engineering context. pages 22.1451.1–22.1451.10.

Rui Meng, Khushboo Thaker, Lei Zhang, Yue Dong, Xingdi Yuan, Tong Wang, and Daqing He. 2021. Bringing structure into summaries: a faceted summarization dataset for long scientific documents. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1080–1089, Online. Association for Computational Linguistics.

Sandeep Sricharan Mukku, Abinesh Kanagarajan, Chetan Aggarwal, and Promod Yenigalla. 2024. MARS: Multilingual aspect-centric review summarisation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 894–909, Miami, Florida, US. Association for Computational Linguistics.

Thomas Palmeira Ferraz, Kartik Mehta, Yu-Hsiang Lin, Haw-Shiuan Chang, Shereen Oraby, Sijia Liu, Vivek Subramanian, Tagyoung Chung, Mohit Bansal, and Nanyun Peng. 2024. LLM self-correction with DeCRIM: Decompose, critique, and refine for enhanced following of instructions with multiple constraints. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7773–7812, Miami, Florida, USA. Association for Computational Linguistics.

Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM evaluators recognize and favor their own generations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830.

Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023. Summarization is (almost) dead. ArXiv, abs/2309.09558.

Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.

Shichao Sun, Ruifeng Yuan, Ziqiang Cao, Wenjie Li, and Pengfei Liu. 2024. Prompt chaining or stepwise prompt? refinement in text summarization. In Findings of the Association for Computational Linguistics ACL 2024, pages 7551–7558, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Sotaro Takeshita, Tommaso Green, Ines Reinig, Kai Eckert, and Simone Ponzetto. 2024. ACLSum: A new dataset for aspect-based summarization of scientific publications. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6660–6675, Mexico City, Mexico. Association for Computational Linguistics.

Bowen Tan, Lianhui Qin, Eric Xing, and Zhiting Hu. 2020. Summarizing text on any aspects: A knowledge-informed weakly-supervised approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6301–6309, Online. Association for Computational Linguistics.

Liyan Tang, Philippe Laban, and Greg Durrett. 2024. MiniCheck: Efficient fact-checking of LLMs on grounding documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8818–8847, Miami, Florida, USA. Association for Computational Linguistics.

Ivan Titov and Ryan McDonald. 2008. A joint model of text and aspect ratings for sentiment summarization. In Proceedings of ACL-08: HLT, pages 308–316, Columbus, Ohio. Association for Computational Linguistics.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu,
Jude Fernandes, Jeremy Fu, Wenying Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models.

Santosh T.y.s.s., Mahmoud Aly, and Matthias Grabmair. 2024. LexABSumm: Aspect-based summarization of legal decisions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10422–10431, Torino, Italia. ELRA and ICCL.

Manya Wadhwa, Xinyu Zhao, Junyi Jessy Li, and Greg Durrett. 2024. Learning to refine with fine-grained natural language feedback. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12281–12308, Miami, Florida, USA. Association for Computational Linguistics.

Shufan Wang, Laure Thompson, and Mohit Iyyer. 2021. Phrase-BERT: Improved phrase embeddings from BERT with an application to corpus exploration. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10857–10851, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations.

Xianjun Yang, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Xiaoman Pan, Linda Petzold, and Dong Yu. 2023. OASum: Large-scale open domain aspect-based summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4381–4401, Toronto, Canada. Association for Computational Linguistics.

Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023. SummIt: Iterative text summarization via ChatGPT. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10644–10657, Singapore. Association for Computational Linguistics.

Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024. Benchmarking Large Language Models for News Summarization. Transactions of the Association for Computational Linguistics, 12:39–57.

Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023a. Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023b. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Yang Zhong, Mohamed Elaraby, Diane Litman, Ahmed Ashraf Butt, and Muhsin Menekse. 2024. ReflectSumm: A benchmark for course reflection summarization. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13819–13846, Torino, Italia. ELRA and ICCL.

Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. Don’t make your llm an evaluation benchmark cheater.
A Examples Comparing General and Aspect-based Summaries

In Table 6, we provide more examples from our datasets, where student reflections on “what they find most interesting / confusing” are well summarized in the aspect-based summaries.

B Aspect Analysis

B.1 Open-Aspect Extraction for OABS

We acknowledge that aspect-based summarization, query-based summarization, and keyword-controlled summarization are intermingled and hard to fully separate. While we agree that one can treat the aspect as a keyword that is extracted from the source reflections, open-aspect summarization fits better in this case, as the aspects can be either specific to the source document (i.e., course concepts) or generalizable across different documents such as “Homework” or “Exams.” More details on aspect analysis are documented in Appendix B.2. Similar to the aspect-based summarization setup in OASUM (Yang et al., 2023), Space and OPPSUM+ (Amplayo et al., 2021), we provided the model with pre-extracted (high-quality human-annotated) aspects in our dataset to guide the generation. For future work, we would like to add baselines that prompt the model to extract aspects independently before generating aspect-based summaries, though it may introduce another layer of complexity. This approach could lead to more contentious evaluations, given the need to guarantee that differing Large Language Models (LLMs) would need to extract consistent aspects with the human-annotated aspects.

Regarding the aspect extraction for real world applications, the ReflectSumm (Zhong et al., 2024) dataset came from a real-world application that has been deployed in real universities. We used their publicly released dataset to conduct our experiments. The process of aspect extraction, conceptualized as phrase-based summarization, has been addressed in existing literature. Luo et al. have outlined systems for phrase-based summarization comprising candidate phrase extraction, phrase clustering, and phrase ranking (Luo and Litman, 2015; Luo et al., 2016). More recent work has also explored applying LLMs to generate phrase summaries from lecture reflections with promising performances (Zhong et al., 2024). Our research primarily investigates LLMs’ capability to produce aspect-based summaries. Thus, we leave the refinement of aspect mining methodologies for future exploration.

B.2 ReflectASP Aspect Analysis

Out of the 1096 phrases collected in Data Curation, 767 are unique. To examine the variations among aspects, we encoded them using PhraseBERT (Wang et al., 2021), followed by the application of the K-means unsupervised clustering algorithm, to organize them into clusters. Our analysis reveals several distinct groups of phrases. The primary group consists of course-specific terminologies, which vary across different courses and are dependent on the lecture and subject matter (i.e., Newton’s Laws in a Physics course). There are also multiple clusters of phrases that are shareable across different lectures, such as “Assignment related problems”, “Quiz and examination”, along with “Other Statements” and “No Confusions”.

The variability of aspects in the first group necessitates open aspects in aspect-based summarization to satisfy the user’s need to learn about interesting/confusing points. Moreover, we observe that reflections tagged with “No Confusion” carry the least amount of information and are deemed superficial. Thus, we excluded the data points with aspects annotated as “No confusion,” reducing the total number of data points to 1064. This refinement helps to focus on more substantive aspects.

The K-means algorithm we used is from the scikit-learn package (Pedregosa et al., 2011)5. The parameters for K-means are { "init": "k-means++", "n_init": 3, "max_iter": 300 }. We search for the best N based on the SSE of cosine similarities. Table 7 is one example of clustering results, with 5 aspects per cluster.

C Human Annotation Details on Reference Summaries

We recruited two in-house annotators (not the authors) to annotate the 313 data points. Both annotators are funded by the project and have taken the courses / possessed the knowledge of course materials covered in the ReflectSumm dataset (Zhong et al., 2024). One of the annotator is a PhD student and the other one is pursing the undergraduate degree.

5https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html
<table>
  <tr>
    <th>General Summary (ReflectSUMM)</th>
    <th>Aspect</th>
    <th>Aspect-based Summary (ReflectASP)</th>
  </tr>
  <tr>
    <td>Many of the students today seemed to struggle with the concepts regarding flux and gauss’ law, as well as gaussian surfaces. Some students also struggled with mathematical calculations, while others struggled with the examples that they were doing in class.</td>
    <td>Gauss Law/Surface</td>
    <td>Students found the topic of Gauss Law and Gaussian surface to be confusing. Although they said that the lesson on flux and Gaussian surface was tied together seamlessly and easier to understand than their high school lessons, students still struggle on certain aspects of the topic. Specifically, students expressed difficulty in choosing a proper Gaussian surface that produces symmetry to allow certain components to cancel out. Students requested more explanations on the topic to help them understand the topic better.</td>
  </tr>
  <tr>
    <td>Most of the responses today included topics about potential and how it confused the students, as well as integrating and setting up their problems that they are given in class. They also had some trouble with some electric field concepts.</td>
    <td>Problem Setup</td>
    <td>Students struggled with the problem set up in this lecture. On the topic of integrals, students said that solving the integration was understandable and not difficult, but they were having problems with setting up the integral. Students were specifically confused about the limits and variables of integration, and how the limit can change midway through the problem. Students stated that they could use more clarity and examples of setting up integrals in different situations.</td>
  </tr>
  <tr>
    <td>Most students were confused about coming up with designing algorithms and writing pseudo-code for the algorithms. Students also were confused about logistics and content of Milestone 2 and graphing in Matlab. A few students were confused about velocity calculations with regression and the contents of the concept quiz.</td>
    <td>Milestone 2</td>
    <td>Many students reported feeling confused about the coding aspect of Milestone 2, specifically regarding which MATLAB functions to use and how to write the pseudo code for the in-class activity. Some students also expressed confusion about the noise aspect of the graphs and how to set the coding for the graphs. Students were also confused about approaching the project and how to best complete the Milestone, and would have appreciated more clear instructions.</td>
  </tr>
</table>

Table 6: Aspect-based summary examples in ReflectASP. In each sample, we highlight the details extracted from students’ reflections, which are helpful for the instructor to better assist students’ learning.

<table>
  <tr>
    <th>ID</th>
    <th>Cluster Size</th>
    <th>Example Aspects</th>
  </tr>
  <tr>
    <td>0</td>
    <td>32</td>
    <td>['excel', 'No Confusion', 'No Confusion', 'No Confusion', 'No Confusion']</td>
  </tr>
  <tr>
    <td>1</td>
    <td>61</td>
    <td>['In-Class Problems', 'In-Class Problems', 'Exam Prep', 'In-class assignments', 'Syllabus', 'Structure of Class']</td>
  </tr>
  <tr>
    <td>2</td>
    <td>69</td>
    <td>['Other Statements', 'Other Statements', 'Other Statements', 'Other Statements', 'Other Statements']</td>
  </tr>
  <tr>
    <td>3</td>
    <td>85</td>
    <td>['Teamwork/Breakout Rooms', 'Capital Investment', 'Groupwork', 'New Project', 'Groupwork']</td>
  </tr>
  <tr>
    <td>4</td>
    <td>88</td>
    <td>['Electric/Uniform Field', 'Energy Calculations and Units', 'Car Carbon Emissions', 'Electric Charges', 'Current/Resistance']</td>
  </tr>
  <tr>
    <td>5</td>
    <td>108</td>
    <td>['Evaluating and citing reliable resources', 'Phone book activity', 'Downloading the file', 'Introduction to the new project', 'Last example question']</td>
  </tr>
  <tr>
    <td>6</td>
    <td>32</td>
    <td>['Assignment 17', 'Assignment 8, A08', 'Assignment 8, A08', 'Assignment 8 OR 5', 'Assignment 13']</td>
  </tr>
  <tr>
    <td>7</td>
    <td>127</td>
    <td>['Redefining Systems', 'Prototyping/Creating Prototypes', 'Engineering Majors', 'Coding', 'Pseudocode and Algorithm']</td>
  </tr>
  <tr>
    <td>8</td>
    <td>62</td>
    <td>['Related to Trig', 'Related to Functions', 'Related to the Quiz', 'Related to the Project', 'Related to Induction']</td>
  </tr>
  <tr>
    <td>9</td>
    <td>50</td>
    <td>['RB BST/Red-Black tree/red-black BST/Red Black BST', 'excel/Excel', '1 vs 100 Sheets Question', 'A10', 'Red Black BST"s']</td>
  </tr>
  <tr>
    <td>10</td>
    <td>118</td>
    <td>['Matlab/matlab/MATLAB', 'Backtracking', 'Porblem scope of the project', 'Breakout Rooms', 'Deck of cards/poker problem']</td>
  </tr>
  <tr>
    <td>11</td>
    <td>107</td>
    <td>['When to use certain graphs', 'Comparing Excel & MatLab', 'Free Body Diagrams', 'Difference between data types (categorical/numerical, nominal/ordinal, discrete/continuous)', 'Histograms']</td>
  </tr>
  <tr>
    <td>12</td>
    <td>47</td>
    <td>['In-Class Demonstrations', 'Meeting People/Professor', 'Videos shown in Class', 'In-Class Demonstrations', 'In-Class Activity or In-Class Assignment']</td>
  </tr>
  <tr>
    <td>13</td>
    <td>19</td>
    <td>['Taum Salk reservoir power activity', 'The Tom Sauk Reservoir', 'Hydropower and Hydroelectricity', 'Hydroelectric dams, power, and reservoirs', 'Taum Sauk Project or Reservoirs']</td>
  </tr>
  <tr>
    <td>14</td>
    <td>91</td>
    <td>['Related to Flux', 'Related to Concepts (Gaussian Surfaces, Faraday Cages, E Fields)', 'Related to Loops', 'Related to Circuits & Graphs', 'Related to Linear Regression']</td>
  </tr>
</table>

Table 7: K-Means clustering results of aspects, K =15.

Two annotation strategies are explored on a first batch of five examples:

(1) Given all the reflections, the annotator needs to first cluster them into different clusters and assign the focused aspect (in noun-phrases). Afterwards, given an assigned aspect, the annotator is tasked to utilize the clusters they built and write a aspect-focused summary from scratch.
Figure 6: ReflectASP input length distribution.

Figure 7: ReflectASP summary length distribution.

Figure 8: ReflectASP summary’s compression ratio.

(2) Alternatively, the annotators are presented with an AI model-generated aspect-based summary (here, we used the generated version of GPT-4), together with the original aspect. The first step of the annotation is to identify the subset of student reflections that are related to the aspect. Then, the annotators are instructed to check and revise the system summaries to make them aspect-based (focusing on talking about the aspects). The revision also includes removing content that does not exist in the original student reflection (i.e., that is not faithful), as well as adding content that annotators feel is important.

We employed OpenAI’s ChatGPT (GPT-4) as our LLM to execute zero-shot aspect-based summarization, similar to Zhang et al. (2023). For each case in the REFLECTASP dataset, we prompted the ChatGPT model to produce a focused summary centered around the aspect. (We include the prompt in Appendix F.1). The instructions emphasized minimal requirements and explicitly requested the avoidance of unrelated text inclusions.

For the first strategy, the average time spent by the annotators on each instance is 40 mins, since assembling and drafting from scratch takes a long time. In contrast, the second strategy took on average 15 mins, as the annotators are more focused at identifying the weaknesses of the system summary and focus on producing a high-quality revised version.

D Dataset Analysis Details

D.1 Metric Details

Here we describe the linguistic metrics and would encourage the reader to read the original papers if interested in more technical details.

Content diversity (Grusky et al., 2018) is a joint measure for extractiveness of coverage and density.

Grusky et al. (2018) first proposed an algorithm to compute a set of extractive components between the input and target. Coverage measures the percentage of words in the summary that come from the source document, while density quantifies how well the word sequence of a summary can be described as a series of extractions, given that word orders can be rearranged to construct new contents.

Compression ratio measures the length ratio between the source and target summary, and Novelty-

Figure 9: ReflectASP content diversity (Grusky et al., 2018). The area of the plot indicates that, when comparing to the other two corpora, ReflectASP’s summary is less extractive (repeating the exact words) but remains faithful based on the high coverage.
n denotes the ratio of new n-grams present in the summary that are not included in the input.

D.2 Statistics

Figure 6 and Figure 7 present the distribution of token lengths (measured by word) in input documents and human written summaries. The compression ratio is plotted in Figure 8. Figure 9 compared ReflectASP to the ABS (OPOSUM+) and OABS corpus (OpenASP), showing that ReflectASP contains less direct copying of long spans from the source (lower density) while still retains good enough coverage.

E Model Implementation Details

All of our experiments are conducted on Nvidia L40S GPUs, each with 48 GB RAM. To tackle the memory limitation and speed up the inference with LLMs, we utilize the vLLM (Kwon et al., 2023) to conduct experiments. The Llama3.1-70B needs four cards for inference, and all other models use one card.

E.1 LLMs

We employ LLAMA 2-13B-chat (Touvron et al., 2023)6, and Mistral-Nemo (Jiang et al., 2023)7 models for experiments. We additionally include the Llama3-8B-Instruct and Llama3.1-8B-Instruct (https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) as well as Llama3.1-70B-Instruct with the quantized version from neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w8a8. We set the temperature at 0.3 and the max new token at 8000 for generation. We manually evaluated the aspect-based summaries generated during a brief manual tuning of the prompt text to determine the appropriate prompt. We include all model outputs in Appendix F.

For the GPT-3.5 model, we used GPT3.5-turbo 1106 from https://platform.openai.com/docs/models/gpt-3-5-turbo as one strong baseline. We additionally included the GPT-4 (gpt4-turbo-0125-preview) and GPT4o (gpt-4o-2024-08-06) models. The temperature is set as 0.5, and the max_token length is set to 256.

<table>
  <tr>
    <th>Role</th>
    <th>Content</th>
  </tr>
  <tr>
    <td>system:</td>
    <td>You are a responsive abstractive summarizer that summarizes the collection of student lecture reflections by focusing on a specific topic.</td>
  </tr>
  <tr>
    <td>user:</td>
    <td>Please write a short summary with no more than 100 words, focusing on the topic of {topic} based on below reflections:</td>
  </tr>
</table>

Table 8: GPT-4 prompt used to generate first draft of reference aspect-based summaries for human revision.

<table>
  <tr>
    <th>Role</th>
    <th>Content</th>
  </tr>
  <tr>
    <td>system:</td>
    <td>You are a TA for a undergraduate-level course, you are given a collection of student reflections after taking one lecture and tasked to write a summary to present to the instructor</td>
  </tr>
  <tr>
    <td>user:</td>
    <td>Given the students’ responses and a focused topic {aspect}, create a short summary using your own words (no more than 100 words). The summary needs to be a coherent paragraph and should include the major points of the summary. Please use the following format: the summary contains only information about reflections, and avoid adding irrelevant sentences or suggestions such as ‘make sure to bring this up in next class’, or ‘Consider this for future lectures’, etc...<br>REFLECTION [reflections]<br>FOCUSED TOPIC: [aspect]<br>SUMMARY:</td>
  </tr>
</table>

Table 9: The baseline prompt used for aspect-based summarization given an aspect.

F Approaches and Prompt Templates

F.1 GPT-4 Prompt for Human-Annotation

We use the prompt in Table 8 to generate system summaries as the initial draft for human revision.

F.2 Self-Refine Method

Inspired by the success of recent lines of study on self-correction (Madaan et al., 2023; Welleck et al., 2023), we employ a Generate-Suggest-Refine framework to use the model to improve its outputs. More specifically, after generating an initial aspect-based summary, we prompt the model to provide suggestions to improve the summary by making it more concise and concentrated on the topic. We carefully craft the prompts to ensure the suggestions are grounded in the original reflections, whilst the revision suggestions should be based on the context of the first version. Lastly, we refine the summary by providing the LLM with all reflections, the initial draft, and improvement suggestions, prompting it to produce a refined version. Our design aligns with the prompt-chaining in (Sun et al., 2024), which was effective and obtained a higher winning rate compared to the stepwise prompt. The prompt for our proposed self-refine framework can be found in Table 10. Our approach differed from prior work (Madaan et al.,
2023; Welleck et al., 2023) in that they relied on few-shot samples and had restricted the feedback formatting. Instead, our work elicited the model’s capability to provide feedback and conducted extensive analysis to evaluate the quality of suggestions and refinement.

F.3 Prompts Used for Experiments
Table 9, 10, 11, and 12 present the final prompts used in our experiments.

F.4 Model Outputs
We include examples of different baseline prompting outputs in Table 13, as well as the comparison of different refinements in Table 14.

G Evaluation Metric Details

ROUGE : We used the implementation in torchmetrics,8 using stemmer and computing the average when multiple references are available.

BERTScore : We used the implementation from huggingface’s evaluate_metrics module9 and followed the default setup.

MCext : We harnessed the SOTA Llama-3.1-Bespoke-MiniCheck-7B (BeSpoke- 1500 MC-7B) released by Bespoke Labs (Tang et al., 2024). The model is fine-tuned from “internlm/internlm2_5-7b-chat” on the combination of 35K data 1503 points following the approach in MiniCheck (Tang et al., 2024). We use the suggested code repo from https://huggingface.co/bespokelabs/Bespoke-MiniCheck-7B. Here we paired the human-annotated extractive cluster of aspect-related reflections and a single sentence from the summary as the doc and claim in the fact-checker. We record the predicted labels for that sentence (1 for being faithful and 0 for not) and report the macro distribution of labels for all sentences in the 313 generated summaries.

MCinput : This is similar to the setting in MCinput, with one exception that we use the full student reflections as the doc for fact-checking.

GPT-4 related metrics : We use the scoring prompt and rubrics from Wadhwa et al. (2024) and cite their prompts in Figure 10 and Figure 11. We use the GPT4o (gpt-4o-1117 2024-08-06) model.

8https://lightning.ai/docs/torchmetrics/stable/text/rouge_score.html
9https://huggingface.co/spaces/evaluate-metric/bertscore

Prompt D.1: Zero-Shot prompt used with GPT-4 to generate a factuality score on a scale of 1-5
Determine whether the provided summary is consistent with the corresponding document. Consistency in this context implies that all information presented in the response is summarized by the document. If not, it should be considered inconsistent.
[ [ instruction ] ]
[ [ response ] ]
The response may have one or more of the following errors:
1. Extrinsic Information: the response contains new information not grounded in the source material
2. Misleading: a property or an entity in the response can be found in the source material, but are associated with the wrong entity
3. Stating Opinion As Fact: the response entails a proposition that’s mentioned in the source material not as a fact, but as someone’s opinion
4. Reasoning Error: the response makes one or more wrong inferences based on information in the source material
5. Tense/Mood Error: the tense or modal (eg: can, may, must) used in the response sentence does not match the tense/mood of the source material
6. Contradiction: the response contradicts the source material
7. Nuanced Meaning Shift: the response twists information from the source material in a subtle way
Given the above, please assign a factuality response on a scale of 1 to 5 based on extent of factual consistency.
5. completely consistent: the response is completely factually consistent with the source material.
4. insignificant inconsistencies: the response is mostly factually consistent, with slight inconsistencies not affecting main points.
3. partially inconsistent: overall factually consistent, with a few inconsistencies with the source material.
2. secondarily inconsistent: the response is factually inconsistent, with severe deviation from main points.
1. completely inconsistent: the entire response is factually inconsistent with the source material.
First output a list of errors that the summary makes, then conclude the response with a score in the following format: "therefore, the score is:"

Figure 10: Zero-shot prompt used with GPT-4 to generate single factuality scores on a scale of 1-5. The figure is created by Wadhwa et al. (2024)
<table>
  <tr>
    <th>Stage</th>
    <th>Role</th>
    <th>Content</th>
  </tr>
  <tr>
    <td rowspan="3">GENERATION</td>
    <td>system:</td>
    <td>You are a responsive abstractive summarizer that summarizes the collection of student lecture reflections by focusing on a specific topic.</td>
  </tr>
  <tr>
    <td>user:</td>
    <td>Please write a short summary with no more than 100 words, focusing on the topic of {topic} based on below reflections:<br><br>{reflections}<br><br>SUMMARY:</td>
  </tr>
  <tr>
    <td>sys:</td>
    <td>[GENERATED TEXT]</td>
  </tr>
  <tr>
    <td>SUGGESTION</td>
    <td>user</td>
    <td>
 Can you provide a short list of 2-3 suggestions to improve the generated summary, making it more concise and focused on the topic – topic? The suggestions should be based on the original reflections and generated summaries, don’t give generic suggestions. 
</td>
  </tr>
  <tr>
    <td></td>
    <td>sys:</td>
    <td>[SUGGESTIONS]</td>
  </tr>
  <tr>
    <th colspan="3">Restart the conversation</th>
  </tr>
  <tr>
    <td rowspan="4">REFINE</td>
    <td>system:</td>
    <td>You are a responsive abstractive summarizer that summarizes the collection of student lecture reflections by focusing on a specific topic</td>
  </tr>
  <tr>
    <td>user:</td>
    <td>Please improve the short summary written below with the suggestions. The revised version should be no more than 100 words, focusing on the topic of {topic} based on below reflections:<br><br>{reflections}.<br><br>ORIGINAL SUMMARY:<br>{GENERATED TEXT}<br><br>SUGGESTIONS FOR IMPROVEMENT:<br>{SUGGESTIONS}<br><br>REFINED SUMMARY:</td>
  </tr>
</table>

Table 10: The self-refine prompt with three stages: GENERATION, SUGGESTION, and REFINE.

<table>
  <tr>
    <th>Stage</th>
    <th>Role</th>
    <th>Content</th>
  </tr>
  <tr>
    <td rowspan="2">GENERATION</td>
    <td>system:</td>
    <td>You are a TA for a undergraduate-level course, you are given a collection of student reflections after taking one lecture and tasked to write a summary to present to the instructor</td>
  </tr>
  <tr>
    <td>user:</td>
    <td>Given the students’ responses, and a focused topic {aspect}, create a short summary using your own words (no more than 100 words). The summary needs to be a coherent paragraph and should include the major points. The summary should focus on the provided topic only, contain only information about reflections, and avoid adding irrelevant sentences or suggestions such as ‘make sure to bring this up in next class’, or ‘Consider this for future lectures’, etc...<br>You are tasked to perform this task in two steps:(1) Extract the list of indexes and students’ reflections in the given REFLECTIONS that are relevant to the focused topic. (2) summarize them into a short summary using your own words (no more than 100 words).<br>REFLECTIONS: [all reflections]<br>FOCUSED TOPIC: {aspect}<br>Your response should be in this json format: { {'Extracted_Reflections' : [your extracted data] (i.e. [STUDENT_REFLECTION_1_TEXT, STUDENT_REFLECTION_2_TEXT, ...]), 'SUMMARY': [your response]}}}</td>
  </tr>
</table>

Table 11: The E2A prompt used in our task.
<table>
  <tr>
    <th>Stage</th>
    <th>Role</th>
    <th>Content</th>
  </tr>
  <tr>
    <td>Input</td>
    <td></td>
    <td>Extracted list of reflections; E2A initial summary</td>
  </tr>
  <tr>
    <td>MiniCheck Detect</td>
    <td></td>
    <td>Run the Minicheck Detector on E2A initial summary sentences, then collect those labeled as not faithful in GIVEN_SENTENCES</td>
  </tr>
  <tr>
    <td rowspan="3">SUGGESTION</td>
    <td>system:</td>
    <td>You are a TA for a undergraduate-level course, you are given a collection of student reflections after taking one lecture and tasked to write a summary to present to the instructor</td>
  </tr>
  <tr>
    <td>user:</td>
    <td>Given the students’ responses, and a focused topic {aspect}, you are provided an extracted list of reflections that are related to the topic and a short summary. (no more than 100 words).</td>
  </tr>
  <tr>
    <td colspan="2">
      ALL REFLECTION: {reflections}<br>
      FOCUSED TOPIC: {aspect}<br>
      Extracted List of original reflection and Initial Summary: {ext_summary}<br>
      Initial Summary: {summary}<br>
      Now, given the below GIVEN_SENTENCES in the summary that is identified as unfaithful, reason if there is any factually inconsistent span in the sentence and propose a way to improve the sentence, making it more concise and focused on the topic {aspect}, utilizing information from the extracted list of reflections. The suggestions should be based on the original reflections and the extracted list of reflections, don’t give generic suggestions.<br><br>
      ***GIVEN_SENTENCES: {revised_sents}***<br>
      ### Task: If GIVE_SENTENCES is []: your response should just return: Suggestions: <no revision needed>. Otherwise, for each sent in GIVEN_SENTENCES, your response should output: Suggestions: original_sent: <SENT1 from GIVEN_SENTENCES>, the error span: <span from SENT1>, the revision suggestion: <your suggested revision>(Delete if there is no need to keep or the post-edit version of SENT1)<br>
      ### EXAMPLE OUTPUTS:<br>
      if GIVEN_SENTENCES: [], you should just return "<no suggestion needed>". if GIVEN_SENTENCES={<SENT1>, <SENT2>}, your output would be Suggestions: "[{original_sent: <SENT1>, the error span: <span in SENT1>, the revision suggestion: <Modified version of SENT1>} {original_sent: <SENT2>, the error span: <span in SENT2> ...}]". Reminder: you should only provide Suggestions on sentences from the GIVEN_SENTENCES. If it is empty, just return Suggestions: <no revision needed>.
    </td>
  </tr>
</table>

| sys: | [suggestions] |

<table>
  <tr>
    <th colspan="2">Restart the conversation</th>
  </tr>
  <tr>
    <td>system:</td>
    <td>You are a TA for a undergraduate-level course, you are given a collection of student reflections after taking one lecture and tasked to write a summary to present to the instructor</td>
  </tr>
  <tr>
    <td>REFINE</td>
    <td>user: Given the students’ responses, and a focused topic {aspect}, a list of extracted responses that are relevant to the topic, and some suggestions on revisions, you are tasked to improve a short summary. (no more than 100 words).<br>Please improve the short summary written below, incorporating the suggestions. The suggestions are on sentences of the summary, so please only modify those highlighted sentences and keep the remainder unchanged. The revised version should be no more than 100 words, focusing on the topic of {aspect} based on below reflections, The summary should be a coherent paragraph and should include the major points. If you think the initial summary is good enough, you can make minimal changes. You also need to pay attention to the extracted list of reflections that are related to the given topic {aspect}:

ALL REFLECTION: {reflections}<br>
FOCUSED TOPIC: {aspect}<br>
Extracted List of original reflection and Initial Summary: {summary}<br>
Revision Suggestions: {suggestions}

You should pay attention to the revision suggestions and decide whether you want to edit the sentences with suggested revisions. If the Revision Suggestions mentions "no suggestion needed", you should not revise the initial summary. Your summary should have minimal changes on the initial summary and be factual.<br>Your response should just output the refined summary and should not include any extra explanation on changes. The format is – REFINED SUMMARY: [your response]</td>
  </tr>
</table>

Table 12: The E2A w/ MC-Refine prompt used for open-aspect-based summarization (OABS).
Prompt D.2: Zero-Shot prompt used with GPT-4 to generate pairwise factuality scores on a scale of 1-5.

Determine whether the provided summary is consistent with the corresponding document. Consistency in this context implies that the information presented in the response is substantiated by the document. If not, it should be considered inconsistent.

[[ Instruction ]]
## Response 1 [[ response1 ]]
## Response 2 [[ response2 ]]

A response can have one or more of the following errors:
1. Extrinsic Information: the response contains new information not mentioned in the source material.
2. Mis-Referencing: a property or an event in the response can be found in the source material, but are associated with the wrong entity.
3. Stating Opinion As Fact: the response entails a proposition that’s mentioned in the source material not as a fact, but as someone’s opinion.
4. Reasoning Error: a response makes one or more wrong inferences from the information in the source material.
5. Tense/Mood/Modality Error: the tense or modal (eg. can, may, must) used in the response sentence does not match the tense/mood/modality of the source material.
6. Contradiction: the response contradicts the source material.
7. Nuanced Meaning Shift: the response twists information from the source material in a subtle way

Given the error categories, rate each response on a scale of 1 to 5 based on the degree of consistency:
1. completely consistent: the response is completely factually consistent with the source material.
2. insignificant inconsistencies: the response is mostly factually consistent, with slight inconsistencies not affecting main points.
3. partially inconsistent: overall factually consistent, with a few minor inconsistencies with the source material.
4. severe inconsistencies: nearly half response is factually inconsistent, with severe deviation from main points.
5. completely inconsistent: the entire response is factually inconsistent with the source material.

For each response, first output a list of errors that the summary makes, then conclude the response with a score in the following format: “therefore, the score is.”

Output Format:

## Response 1
...
## Response 2
...

Figure 11: Zero-shot prompt used with GPT-4 to generate pairwise factuality scores on a scale of 1-5. The figure is created by Wadhwa et al. (2024).
<table>
  <tr>
    <th>Model</th>
    <th>Summary</th>
  </tr>
  <tr>
    <td>FLANT5-XXL</td>
    <td>I thought the demonstration with the lightbulb was fun to watch. It gave me a visual representation of what was happening. I found it interesting how similar the forces caused by charges and gravity are.</td>
  </tr>
  <tr>
    <td>LLAMA2</td>
    <td>The students found the light bulb demonstration and the plastic sheet experiment to be the most interesting and helpful in understanding the concepts of electric charge and forces. They appreciated the visual aids and the interactive nature of the lecture, including the clicker questions and breakout room discussions. Many students found the similarities between electric and gravitational forces to be fascinating, and the concept of electric charge properties was well received. The demonstration of energy conservation through the splitting of a gamma ray photon into an electron and positron was also found to be intriguing. Overall, the students enjoyed the interactive and visual aspects of the lecture and found them to be effective in deepening their understanding of the topics covered.</td>
  </tr>
  <tr>
    <td>LLAMA3</td>
    <td>The students found the lightbulb demonstration to be a highlight of the lecture, with many mentioning that it helped them understand the concept of electric forces and charges. The visual representation of the demonstration made it easier for students to grasp the abstract concepts, and some even found it "fun to watch". The similarity between electric and gravitational forces was also a point of interest, with students noting that it helped them understand the concept better. Additionally, the interactive nature of the lecture, including the use of breakout rooms and clicker questions, was appreciated by many students. Overall, the lightbulb demonstration and related discussions seemed to have a significant impact on students' understanding of electric charges and forces.</td>
  </tr>
  <tr>
    <td>LLAMA3.1</td>
    <td>Students found the light bulb demonstrations to be engaging and effective in illustrating complex concepts, such as the separation of charges and the distribution of electric force. Many appreciated the visual representation and hands-on experience, which helped to clarify their understanding of these abstract topics. Additionally, the demonstrations sparked interesting comparisons between electric and gravitational forces, highlighting the similarities between the two. Overall, the interactive nature of the lecture, including breakout rooms and practice problems, was well-received and contributed to a deeper understanding of the material.</td>
  </tr>
  <tr>
    <td>LLAMA3.1-70B</td>
    <td>Students found the lightbulb demonstration to be a highlight of the lecture, with many noting that it provided a clear visual representation of electrostatic concepts. The separation of charges and the resulting lighting of the bulb was particularly interesting to several students. The demonstration was also seen as engaging and fun to watch, helping to capture students' attention and make the topic more accessible. Overall, the use of visual aids and hands-on demonstrations was well-received by students, who appreciated the ability to see complex concepts in action.</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>Students found the light bulb demonstrations engaging and helpful in understanding abstract concepts like electric force and charge separation. They appreciated the visual representations, with several mentioning that it made complex topics easier to grasp. The interactive nature of these demonstrations, allowing them to see the results of changing variables, was particularly appealing. Some students, however, noted that the pace of explanations during these demonstrations could be overwhelming, suggesting a need for slower, clearer communication.</td>
  </tr>
  <tr>
    <td>Human Reference</td>
    <td>The students found the demonstrations and interactive aspects of the lecture to be the most interesting, including the plastic sheet and lightbulb experiment, the split-screen video, and the group work in solving the distribution of charge. They also appreciated the visual representations of concepts, such as the lightbulb demonstration, as it made the topics easier to understand.</td>
  </tr>
</table>

Table 13: An example of different baseline system summaries. The aspect is “Light Bulb/Demonstration”.
<table>
  <tr>
    <th>Model</th>
    <th>Summary</th>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>Students found the lightbulb demonstration to be a highlight of the lecture, with many noting that it provided a clear visual representation of electrostatic concepts. The separation of charges and the resulting lighting of the bulb was particularly interesting to several students. The demonstration was also seen as engaging and fun to watch, helping to capture students’ attention and make the topic more accessible. Overall, the use of visual aids and hands-on demonstrations was well-received by students, who appreciated the ability to see complex concepts in action.</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B w/ Self-Refine</td>
    <td>Students found the lightbulb demonstration to be a highlight of the lecture, providing a clear visual representation of electrostatic concepts, particularly the separation of charges and the resulting lighting of the bulb. Many students noted that the demonstration was engaging and fun to watch, making complex concepts more tangible. The visual representation of electrostatic concepts, such as the separation of charges, was especially interesting to several students, helping to illustrate key principles in a memorable way.</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B w/ DCR</td>
    <td>Students appreciated the use of visual aids and hands-on demonstrations in the lecture. The lightbulb demonstration was a highlight of the lecture, helping to capture students’ attention and make the topic more accessible. Overall, the use of visual aids and hands-on demonstrations was well-received by students, who appreciated the ability to see complex concepts in action.</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B E2A</td>
    <td>Students found the lightbulb demonstrations to be engaging and interesting, with the visual representation helping to clarify complex concepts. The separation of charges and the experiment setup caught their attention, making the topic more enjoyable and easier to understand. The demonstrations were considered a valuable learning tool, providing a clear and captivating way to illustrate key principles.</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B E2A w/ Self-Refine</td>
    <td>Students found the lightbulb demonstrations to be engaging and interesting, with the separation of charges and experiment setup catching their attention and making the topic more enjoyable and easier to understand. The demonstrations provided a valuable learning tool, offering a clear and captivating way to illustrate the effects of electric forces in action, making complex concepts more tangible and relatable.</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B E2A w/ MC-Refine</td>
    <td>Students found the lightbulb demonstrations to be engaging and interesting, as they provided a clear visual representation of the concept. The separation of charges and the experiment setup caught their attention, making the topic more enjoyable and easier to understand. The demonstrations were considered a valuable learning tool, providing a clear and captivating way to illustrate key principles.</td>
  </tr>
  <tr>
    <td>Human Reference</td>
    <td>The students found the demonstrations and interactive aspects of the lecture to be the most interesting, including the plastic sheet and lightbulb experiment, the split-screen video, and the group work in solving the distribution of charge. They also appreciated the visual representations of concepts, such as the lightbulb demonstration, as it made the topics easier to understand.</td>
  </tr>
</table>

Table 14: An example of different refined summaries. The aspect is “Light Bulb/Demonstration”.
H  More Automatic Results

Due to space limit, we omit one variant E2A w/ Self-Refine, which applies the Self-Refine approach on E2A results in Table 3 and two weaker LLM baselines. Here we included them with some discussions.

H.1  Additional Reference-based Evaluation Results

In Table 15, we include the results of a strong encoder-decoder-based model Flan-T5-XXL (Chung et al., 2022), one of the most capable encoder-decoder models comparable in size (11B) and fine-tuned through instructions tuning on multiple tasks. We also add extra LLM backbone models (LLAMA2 and MISTRAL) as well as the additional E2A w/ Self-Refine rows in the extended version of Table 3.

Compared to the LLM baselines in Table 3, the zero-shot prompted Flan-T5-XXL model tends to generate extractive summaries – original student reflections (see example in Table 13), leading to lower reference-based performance against human-written abstractive summaries. Older models generate worse summaries evaluated by ROUGE and BERTScore, meanwhile, the E2A approach does not help as the models fail to follow instructions in generating extracted list. Additionally, as seen in Table 15, applying Self-Refine on E2A outputs (rows 10-12) experienced performance drops on ROUGE and BERTScores when compared to the original version (rows 4, 10 and 16 in Table 3). Similar results were observed in faithfulness scores, with one exception in LLAMA3.1-70B, where the MC scores improved. This indicates that stronger models can generate more useful suggestions, leading to more effective revisions.

H.2  Additional Automatic GPT-based Evaluation

Table 16 shows the additional GPT-based automatic evaluation results. Smaller models may suffer from generating non-helpful suggestions, which leads to a drop of summarization quality (LLAMA3 E2A vs. w/self-refine in the first chunk of the bottom block).
<table>
  <tr>
    <th>ID</th>
    <th>Model</th>
    <th>R-1</th>
    <th>R-2</th>
    <th>R-L</th>
    <th>BS</th>
    <th>MC<sub>EXT</sub></th>
    <th>MC<sub>INPUT</sub></th>
    <th># Sents</th>
    <th># Words</th>
  </tr>
  <tr>
    <td>1</td>
    <td>FlanT5-XXL</td>
    <td>26.83</td>
    <td>7.62</td>
    <td>23.83</td>
    <td>86.17</td>
    <td>36.06</td>
    <td>78.22</td>
    <td>5.83</td>
    <td>112.1</td>
  </tr>
  <tr>
    <td>2</td>
    <td>LLAMA2</td>
    <td><b>45.05</b></td>
    <td><b>20.61</b></td>
    <td><b>41.05</b></td>
    <td>89.61</td>
    <td>34.12</td>
    <td>80.93</td>
    <td>5.63</td>
    <td>130.4</td>
  </tr>
  <tr>
    <td>3</td>
    <td>w/ Self-Refine</td>
    <td>36.04</td>
    <td>14.68</td>
    <td>32.75</td>
    <td>88.20</td>
    <td>26.07</td>
    <td>59.31</td>
    <td>7.42</td>
    <td>185.7</td>
  </tr>
  <tr>
    <td>4</td>
    <td>w/ DCR</td>
    <td>42.32</td>
    <td>16.46</td>
    <td>38.27</td>
    <td><b>89.76</b></td>
    <td><b>55.33</b></td>
    <td><b>86.89</b></td>
    <td>3.62</td>
    <td>59.8</td>
  </tr>
  <tr>
    <td>5</td>
    <td>E2A</td>
    <td>40.55</td>
    <td>16.44</td>
    <td>36.20</td>
    <td>88.62</td>
    <td>28.47</td>
    <td>62.82</td>
    <td>7.55</td>
    <td>185.2</td>
  </tr>
  <tr>
    <td>6</td>
    <td>MISTRAL</td>
    <td><b>43.55</b></td>
    <td>13.66</td>
    <td><b>38.40</b></td>
    <td><b>89.50</b></td>
    <td>44.33</td>
    <td><b>85.10</b></td>
    <td>3.67</td>
    <td>78.71</td>
  </tr>
  <tr>
    <td>7</td>
    <td>w/ Self-Refine</td>
    <td>40.05</td>
    <td>11.33</td>
    <td>34.98</td>
    <td>88.81</td>
    <td>39.88</td>
    <td>74.01</td>
    <td>3.43</td>
    <td>83.4</td>
  </tr>
  <tr>
    <td>8</td>
    <td>w/ DCR</td>
    <td>32.57</td>
    <td>10.17</td>
    <td>29.28</td>
    <td>89.25</td>
    <td>51.22</td>
    <td>86.75</td>
    <td>2.35</td>
    <td>34.8</td>
  </tr>
  <tr>
    <td>9</td>
    <td>E2A</td>
    <td>43.26</td>
    <td><b>14.55*</b></td>
    <td>38.06</td>
    <td>89.45</td>
    <td><b>53.25*</b></td>
    <td>71.91</td>
    <td>3.62</td>
    <td>63.2</td>
  </tr>
  <tr>
    <td>10</td>
    <td>LLAMA3 E2A w/ Self-Refine</td>
    <td>44.32</td>
    <td>15.68</td>
    <td>38.78</td>
    <td>89.63</td>
    <td>49.43*</td>
    <td>85.02*</td>
    <td>2.95</td>
    <td>79.3</td>
  </tr>
  <tr>
    <td>11</td>
    <td>LLAMA3.1 E2A w/ Self-Refine</td>
    <td>43.68</td>
    <td>15.29</td>
    <td>38.41</td>
    <td>89.53</td>
    <td>38.98</td>
    <td>79.10</td>
    <td>3.18</td>
    <td>90.1</td>
  </tr>
  <tr>
    <td>12</td>
    <td>LLAMA3.1-70B E2A w/ Self-Refine</td>
    <td>44.71</td>
    <td>16.14</td>
    <td>39.36</td>
    <td>89.69</td>
    <td>48.26*</td>
    <td>80.99</td>
    <td>3.21</td>
    <td>87.9</td>
  </tr>
</table>

Table 15: Extra experimental results on ReflectASP: BS refers to BERTScore F1. MiniCheck scores are reported. For ROUGE (R-1/L), BS, and MC scores, all results are averaged over three runs, and * means the score is significantly better than the baseline models within each block. Gray rows indicate the baseline models. The best scores for each backbone are bold.

<table>
  <tr>
    <th>Model</th>
    <th>G↑</th>
    <th>ΔG↑</th>
    <th>Pair ΔG↑</th>
    <th>W↑</th>
    <th>S</th>
    <th>L</th>
  </tr>
  <tr>
    <th colspan="7">Pairwise Comparison with Baseline Summary as the Original Input</th>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>2.74</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>w/ Self-Refine</td>
    <td>2.62</td>
    <td>-0.14</td>
    <td>-0.18</td>
    <td>0.12</td>
    <td>0.65</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td>E2A</td>
    <td>2.84†</td>
    <td>0.08*</td>
    <td>0.20*</td>
    <td>0.27</td>
    <td>0.64</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>LLAMA3</td>
    <td>2.74</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>E2A w/ Self-Refine</td>
    <td>2.80</td>
    <td>0.03*</td>
    <td>0.12*</td>
    <td>0.23</td>
    <td>0.64</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>LLAMA3.1</td>
    <td>2.76</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>w/ Self-Refine</td>
    <td>2.67</td>
    <td>-0.09</td>
    <td>0.03*</td>
    <td>0.18</td>
    <td>0.67</td>
    <td>0.15</td>
  </tr>
  <tr>
    <td>w/ DCR</td>
    <td>2.80</td>
    <td>0.01</td>
    <td><b>0.33*</b></td>
    <td><b>0.45</b></td>
    <td>0.42</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>E2A</td>
    <td>2.88†</td>
    <td>0.07*</td>
    <td>0.22*</td>
    <td>0.27</td>
    <td>0.66</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>E2A w/ MC-Refine</td>
    <td><b>2.89</b></td>
    <td><b>0.10*</b></td>
    <td>0.18*</td>
    <td>0.24</td>
    <td>0.69</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>LLAMA3.1-70B</td>
    <td>2.85</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>E2A w/ Self-Refine</td>
    <td>2.88</td>
    <td>0.02</td>
    <td>0.14*</td>
    <td>0.24</td>
    <td>0.66</td>
    <td>0.10</td>
  </tr>
  <tr>
    <th colspan="7">Pairwise Comparison with E2A Summary as the Original Input</th>
  </tr>
  <tr>
    <td>LLAMA3 E2A</td>
    <td>2.85</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>w/ Self-Refine</td>
    <td>2.81</td>
    <td>-0.03</td>
    <td>-0.22</td>
    <td>0.06</td>
    <td>0.68</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>LLAMA3.1 E2A</td>
    <td>2.87</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>w/ MC-Refine</td>
    <td>2.88</td>
    <td>0.03</td>
    <td>-0.01</td>
    <td>0.07</td>
    <td>0.84</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>LLAMA3.1-70B E2A</td>
    <td>2.87</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>w/ Self-Refine</td>
    <td>2.88</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.12</td>
    <td>0.78</td>
    <td>0.10</td>
  </tr>
</table>

Table 16: Additional GPT-related evaluation results of different methods. Within each block (this table is an extension of Table 4), pairwise metrics compare the outputs of the given system to those in the highlighted rows. A † indicates significant improvement over the previous row (p<0.05) based on a paired bootstrap test, while * denotes that the absolute value is significantly different from zero.
H.3 E2A MC-Refine Extraction of Supporting Reflections

In this section, we examine the quality of extracted supporting reflections by model.

<table>
  <tr>
    <th>Model</th>
    <th>R-1</th>
    <th>R-2</th>
    <th>R-L</th>
  </tr>
  <tr>
    <td>LLAMA3</td>
    <td>61.25</td>
    <td>51.83</td>
    <td>60.59</td>
  </tr>
  <tr>
    <td>LLAMA3.1</td>
    <td>61.16</td>
    <td>54.19</td>
    <td>60.66</td>
  </tr>
  <tr>
    <td>LLAMA3.1-70B</td>
    <td>79.78</td>
    <td>75.26</td>
    <td>79.57</td>
  </tr>
  <tr>
    <td>Human</td>
    <td>79.06</td>
    <td>73.52</td>
    <td>78.68</td>
  </tr>
</table>

Table 17: Performance of different LLMs’ extracted supporting reflections using the E2A approach compared to human extracted clusters. ROUGE (Lin, 2004) is used as a proxy to inter-annotator agreement.

We evaluate the quality of different LLMs’ extracted reflections against the human annotated ones. We estimate the human upper bound by measuring the ROUGE score between double-annotated clusters and report the ROUGE score, following Angelidis and Lapata (2018). As shown in Table 17, LLAMA3.1-70B extracted the set of aspect-related reflections similarly to human performance.

I Amazon Mechanical Turk Crowd-sourcing Evaluation Details

We source crowd-workers from Amazon Mechanical Turk, requiring them to have more than 95% approval rate and more than 5000 approved HITS. Workers were instructed to thoroughly read the annotation guidelines, which included examples and are illustrated in Figures 12 to 14. We ensured that our compensation met the minimum hourly wage requirement (we pay more than $8 per hour in Pennsylvania). We released five test samples which are known to contain faithfulness errors (each can be annotated for at most 20 times) to a pool of 100 workers who have been listed on a white list as the qualification task. We filtered unqualified workers who selected a score of 3 for the summary-level evaluation criteria or did not participate.

We have in total of 1500 annotations, spanning 500 summaries and 24 annotators, which resulted in an extreme sparse annotation matrix. Meanwhile, over 1100 of each document-level label (Relevance to Aspect and Consistency) are dominated by the value of 3, making the annotation task highly imbalanced. This makes reliability testing more challenging, with mainstream inter-annotator agreement scores, as measured by Krippendorff’s Alpha (Krippendorff, 2011), nearing 0.1. One approximation for the agreement is perfect-agreement (which means all 3 annotators picked the same score). For Relevance, 97 summaries are rated with all same scores, and the remainder 247 have a majority voting of 3. For Consistency, 101 have the perfect-agreement and 246 has majority label of 3, suggesting that reviewers can have high agreement in picking the score of 3. While prior work (Angelidis et al., 2021a; Zhang et al., 2023; Amar et al., 2023) did not report the quality of human annotations, we investigated the challenges of measuring large-scale sparse annotations and emphasized the need for better-designed human evaluation protocols for future work.

The Amazon Mechanical Turk annotation interface is in Fig 15. The full annotation cost over 200 US dollars.

J Supplementary Materials about Revision Analysis

J.1 Edit Intention Taxonomy and the Pipeline Model

We adopt the edit intention taxonomy from Jiang et al. (2022). There are seven fine-grained intention labels:

1. Improve Language – More Accurate/Specific: Minor adjustment to improve the accuracy or specificity of the description.

2. Improve Language – Improve Style: Make the text sound more professional or coherent without altering the meaning.

3. Improve Language – Simplify: Simplify complex concepts or delete redundant content to improve readability.

4. Improve Language – Other: Other language improvements that don’t fall into the above categories.

5. Correct Grammar/Typo: Fix grammatical errors, correct typos, or smooth out grammar needed by other changes.

6. Update Content: Update a large amount of scientific content, add or delete major facts.

7. Adjust Format: Adjust table, figure, equation, reference, citation, and punctuation, etc.
Introduction to the Human Evaluation Study

Your task involves assessing the quality of a summary generated by a set of student reflections. These summaries are generated by a computer program using a specific aspect-based summarization method.

The summaries are generated to condense student opinions, focusing on specific aspects. These aspects can include course topics such as “Gaussian Surface” or “Assignments.”

To perform this evaluation effectively, please adhere to the following steps:

1. Read the system summaries
2. For each summary, you will need to evaluate:
   a. Relevance to the aspect
   b. Aspect-based Factuality
   c. Overall Consistency
3. For each summary, you should pick from fully supported (3), partially supported (2) or not supported (1).
4. Sentence-level Aspect-based Factuality: If the sentence is relevant to the aspect, then the student’s opinion should be fully supported (3), partially supported (2), or not supported at all (1).

Examples

Input Reflections
Student reflections
(Highlighted reflections are manually annotated as related to the given aspect)

I had to check it over for a few minutes after class ended for it to come to me.
I did not understand why the electric field inside of a solid, uniform, insulating sphere (in) is greater than the value outside of the sphere.
I think the angle at which the flux flows from an object is a bit confusing. The first problem in the current quiz does not help.
I think the ideas and concepts make sense, but I am confused about what shape to choose as the Gaussian surface. Also, the conductor is included in reference to Gauss’ law is something new to me.
Flying out while Gaussian surface is drawn in problems.
I was not confident about the field lines of the right angle shown question for the axis we were doing.
The students found the explanation of the different space fields around the uniformly charged inner sphere, just translating from one lesson to the next we might slightly confused.
I was confused about how we find the area of the three different parts of the cylinder, and we do this.
In the last example problem video, the highlighted student reflection was not supported by the highlighted student reflection.
Did those the only three synopses we will see when?

Figure 12: A screenshot of the human annotation guideline to evaluate system generated aspect-based summaries. (1/3)

Example 1: (Relevance to the aspect)
Whether the summary discussed the given aspect exclusively, partially, or not at all.
This will not be assessed in the third question.

Aspect: Gaussian Surface

Sentence 1:
The students found the concept of Gaussian surface to be challenging, particularly the choice of the surface and its relationship to the electric field. They also struggled with the idea that the electric field is zero inside a uniformly charged insulator. The students also found the explanation of the different space fields around the uniformly charged inner sphere, just translating from one lesson to the next we might slightly confused.

Relevance to the aspect: Partially supported

Student Reflections
Student reflections
(Highlighted reflections are manually annotated as related to the given aspect)

I was not confident about the field lines of the right angle shown question for the axis we were doing.
The students found the explanation of the different space fields around the uniformly charged inner sphere, just translating from one lesson to the next we might slightly confused.
I was confused about how we find the area of the three different parts of the cylinder, and we do this.

Relevance to the aspect: Partially supported

Figure 13: A screenshot of the human annotation guideline to evaluate system generated aspect-based summaries. (2/3)

Example 2: (Aspect-based Factuality)
Look at the highlighted/input reflections. For each sentence in the summary, determine whether the content mentioned in the summaries are noted discussed in the reflections. You should pick from fully supported (3), partially supported (2), or not supported at all (1).

Digital Student Reflections
Aspect: Gaussian Surface
Highlighted reflections are manually annotated as related to the given aspect

• I think the ideas and concepts make sense, but I am confused about what shape to choose as the Gaussian surface. Also, the conductor is included in reference to Gauss’ law is something new to me.
• Flying out while Gaussian surface is drawn in problems.
• Nothing was too confusing. The only part I found a little difficult was the different space fields around the uniformly charged inner sphere. It was hard from day one to the last for me slightly confused in your last example problem video.
• Did those the only three synopses we will see when?

Aspect: Gaussian Surface
Highlighted reflections are manually annotated as related to the given aspect

1: Not supported at all
2: Partially Supported
3: Fully supported

(Rationale):
There are in total 5 sentences,
Sentence 1: Not supported at all
Sentence 2: Partially supported
Sentence 3: Fully supported
Sentence 4: Fully supported
Sentence 5: Fully supported

Figure 14: A screenshot of the human annotation guideline to evaluate system generated aspect-based summaries. (3/3)
Please read the instruction carefully before starting the tasks
You are tasked with evaluating the quality of one summary derived from a selection of student reflections, with a focus on a specific aspect. You will be asked to rate items such as “Causation/Sufficiency” or “Assignments”.
Criteria Definitions
• Reference to the Aspect
Determine if the summaries address the specified aspect exclusively, partially, or not at all.
• Overall Consistency
Determine if the facts in the summary are consistent with the facts in the original article. You should pick from fully supported, partially supported, or not supported.
• Aspect-based Factuality (Sent Level)
For each sentence in the summary, determine whether the contents mentioned in the summary are indeed discussed in the corresponding student reflection. You should pick from fully supported (1), partially supported (2), or not supported at all (3).
Instructions
Please read the linked instructions and examples carefully before you proceed with the annotation tasks.

Figure 15: A screenshot of MTurk HIT used to evaluate the quality of system generated aspect-based summaries.

We recommend the reader check the original paper for more details.

Pipelined Model We run the pipeline script from https://github.com/chaojiang06/arXivEdits/tree/main/code/pipeline to predict edit and intention results.

J.2 Suggestion Analysis
Besides directly analyzing the difference between versions of summaries, it is also crucial to inspect the suggestions provided by different refinement approaches. In Table 18, we present samples of suggestions for three systems (Self-Refine, DCR, and MC-Refine) on LLAMA3 and LLAMA3.1-70B outputs.

We find that different refinement approaches have distinct revision suggestions. For Self-Refine, the suggestions are presented in a structured list format (e.g., "1. Emphasize the interactive aspects; more illustration of the suggestion"). We examined the text preceding the colon and employed the NLTK toolkit (Bird et al., 2009) to identify the most common n-grams. Frequently observed strategies include “concise and focused,” “focused on <ASPECT>”, “use of more specific language,” and “incorporation of specific examples”, supported with model generated concrete revision demonstrations. DCR suggestions are also in the list format. Given that they are trained on a synthesized datasets, these suggestions follow the format of (1) evaluating the sentence in the context of full summary, (2) identifying the error span and (3) suggesting changes of a single sentence. Examples in the following paragraph demonstrate that DCR’s suggestions may focus on removing less confident details, which may not help improve the summarization quality. Lastly, the MC-Refine approach’s suggestion comes with a format: original_sent, error_span, and the revision suggestion. We also notice that stronger LLMs can provide reasoning on the revisions suggested.

Adhering to the findings in §5, we find that DCR’s suggestions focused on removing details from the original summary (highlighted in blue colors). For instance, in the second point of column 1 row 2, DCR’s suggestion is The error span is: ‘They appreciated the demonstration of how current flows through a circuit, including the role of grounding and how it affects the calculation.’ To fix this, consider changing the span to ‘Students found the concept of current to be fascinating’. Such removal, while improving the factuality metrics, did not help improve the quality of the aspect-based summaries. In contrast, Self-Refine employed diverse strategies to improve the specificity, removing non-related phrases, as well fusing contents to make it more concise. As for E2A w/ MC-Refine, leveraging high-quality extracted supporting reflections, and the fact-checker that help localize the
<table>
  <tr>
    <th>Method</th>
    <th>LLAMA3</th>
    <th>LLAMA3.1-70B</th>
  </tr>
  <tr>
    <td>Self-Refine</td>
    <td>
      <b>1. Focus on the most prominent points:</b> While the original summary covers a wide range of topics, it could be condensed to focus on the most prominent points. For example, the loop rule, grounding, and the role of ammeters and voltmeters could be highlighted as the main takeaways.<br>
      <b>2. Remove repetitive language:</b> Some phrases, such as "I found it interesting" or "I thought it was interesting," are repeated throughout the reflections. These could be condensed or removed to make the summary more concise.<br>
      <b>3. Emphasize the key concept:</b> The topic of current is the main focus of the reflections, but the summary could be more explicit in highlighting this concept. For example, the summary could start with a sentence like "The concept of current was a central theme in today's lecture, with students exploring its calculation, measurement, and applications."
    </td>
    <td>
      Suggestions:<br>
      1. To make the summary more concise, consider removing the phrase 'Additionally, students appreciated the simplicity of using the loop rule to solve problems and how it can make complex circuits more manageable' as it slightly deviates from the focused topic of Current and is not as directly relevant as the other points mentioned.<br>
      2. To improve focus on the topic of Current, rephrase the sentence 'Many were interested in learning about how current flows through a wire and how it can be split when grounded points are added to a circuit.' to 'Students were interested in learning about how current flows and splits in a circuit, particularly when grounded points are added' to make it more direct and concise.<br>
      3. To make the summary more concise, consider combining the ideas of the first and third sentences into one sentence, such as 'Students found the concept of current in circuits to be intriguing, particularly how it can be measured using ammeters and voltmeters, and the idea that all grounded points are connected to each other.'
    </td>
  </tr>
  <tr>
    <td>DCR</td>
    <td>
      1. The summary introduces specific details about the loop rule and its application that are not present in the given reflections. This information is extrinsic and adds details not grounded in the document. The error span is: 'the idea that the loop rule simplifies the calculation of current in a circuit' To fix this, consider changing the span to 'Students found the concept of current to be fascinating.'<br>
      2. The summary introduces specific details about the demonstration of current flow and the role of grounding that are not mentioned in the provided reflections. This information is extrinsic and adds details not grounded in the document. The error span is: 'They appreciated the demonstration of how current flows through a circuit, including the role of grounding and how it affects the calculation.' To fix this, consider changing the span to 'Students found the concept of current to be fascinating.'<br>
      3. The summary introduces specific details about the use of ammeters and voltmeters that are not mentioned in the provided reflections. This information is extrinsic and adds details not grounded in the document. The error span is: 'the use of ammeters and voltmeters.' To fix this, consider changing the span to 'the importance of measuring voltage and current.'<br>
      4. The summary introduces information about the internal resistance of batteries and its effect on current flow that is not present in the provided reflections. This information is extrinsic and adds details not grounded in the document. The error span is: 'students found it interesting to learn about the internal resistance of batteries and how it affects the flow of current' To fix this, consider changing the span to 'students appreciated the demonstration of how current flows through a circuit, including the role of grounding and how it affects the calculation.'<br>
      5. The summary introduces information about students' enjoyment and exploration of the concept of current, which is not present in the provided reflections. This information is extrinsic and adds new details not grounded in the document. The error span is: 'students enjoyed exploring the concept of current and its applications in real-life circuits.' To fix this, consider changing the span to 'Students found the concept of current to be fascinating.'
    </td>
    <td>
      1. The summary introduces specific details about the measurement of current using ammeters and voltmeters and how these instruments affect the circuit, which are not mentioned in the provided reflections. This information is extrinsic and adds details not grounded in the document. The error span is: 'Students found the concept of current in circuits to be intriguing, particularly how it can be measured using ammeters and voltmeters, and how these instruments can affect the circuit itself.' To fix this, consider changing the span to 'Students found the concept of current in circuits to be intriguing.'<br>
      2. The summary introduces specific details about how current flows through a wire and how it can be split when grounded points are added to a circuit, which are not mentioned in the provided reflections. This information is extrinsic and adds details not grounded in the document. The error span is: 'Many were interested in learning about how current flows through a wire and how it can be split when grounded points are added to a circuit.' To fix this, consider changing the span to 'Students found the concept of current in circuits to be intriguing.'<br>
      3. The summary introduces specific details about the concept of current in circuits that are not directly mentioned in the provided reflections. This information is extrinsic and adds details not grounded in the document. The error span is: 'Students found the concept of current in circuits to be intriguing, particularly how it can be measured using ammeters and voltmeters, and how these instruments can affect the circuit itself' To fix this, consider changing the span to 'Students found the concept of current in circuits to be intriguing'<br>
      4. The summary introduces information about the loop rule that is not present in the provided reflections. This information is extrinsic and adds details not grounded in the document. The error span is: 'the simplicity of using the loop rule to solve problems and how it can make complex circuits more manageable' To fix this, consider changing the span to 'the simplicity of using the loop rule to solve problems'
    </td>
  </tr>
  <tr>
    <td>E2A w/ MC-Refine*</td>
    <td>
      Suggestions; <i>original sent:</i> Students found the concept of current to be interesting and engaging, particularly in the context of circuits and the role of ammeters and voltmeters in measuring voltage and current. They appreciated the idea that current remains constant regardless of the path taken to find it, and were fascinated by the ways in which grounding points affected the flow of current. Additionally, students were intrigued by the impact of measurement instruments on the circuit itself and the importance of considering this when taking readings., <b>error span:</b> the last part of the sentence, the revision suggestion: The students were also intrigued by the practical applications of current in real-life circuits and the role of instruments in measuring it accurately.
    </td>
    <td>
      Suggestions; <i>original sent:</i> Students found the concept of current in circuits fascinating, particularly how it flows through wires, splits when grounded points are added, and can be measured using ammeters and voltmeters., <b>error span:</b> how it flows through wires, <b>the revision suggestion:</b> Students found the concept of current in circuits fascinating, particularly how it can be measured when grounded points are added, and how it can be measured using ammeters and voltmeters. Reasoning: The original sentence mentions 'how it flows through wires' which is not explicitly mentioned in the extracted list of reflections. While it is a related concept, the reflections focus more on the measurement and behavior of current in circuits. The revised sentence focuses on the aspects of current that are directly mentioned in the reflections, such as splitting when grounded points are added and measurement using ammeters and voltmeters.
    </td>
  </tr>
</table>

Table 18: Suggestions generated by different refinement approaches using two backbones LLAMA3 and LLAMA3.1-70B. The examples remained the same to Fig 2. * means that the original version for the E2A w/ MC-Refine is E2A generated summaries, while the former two are generated by the baseline.

errors, this approach made minimal edits and retain the contents that are deemed good quality.

J.3 Linguistic Analysis on Different Model Summaries

Figure 16 presents a linguistic analysis of system diversity in relation to the original reflections. Each row represents the set of approaches operated on
one backbone LLM. DCR demonstrates the most significant compression (for example, in row 1, the minimum compression ratio is 16:1 for DCR with LLAMA3, which is significantly higher than that of the other methods), which is consistent with the findings in §4.1. Notably, system-generated aspect-based summaries exhibit higher Novelty-N values compared to human-edited summaries (which are reported as 0.19/0.63/0.84 for Novelty-1/2/3, respectively, in Table 1), indicating that these systems incorporate vocabulary not present in the original reflections.

Additionally, the E2A approach expands the area of the plots, suggesting that the generated summaries have more diverse content compared to the baseline. This improvement can be attributed to the effective extraction of supporting reflections during the extraction phase.

A pairwise comparison between E2A and E2A w/ MC-Refine reveals that larger models tend to produce smaller plot areas, reflecting increased text diversity, which consequently enhances summary quality.

K Other Domain Results

In addition to our ReflectASP dataset, we additionally tested the proposed approaches on a out-of-domain dataset, SPACE (Angelidis et al., 2021a). We used their testing split, which contains reviews about 25 unique hotels. Each set contains 100 real reviews, and aspect-based summaries on each of six popular aspects: building, cleanliness, food, location, rooms, and service.

K.1 Baselines

We used the baseline system outputs released by Hosking et al. (2023), including QT asp (Angelidis et al., 2021b), HERCULESext and HERCULESabs (Hosking et al., 2023), as well as AceSum_ext and AceSum (Amplayo et al., 2021).

We include the following methods in our work: (1) Baseline, which uses the basic prompt, (2) Self-Refine, and (3) MC-Refine applied to the baseline outputs. Initial results show that LLMs struggled to process longer inputs (an average of 14k words for the combined length of all 100 reviews) and to follow instructions for generating E2A results. Thus, we exclude the E2A approach and replace the E2A with MC-Refine with the MC-Refine detector, utilizing the ground-truth clusters provided in the dataset.

K.2 Evaluation

We report the same set of evaluation metrics as in §4.1. One difference is that we follow the ROUGE implementation in Hosking et al. (2023), which is the ‘jackknifing’ method for multiple references as implemented for the GEM benchmark (Gehrmann et al., 2021), to make the evaluation consistent with scores reported in prior papers.

K.3 Result

Table 19 showed the results of different approaches. While the older baselines (rows 1–5) achieved higher ROUGE and BERTScore metrics, their factuality scores were lower than those of LLMs. We attribute this discrepancy to the lexical differences between the older supervised models and the zero-shot LLMs. As shown in Table 20, the outputs of the best baseline model, AceSum(abs), closely resemble human references, while LLM outputs tend to include more details from the original reviews.

Our proposed MC-Refine, combined with human-selected clusters for fact-checking, obtained substantial performance gains compared to the baseline and other methods. These improvements were more notable on the larger LLAMA3.1-70B model (row 16 vs. row 13). Meanwhile, the faithfulness of generated contents is also improved.

K.4 Insights from Out-domain Experiments

Here we draw multiple insights from the experimental results above and additional analysis:

1. Our dataset’s high-quality annotation resolves the inconsistency between reference-based metrics and the Faithfulness Evaluation. Older datasets can suffer from noisy annotations or heuristic-based constructions (Goyal et al., 2022; Dahan and Stanovsky, 2025). Prior work (Goyal et al., 2022) found that LLM-generated summaries outperform those from older models, while Dahan and Stanovsky (2025) highlights that some reference summaries are low-quality and heuristic-driven. One key issue is the evaluation inconsistency we spotted in the prior dataset (SPACE for review summarization), where overly abstractive patterns and low quality of the reference summaries hinder the assessment of novel approaches in the LLM era:

• In Table 19, regarding ROUGE scores, the prior best-fine-tuned model, ACESUM, outperformed all LLM baselines by a large mar-
Figure 16: Density and coverage distributions of generated summaries across different models and LLM backbones. We additionally report the compression ratio and novelty-n metrics on the top-left corner. Each box is a normalized bivariate density plot of extractive fragment coverage (x-axis) and density (y-axis), the two measures of extraction described in Section 2 and Appendix D.1.
gin, but their faithfulness evaluation result tells a different story.

• Our dataset analysis on the SPACE dataset reveals that human reference summaries often follow generic patterns (e.g., The staff was very friendly, helpful, and polite appeared multiple times for summaries focused on the aspect of staff ) that do not reflect actual user feedback. In one such data point’s annotated cluster, 11/37 reviews mention negative staff experiences (rude, busy, never answering the call…), which the ACESUM fails to capture, leading to a low agreement between reference-based and automatic faithfulness evaluations. This discrepancy makes improving LLMs more challenging.

In contrast, our dataset provides human-edited summaries grounded in original reflections and aligned with aspect-specific clusters. As shown in Table 3, our dataset exhibits more consistent evaluation patterns when scaling up LLMs and testing different approaches, establishing a robust foundation for future aspect-based summarization research.

2. Summary Analysis Findings: Our dataset contains richer information in the summaries that well summarize the input.

The prior dataset’s high compression ratio also limits the aspect-based summary’s coverage (details in Table 1). Looking at the example in Table 20, the reference summary in SPACE “This charming hotel is located within steps of all the major sights” could not give any further information and would be misleading for a tourist who is interested in some specific sights that are far away from the hotel. Our dataset can also retain the diverse students’ opinions regarding the specific aspect of the lecture and serves as a good testbed for future work in the field. For instance, one potential extension is to make the aspect-based summaries more accountable by assigning different weights to the summarization details, on which the evaluation relied more for coverage of the summary.

3. Our dataset motivated innovations in faithfulness-guided approaches. Given the manageable scales of input size, our manual evaluation identified two major challenges in aspect-based summarizations: (1) the inclusion of aspect-unrelated details and (2) the oversimplification of aspect-specific content, which in turn motivates us to develop the E2A and the Refinement approaches that can tailor them under a zero-shot setting. Prior datasets do not support this, as they either suffer from low-quality references (SPACE, OPOSUM+) or lack grounding texts for validation (OPENASP). Our extensive experiments find that LLMs can enhance their ability to generate better aspect-based summaries with these approaches. This improvement cannot be verified on other datasets due to their limitations.

L Survey of Class Size

We annotated students’ reflections from the publicly available ReflectSumm dataset, which collects data via an application used by real classrooms. Our research into undergraduate courses at several public universities based on their released common data set indicates that over 92% of undergraduate classes have fewer than 100 students, and approximately 84% have 50 or fewer students (see Table 21). Given the self-volunteering nature of data collection and variations in attendance throughout the semester, we believe our ReflectASP dataset, with an average of 43 students’ reflections per lecture, represents the majority of real-world scenarios.
<table>
  <tr>
    <th>ID</th>
    <th>Model</th>
    <th>R-1</th>
    <th>R-2</th>
    <th>R-L</th>
    <th>BS</th>
    <th>MC<sub>EXT</sub></th>
    <th>MC<sub>INPUT</sub></th>
    <th># Sents</th>
    <th># Words</th>
  </tr>
  <tr>
    <td>1</td>
    <td>QT<sub>exp</sub></td>
    <td>31.61</td>
    <td>10.24</td>
    <td>22.64</td>
    <td>87.93</td>
    <td>28.31</td>
    <td><b>47.87</b></td>
    <td>5.79</td>
    <td>46.80</td>
  </tr>
  <tr>
    <td>2</td>
    <td>AceSum<sub>best</sub></td>
    <td>35.09</td>
    <td>12.10</td>
    <td>27.15</td>
    <td>88.95</td>
    <td><b>29.10</b></td>
    <td>44.82</td>
    <td>1.99</td>
    <td>29.56</td>
  </tr>
  <tr>
    <td>3</td>
    <td>HERCULES<sub>ext</sub></td>
    <td>28.57</td>
    <td>7.91</td>
    <td>19.93</td>
    <td>87.76</td>
    <td>10.30</td>
    <td>23.92</td>
    <td>4.01</td>
    <td>56.40</td>
  </tr>
  <tr>
    <td>4</td>
    <td>HERCULES<sub>best</sub></td>
    <td>33.56</td>
    <td>10.04</td>
    <td>25.34</td>
    <td>89.59</td>
    <td>23.87</td>
    <td>39.90</td>
    <td>3.99</td>
    <td>33.85</td>
  </tr>
  <tr>
    <td>5</td>
    <td>AceSum</td>
    <td>36.38</td>
    <td>12.65</td>
    <td>29.08</td>
    <td><b>89.65</b></td>
    <td>26.59</td>
    <td>40.79</td>
    <td>2.21</td>
    <td>27.84</td>
  </tr>
  <tr>
    <td>6</td>
    <td>MISTRAL</td>
    <td>28.67</td>
    <td><b>7.54</b></td>
    <td>20.06</td>
    <td>88.19</td>
    <td>25.36</td>
    <td>59.59</td>
    <td>5.00</td>
    <td>74.70</td>
  </tr>
  <tr>
    <td>7</td>
    <td>w/ Self-Refine</td>
    <td>23.98</td>
    <td>5.59</td>
    <td>16.77</td>
    <td>87.52</td>
    <td>23.49</td>
    <td>63.66</td>
    <td>5.34</td>
    <td>124.18</td>
  </tr>
  <tr>
    <td>8</td>
    <td>w/ MC-Refine</td>
    <td><b>30.50*</b></td>
    <td>7.52</td>
    <td><b>22.00*</b></td>
    <td><b>88.51*</b></td>
    <td><b>34.63*</b></td>
    <td><b>78.79*</b></td>
    <td>2.75</td>
    <td>55.16</td>
  </tr>
  <tr>
    <td>9</td>
    <td>LLAMA3.1</td>
    <td><b>30.26</b></td>
    <td>7.05</td>
    <td>21.96</td>
    <td><b>88.64</b></td>
    <td>31.61</td>
    <td>74.03</td>
    <td>1.78</td>
    <td>47.70</td>
  </tr>
  <tr>
    <td>10</td>
    <td>w/ Self-Refine</td>
    <td>25.97</td>
    <td>5.97</td>
    <td>20.88</td>
    <td>88.10</td>
    <td>17.89</td>
    <td>52.37</td>
    <td>1.64</td>
    <td>52.66</td>
  </tr>
  <tr>
    <td>11</td>
    <td>w/ DCR</td>
    <td>23.27</td>
    <td>5.57</td>
    <td>18.31</td>
    <td>88.10</td>
    <td>43.35</td>
    <td>60.43</td>
    <td>1.15</td>
    <td>21.48</td>
  </tr>
  <tr>
    <td>12</td>
    <td>w/ MC-Refine</td>
    <td>30.23</td>
    <td><b>7.23</b></td>
    <td><b>22.48</b></td>
    <td>88.55</td>
    <td><b>55.51*</b></td>
    <td><b>76.54*</b></td>
    <td>1.94</td>
    <td>54.16</td>
  </tr>
  <tr>
    <td>13</td>
    <td>LLAMA3.1-70B</td>
    <td>31.42</td>
    <td>7.70</td>
    <td>22.93</td>
    <td>89.95</td>
    <td>29.39</td>
    <td>66.70</td>
    <td>1.69</td>
    <td>48.12</td>
  </tr>
  <tr>
    <td>14</td>
    <td>w/ Self-Refine</td>
    <td>26.41</td>
    <td>6.46</td>
    <td>21.68</td>
    <td>88.57</td>
    <td>19.63</td>
    <td>54.19</td>
    <td>1.26</td>
    <td>46.19</td>
  </tr>
  <tr>
    <td>15</td>
    <td>w/ DCR</td>
    <td>26.70</td>
    <td>5.98</td>
    <td>21.49</td>
    <td>88.78</td>
    <td>46.30</td>
    <td>66.45</td>
    <td>1.20</td>
    <td>22.78</td>
  </tr>
  <tr>
    <td>16</td>
    <td>w/ MC-Refine</td>
    <td><b>32.88*</b></td>
    <td><b>8.41*</b></td>
    <td><b>24.85*</b></td>
    <td><b>89.30*</b></td>
    <td><b>50.06*</b></td>
    <td><b>78.38*</b></td>
    <td>1.24</td>
    <td>39.71</td>
  </tr>
</table>

Table 19: Experimental results on SPACE. All LLM results are averaged over three runs, and * means the score is significantly better than the baseline models within each block. Gray rows indicate the baseline models, The best scores for each backbone and the prior baselines are bold.

<table>
  <tr>
    <th>Model</th>
    <th>Summary</th>
  </tr>
  <tr>
    <td>Human References</td>
    <td>[“This charming hotel is located within steps of all the major sights”, 'The hotel is perfectly situation a few steps away from all major sights to see in Rome.', 'It's in a great location with an available walking tour of Rome.]</td>
  </tr>
  <tr>
    <td>AceSum</td>
    <td>the location is ideally located just 2 minutes walk from piazza navona and a 15 minute walk to st. peter’s and the roman forum.</td>
  </tr>
  <tr>
    <td>LLama3.1-70B</td>
    <td>Hotel Navona’s location is excellent, within walking distance to major attractions like Piazza Navona, Pantheon, Vatican, and Spanish Steps, making it an ideal base for exploring Rome, with many restaurants and cafes nearby, and public transportation easily accessible.</td>
  </tr>
  <tr>
    <td>LLama3.1-70B w/ Self-Refine</td>
    <td>Hotel Navona is nestled in the heart of Rome’s historic center, steps away from Piazza Navona and the Pantheon, with easy access to Vatican, Spanish Steps, and nearby restaurants, shops, and public transportation, making it an ideal base for exploring Rome.</td>
  </tr>
  <tr>
    <td>LLama3.1-70B w/ DCR</td>
    <td>Hotel Navona’s location is ideal for exploring Rome, with many restaurants and cafes nearby, and public transportation easily accessible.</td>
  </tr>
  <tr>
    <td>llama3.1-70B w/ MC-Refine</td>
    <td>Hotel Navona’s location is excellent, being only 2 minutes from the Pantheon and Piazza Navona and a 15 minute walk to St. Peter’s and the Roman Forum, making it an ideal base for exploring Rome.</td>
  </tr>
</table>

Table 20: Example summaries of different approaches on SPACE.

<table>
  <tr>
    <th>School</th>
    <th>% of classrooms <100 students</th>
    <th>% of classrooms <50 students</th>
    <th>Link</th>
  </tr>
  <tr>
    <td>U Pittsburgh</td>
    <td>94%</td>
    <td>83%</td>
    <td>Page 59</td>
  </tr>
  <tr>
    <td>Penn State</td>
    <td>92%</td>
    <td>84%</td>
    <td>Page 56</td>
  </tr>
  <tr>
    <td>U Rochester</td>
    <td>97%</td>
    <td>88%</td>
    <td>Page 57</td>
  </tr>
</table>

Table 21: Statistics on undergraduate classroom size from three well-known schools in the northeast US. (REFLECTSUMM reports that the data is collected from one midwest and another northeast university.)

Reviewer paper summaries: The paper examines the challenges of evaluating factual consistency in long-document summarization, highlighting the need for standardized reference labels and comparing various evaluation systems, including both specialized fact-checkers and LLM-based approaches.

In this work, the authors aim to systematically evaluate some of the recent diverse factuality evaluation systems for long-document summarization. They conduct empirical evaluations to 1) determine the optimal granularity-level metrics for measuring the factuality of summaries, 2) the transferability of sentence-level predictions to summary-level, 3) the capabilities of different systems in detecting fine-grained error types, and 4) the performance of evaluation systems across different generations of summarization models.

The study utilizes four long-document datasets: DiverSumm, LongEval, Ramprasad’24, and UniSumEval. Evaluation systems include specialized fact-checkers like Align-Score, Infuse, and MiniCheck FT5-SENT; LLM-based systems such as GPT4o, Gemini, Llama-3.1-BeSpoke-MiniCheck-7B, and FineSurE; and linguistic-informed approaches like StructScore. Experiments show that fine-grained continuous scores provide more reliable assessments than binary classification, and that sentence-level and summary-level model performance are somewhat transferable. Additionally, the best evaluation systems show higher recall in error detection but struggle with false-positives and fine-grained error detection.

This paper presents a comprehensive benchmark study evaluating 11 factual consistency metrics across five summarization datasets spanning diverse domains and summary styles. It investigates how these metrics perform at both instance-level and system-level evaluations and further analyzes their sensitivity to six common types of factual errors.
Summary of strengths: 1.	The paper systematically evaluates multiple factual-consistency assessment methods across various long-document datasets, contributing significantly to a deeper understanding of the strengths and limitations of current summarization evaluation techniques.
2.	The authors examine correlations between evaluations conducted at sentence-level and those at summary-level, offering valuable insights into the ways different models approach factual consistency.
3.	The authors highlight the necessity for continual enhancement of evaluation benchmarks and systems, which is vital as generative models as such LLMs advance.
4.	The study highlights the superiority of fine-grained continuous scoring methods compared to binary classifications, filling a significant gap in the evaluation of long summaries

- The paper provides a thorough analysis of factuality evaluation for long-document summarization by using a wide set of evaluation systems and summarization datasets.
- The detailed analysis offers useful insights into the current state of evaluation systems and benchmarks for summarization tasks.
- Overall, this is a well-written paper.

1. This paper tackles the important issue of evaluating factual consistency in summarization, a critical bottleneck in real-world NLP applications.

2. Evaluates 11 factuality metrics across 5 diverse summarization datasets, enabling broad and meaningful comparisons.

3. Includes evaluation by error types , providing diagnostic insights often missing in prior work.
Summary of weaknesses: 1.	While the paper gives a thorough evaluations on multiple datasets and models, it is not clear how the generated text of LLM-based model differs from the non-LLMs.    
2.	The paper mentions novel LLM-based approaches that decompose summaries but does not explore or evaluate these methods, leaving a gap in the discussion about their potential effectiveness compared to traditional methods

- As the authors discuss in the Limitations section, this work focuses mainly on factual consistency. Considering other dimensions like completeness could provide more value to this work.

1. While the paper identifies well-known limitations of factuality metrics, it does so without theoretical rigor, methodological robustness, or actionable insights.

2. The paper provides a comprehensive diagnostic of existing factuality metrics, but it fails to offer any concrete improvement or new 
methodology.

3. The paper benchmarks various metrics but treats them as black boxes, without analyzing why they fail or succeed on different examples. There's no introspection into the model components, and no insight into how their architectural biases affect generalization.

4. There appear to be notable overlaps with [1], both in terms of case study on long document summarization evaluation. A clearer distinction from [1] is needed.


------

**references**

[1] Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization (NAACL 2025)
Comments, suggestions and typos: see above

See Weaknesses.

it would be helpful to include a qualitative analysis with concrete examples to illustrate what aspects the current metrics fail to evaluate effectively. This would provide clearer insight into the limitations of the metrics used.
Overall assessment scores: 2

3

2