ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedback
Authors: Taewon Yun
Jihwan Oh
Hyangsuk Min
Yuho Lee
Jihwan Bang
Jason Cai
Hwanjun Song
Venue: COLM 2025 (2025)
Keywords: Text Refinement
Text Summarization
LLMs
Reasoning
TL;DR: Reflective Reasoning for Text Refinement

Abstract: Summarization refinement faces challenges when extending to multi-dimension. In this paper, we introduce ReFeed, a powerful summarization refinement pipeline that enhances multiple dimensions through reflective reasoning on feedback. To achieve this, we release SumFeed-CoT, a large-scale Long-CoT-based dataset optimized for training a lightweight model with reflective reasoning. Our experiments reveal how the number of dimensions, feedback exposure, and reasoning policy influence refinement performance, highlighting reflective reasoning and simultaneously addressing multiple feedback is crucial to mitigate trade-off between dimensions. Furthermore, ReFeed is robust to noisy feedback and feedback order. Lastly, our finding emphasizes that creating data with a proper goal and guideline constitutes a fundamental pillar of effective reasoning. The dataset and model is available at https://github.com/DISL-Lab/ReFeed.

Published as a conference paper at COLM 2025
ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedback
Taewon Yun', Jihwan Oh', Hyangsuk Min
Yuho Lee", Jihwan Bang', Jacon Cai²; Hwanjun Song If Korea Advanced Institute of Science and Technology (KAIST) Amazon Web Services, Al Labs (ytaewon0415, jonghwanjun ac.
Abstract
Summarization refinement faces challenges when extending to multi- dimension. In this paper, we introduce ReFeed, a powerful summarization refinement pipeline that enhances multiple dimensions through reflective reasoning on feedback. To achieve this, we release SumFeed-CoT, a large- scale Long-CoT-based dataset optimized for training lightweight model with reflective reasoning. Our experiments reveal how the number of di- mensions, feedback exposure, and reasoning policy influence refinement performance, highlighting reflective reasoning and simultaneously address- ing multiple feedback is crucial to mitigate trade-off between dimensions. Furthermore, ReFeed is robust to noisy feedback and feedback order. Lastly, our finding emphasizes that creating data with a proper goal and guideline constitutes fundamental pillar of effective reasoning. The dataset and model is available at https:/ Cachhub.com/DDii. LabSL-LLbbReeeed. ReFeed.
1 Introduction
In text summarization, large language models (LLMs) have primarily aimed to enhance faithfulness by incorporating advanced optimization methodologies, such as SYNFAC- EDIT (Mishra et al., 2024) and CPO (Feng et al., 2024). However, this emphasis on a single dimension, such as faithfulness, raises an important question: "Does improving only one dimension come at the cost of other aspects of summarization? Generally, aligning model's responses with a single aspect often compromises other crucial aspects of quality (Guo et al., 2024; Song et al., 2025). In light of these conflicts, it becomes important to enhance LLM's responses from multi-dimensional perspective. Yet, effectively improving all dimensions remains challenging not only because collecting multifaceted data is costly (Song et al., 2025) but also optimizing multi-dimension is complex (Ryu et al., 2024).
One potential pathway is to refine an initial response through post-hoc refinement, with high-quality feedback, which can produce remarkable gains in various tasks (Madaan et al., 2024; Pan et al., 2024). Text refinement offers advantages in cost efficiency (Cai et al., 2024) compared to supervised fine-tuning (SFT) for summarizer, which require substantial amounts of human-annotated training data for each domain and dimension. Moreover, this technique is further enhanced by incorporating external feedback generated by LLMs (Liu al., 2023b; Wadhwa al., 2024; Wan al., 2024). Given that feedback can be easily configured as multifaceted through prompt engineering (Madaan et al., 2024), the refinement can be extended to multiple desired dimensions using external feedback.
However, extending refinement methods beyond single dimension presents several chal- lenges that must be addressed to be sufficient for multi-dimension. Firstly, multi-diieensionll refinement must account for trade-off across various dimensions. What may be a correct refinement for one dimension could, in fact, be the result sacrificing another dimension.
"This work is conducted independently and is not related to the author's position at Amazon. I *Corresponding author.
1

---

Published as a conference paper at COLM 2025
SumFeed-CoT
Refeed (Reason+Refine)
Initial Feedback Collection
Sow, Fondback
Quullty (fone, High
RessoN Rofiee
[ [97]
Fald: [SI
Cone: (0
Comp-[K]]
ReFeed
Revised Sumumarry
Document
LLS
Summary
LLMS
(Imined on SumFFed-Coot)
Reflective Reasoning Collection
Refinement with Reflective Reasoning on Feedback
J. Goal Specification
2 Roosoning with Grideline
3. Quality Control
Intial Ferdbeck-bscei
DhD Sani Fendhard Nest Swar
Falk: [S0)
Rrasening
Relletive
The mxteed cummary incluting all congiranee furtbokk longer nand to belance
Bucktrucking : resulve trade-off
Relined Summary
Simultaneous prevent order bias
Best Summarr Selection
Resenning Genension
(3) Formve Filrring
Leeet to inclule the macnig bey ortores whilz mermiining conder....
Best Summary
(Goal)
Cundeline
Rossoning
Rifiectve
(?) Vonitration
Noise filtering: : validate feedback
...irmm my collaaber, thes ventanee amsera importeni infermation about the docsreeee
Teocher)
Dimumiion Stufile
Figure 1: Overview of data construction of SumFeed-CoT, dataset for training ReFeed and ReFeed. Faith., Comp., and Conc. denote faithfulness, completeness, and conciseness.
Secondly, there exists an ordering bias. Similarly to training paradigms, nulti-dimensional refinement can be approached in two main ways: "sequential," where each prompt addresses one dimension at time, or "simultaneous," a single prompt with aggregating across all dimensions. Depending on the approaches, the refinement model may exhibit a preference for the specific positioning of dimensions within the prompt (Zhang et al., 2024) or prioritize subsequent feedback. Lastly, external feedback, derived from LLM evaluations on text, may exhibit deficiencies such as low accuracy, stemming from latent biases across various domains and models (Wang et al., 2024; Ye et al., 2024). This defective feedback can be characterized as noisy feedback. Although refinement methods are designed to adapt to exter- nal feedback, the assumption that feedback is always accurate can be problematic. Thus, incorporating noisy feedback may inadvertently affect the target dimension and beyond.
In this paper, address the three challenges by proposing ReFeed (Refinement with Reflective Reasoning on Feedback), specificlll focusing on refining summaries by empha- sizing three key dimensions: faithfulness, completeness, and conciseness Our approach distills large reasoning model (LRM) such as OpenAl's ol (Jaech et al., 2024) and integrates reflective reasoning on feedback into the refinement pipeline. LRMs like ol exhibit reflective reasoning through long ofthughtt (Long-CoT). Its thinking process involves "backtrack- ing," where the model identifies mistakes and restarts its reasoning (Wu et al., 2024; Zhong et al., 2024). This has the potential to revisit feedback incorporation when trade-offs emerge during refinement, simultaneously handling multiple dimensions. Furthermore, reflective reasoning enables a critical analysis of feedback, allowing for continuous validation of its correctness by tracing back through the reasoning process rather than adopting a receptive stance. Yet, given the high computational cost of such reasoning models, distilling this capability into smaller models presents promising direction.
As shown in Figure 1, we first construct a large-scale dataset to enable complex reasoning in LLMs. Our dataset includes high-quality LLMs' reflective reasoning on feedback covering three perspectives: (i) backtracking to resolve trade-offs during the refinement; (ii) adopting simultaneous-style refinement on multi-dimension; and (iii) validating feedback to filter out noise, We utilize reflective reasoning data from a LRM, and then curate only well-structured and successful reasoning on feedback for refinement on summaries. Lastly, we train model on the constructed dataset (SumFeed-CoT), where input feedback undergoes shuffled permutations across multiple dimensions to better mitigate order bias. We further perform comprehensive analysis to identify which refinement pipeline, including the prior two approaches, DCR (Wadhwa et al., 2024) and ACUEval (Wan et al., 2024), achieves a balanced improvement across multi-dimension mitigating trade-offs, and maintains robustness in the presence of order bias and noisy feedback.
Recent works emphasize that a human-preferred summary should be factually correct (faithful- ness), cover key information (completeness), and avoid irrelevant details (conciseness) (Song et al., 2024; Tang et al., 2024b).
2

---

Published as a conference paper at COLM 2025
Our main contributions are: (1) We propose ReFeed, multi-dimennional refinement pipeline utilizing reflective reasoning to address trade-offs, ordering bias, and noise while developing lightweight models; (2) We release SunFeed-CoT, large-scale dataset that captures Long- CoT reasoning on feedback, enabling effective refinement by considering the three key perspectives; and (3) We design and analyze five potential refinement pi pipelines, including ReFeed, examining the trade-offs of prioritizing a single dimension and the vulnerabilities to order bias and noisy feedback.
2 Related Work
Text Refinement. LLMM-based refinement is widely adopted approach for enhancing text quality, leveraging either self-feedback (Madaan et al., 2024) or external feedback from other models (Paul et al., 2024; Xu et al., 2024). In summarization tasks, previous studies have primarily targeted faithfulness (Fabbri et al., 2022; Liu et al., 2023b) by designing models and pipelines to correct factual errors. More recent approaches decompose the refinement process into distinct critique and refinement modules (Zhang et al., 2023; Sun et al., 2024). However, these methods are known to introduce high error densities during refinement (Wadhwa et al., 2024), or to overly correct when applied to LLMs' summaries. In this context, a detection step required to localize factuality errors using binary labels from automated evaluators (Wadhwa et al., 2024; Wan et al., 2024). Binary feedback has proven more effective than scalar feedback itself and, when combined with critique, offers synergistic benefits (Xu et al., 2024). Nonetheless, this advantage depends the detector's 's performance, leaving the pipeline vulnerable to biases and inaccuracies in the detector model.
Multi-Dimensional Evaluation. Faithfulness evaluation has emerged as a more reliable approach for aligning with human preferences. Although faithfulness is a mainstream criterion for evaluating LLMs, one-size-fits-al evaluation metric is impractical, as human preferences are inherently diverse. Thus, the multi-diimnssonal evaluation (Mehri & Eske- nazi, 2020; Fabbri et al., 2021) is becoming the prominent regime in the evaluation for LLMs. In summarization tasks, Song al. (2024) propose the cilli-dimensional evaluation frame- work by introducing faithfulness, completeness, and conciseness, with clear definitions and objective criteria. Specifically, faithfulness and conciseness are evaluated at the sentence level, where faithfulness is measured by factual errors and conciseness by the inclusion of key facts in summary sentences Completeness is assessed at the key fact level, marking any missing key facts. Lee et al. (2024) benchmark LLMs on these dimensions, finding that while faithfulness remains strong, the other two dimensions often underperform in certain domains. Since previous works validate the importance of multi-dimensional evaluation, we explore extending text refinement to incorporate multiple dimensions.
3 Preliminary: Refinement Pipeline
3.1 Refinement Modules
In general, the text refinement pipeline involves three key modules, namely detection, reasoning (critique), and refinement modules (Wadhwa et al., 2024).
Detection Module. This employs automated evaluators like Minicheck (Tang et al., 2024a) to pinpoint parts requiring revision, such as sentences or atomic facts, using binary labels. Notably, binary labels alone can drive effective refinements (Wan et al., 2024; Wadhwa al., 2024), making the detection label itself a form of feedback. The labels (feedback) are passed to the critique module.
Reasoning (Critique) Module. This generates a specific reason why correction is needed and suggests how to fix. is often used sequentially after the detection module (Wadhwa et al., 2024) or concurrently with it (Song et al., 2024). Feedback can thus be generated by either single module (detection reasoning) or by critique based detection labels.
3

---

Published as a conference paper at COLM 2025
While such step is referred to "critique," in prior works, we instead refer to it "reasoning" to better represent their role.
Refinement Module. This refines text either without feedback or with provided feedback. Refinement is generally more effective when feedback is available rather than when per- formed directly without it (Madaan et al., 2024). In such cases, the feedback quality plays crucial role in determining the effectiveness of the refinement outcome.
3.2 Extension to Multi-Dimension
Unlike previous refinement pipelines focusing on single dimension, multi-diimnnsionll feedback introduces new complexity by requiring the integration of Here, we introduce three key aspects, A1-A3, essential for designing a multi-dieensional refinement pipeline.
diverse feedback at once.
A1: Trade-off in Multi-Feedback. Reasoning and refinement modules should reconcile the feedback derived from multiple detection modules to address multiple dimensions, thereby making each feedback component interdependent with the others. Dependency here can act as constraints similar to multi-objecctvve optimization (Guo et al., 2024). For instance, when rddressnng faithfulness and completeness at once, it is preferable to replace only erroneous phrases to preserve completeness, while deleting the entire sentence to fix minor faithfulness issue. The latter is considered "over-correction" (Shridhar et al., 2024), leading to the loss of key information, which harm completeness. Therefore, considering trade-offs from interdependent nulti-feeddbak is crucial reasoning.
A2: Sequential Simultaneous. The refinement module can incorporate multi- dimensional feedback via two strategies inspired by training methodologies; "sequential" and "simultaneous. The sequential approach (Lou et al., 2024) provides feedback stepwise for each individual dimension, allowing concentrated attention on a target dimension that may yield higher gains. In contrast, the simultaneous approach (Wu et al., 2023; Guo et al., 2024) aggregates feedback across all dimensions into a single prompt. This ensures a more balanced consideration of all dimensions, though it lacks the concentrated attention the sequential one.
A3: Inaccurate Feedback. While reasoning and refinement modules are designed for "receptive" reasoning to feedback, it is important to note that LLM-generated feedback may be biased or inaccurate. Hence, LLMs' feedback may amplify flawed reasoning (Turpin et al., 2024). In this context, LRM like OpenAl's ol has gained attention for its ability to perform "reflective" reasoning through Long-CoT. By iteratively reviewing feedback and retracing the reasoning, reflective reasoning can identify inaccurate feedback and mitigate the continuation of erroneous reasoning.
3.3 Simple Pipeline for Multi-Dimennion
We design four simple pipelines addressing three key aspects, all which serve as baselines. We integrate reasoning and refinement into single module, referred to "reason+refine" to foster closer alignment between the reasoning behind suggestions to fix and the refined text. In this pipeline, document and its summary are first processed by detection module. A reason+ refine module then generates both a reasoning on feedback and a revised summary.
We introduce four baselines (P1-P4), each starting with single-dimension refinement and gradually integrating the three aspects from Al to A3.
P1: This is the base pipeline designed specifically for single-dimension feedback, following "detect- reffine" structure.
P2: This extends P1 to support multi-dimennional feedback sequentially. The reason+refine model iterates three times, each using the improved summary from the previous run and integrating feedback for faithfulness, completeness, and conciseness in that order.
4

---

Published as a conference paper at COLM 2025
P3: This is identical to P2, except for the prompt strategy. While P2 employs single- turn prompt that resets the session at each step, P3 uses a multi-turn approach, retaining memory of previous runs for improved continuity. Compared to P2, the subsequent step in this pipeline can perceive the refinement results of the previous dimension, making the refinement results more dependent.
P4: Unlike P2 and P3, this shifts sequential prompting to simultaneous approach, providing feedback for all three dimensions.
The detailed categorization of the four baselines presented in columns 1-5 of Table 1, and the details of the prompts are given in Appendix A.
4 ReFeed: Refinement with Reflective Reasoning on Feedback
ReFeed differs from the four baselines, resembling P4 (multi-dimension, simultaneous feedback) but uniquely integrating reflective reasoning in refinement. To distill this capability, it includes data construction and a dedicated training process. Note that, in refinement, our focus is on enhancing the quality of the initial summary across three key dimensions (Song et al., 2024; Tang et al., 2024b):
Faithfulness: The factual consistency of the summary with its grounded document.
Completeness: The coverage of key information in the summary.
Conciseness: The succinctness of the summary, avoiding unnecessary details.
We present the details of them in Appendix B.
4.1 SumFeed-CoT Dataset
As existing LRMs are mainly trained on structured problems, such as solving math problems and code generation, they exhibit some misalignment with the need for summarization refinement. Thus, we guide the teacher model to refine summary based on reflective rea- soning on feedback, thereby constructing training dataset. Motivated by previous works (Jaech et al., 2024; Xu et al., 2025a), we construct our dataset through three stages in Figure Goal Specificatton, Guideline Formulation, and Quality Control.
A well-defined goal, such golden label, provides a clear reference for the LRM, steering toward an optimal reasoning path. Detailed guidelines further reinforce this process by ensuring alignment between the reasoning trajectory and the intended objective (Jaech et al., 2024). Finally, rigorous quality control essential for curating high-quality reasoning data (Lu et al., 2024), which, in turn, enhances student models' ability to engage in reflective reasoning for refinement.
We outline the three stages for data construction in the following sections.
4.1.1 Initial Feedback Collection
We generate initial feedback for reasoning using document-summary pairs. Following Song et al. (2025), we sample 200 documents from the train split of seven datasets across diverse domains and generate summaries using 13 different models to ensure diversity of summary conditions. Each summary evaluated with binary labels that identify sentences to fix for faithfulness or conciseness, or missing key facts that should be added for completeness through LLM-based evaluator To address various quality feedback, we adapt the backbone of the LLM-based evaluator using both LLaMA-3.1 models. We then map the binary labels from the automatic evaluator to corresponding sentences and key facts, each of which serves as feedback to the refinement model.
(low quality) and -70B (high quality)
After filtering feedback's 's format errors, obtain 29K <document, summary, feedback triplets, each serving as input to the reason+refine model for summary refinement. Refer to Appendix C for details source data, summarizers, and prompts.
We used the most recent automated evaluator tailored for summary, FineSurF (Song al., 2024).

---

Published as a conference paper at COLM 2025
4.1.2 Reflective Reasoning Collection
Summary refinement currently lacks clear goal and guidelines. To address this, we define the ideal summary as one that needs no further feedback and concisely captures all key facts. We also establish guidelines to address three key aspects of multi-dimensional refinement: (1) backtracking for trade-offs, (2) simultaneous style to avoid order bias and (3) noise filtering. Lastly, we curate high-quality reasoning via two criteria: format filtering, which retains Long-CoT without formatting errors, and verificalion- filtering, which ensures that the reasoning meets established quality standards. The reasoning collection process in our dataset is detailed as follows:
Goal Specification. We define the best summary as the one selected from those generated by the 13 summarizers during the feedback collection step. For each document, the best one the summary with high quality that only needs minimal feedback. Based on the initial feedback, select the summary with the highest average across faithfulness, completeness, and conciseness as the best one.
Reasoning with Guideline. We define guideline to distill reflective reasoning from tearcher LRM for summary refinement. The reasoning is collected by providing <document, summary, feedback, best summary> as input to the teacher LRM, along with our well- formed guideline, to induce reflective reasoning. The reasoning derives how to refine the given summary by referencing the best summary (goal) as a standard.
Specifically, the guideline prompts the teacher LRM to reason across multiple dimensions si- multaneously, incorporating reflective reasoning such as backtracking to manage trade-offs and validating feedback. Additionally, it encourages the model to analyze the best sum- mary's attributes, allowing it to better filter out noisy feedback in the input on completeness and conciseness. Yet, this does not enhance the faithfulness critique. Thus, introduce explicit factual error definitions, aiding more accurate faithfulness feedback assessment (Wadhwa et al., 2024). See Appendix D for details.
Quality Control. We perform format filtering to retain only samples with over 5K tokens in reasoning after removing malformed outputs, ensuring format compliance and compati- bility with the student model's token limit. Next, we conduct a verification-base filtering, verifying the revised summary with the collected reasoning indeed meets the following criteria: (1) faithfulness = 1, completeness 2 0.5, and conciseness 2 0.5; and (2) A score is >> 0 across all dimensions, meaning that the revised summary must improve upon the original one in every dimension.
Consequently, our final dataset includes 7.7K training samples, each with <document, summary, feedback> for the input prompt and <reflective reasoning, refined summary> for the output response. We then randomly shuffle the order of dimensions in the feedback to mitigate order bias. The details of training data can be found in Appendix E.
4.2 Training and Inference Details
We fine-tune
(student) using LoRA (Hu et al., 2022) and DeepSpeed
(ZeRO-3) (Rasley et al., 2020). The user prompt follows the same format as reasoning generation, excluding the best summary (i.e., goal) and error types. At inference time, the model generates both reasoning and refined summary enclosed within separate XML tags, as shown below. For detailed input/output formats, training parameters, and inference prompt, refer to Appendix F.
Summary of Input/ Output in ReFeed
Input: Document, Summary, Feedback
Output <think> Reflective Reasoning </think> <answer> Refined Summary </answer>
We used Owreee model, which exhibits Long-CoT capabilities comparable to ol-mini.

---

Published as a conference paper at COLM 2025
Pipeline
Dim.
Dep.
Simul.
Reflect.
Faith.
Comp.
Conc.
Avg.
Before Refine
78.0
46.4
76.4
66.9
P1-Faith
Faith.
X No
X No
X No
80.7"(42.)
45.9 (-0.5)
80.4 (+4.0) (0.4"(+4.0)
69.0" (+2.1)
P1-Comp
Comp.
No
No
X
80.4 (+2.4)
62.0" (+15.6)
79.0" (+2.6)
73.8 (+6.9)
P1-Cons
Conc.
X No
X No
X No
76.7 (-1.3)
44.2 (-2.2)
86.3" (+9.9)
69.1 (+2.2)
P2
Multi
X No
X No
X No
78.4 (+0.4)
51.5 (+5.1)
84.8 (+8.4)
71.6' (44.7)
P3
Multi
/ Yes
X No
X No
78.9 (+0.9)
53.2 (+6.8)
80.0" (43.6)
70.7
P4
Multi
/ Yes
/ Yes
x No
80.1 (021)(42.1)
56.0" (+9.6)
83.6" (47.2)
73.2 (46.3)
ReFeed
Multi
/ Yes
/ Yes
/ Yes
82.7" 82.7" (+4.7)
60.0" (+13.6
83.4 (+7.0)
75.3 (48.0)
Table 1: Summary quality of revised summaries on three dimensions across seven refinement pipelines. "Dep.", "Simul.", and "Reflect." denote feedback interdependency, simultaneous approach, and reflective reasoning, respectively. In the sequential approach (P2, P3), feed- back is incorporated in the order of faithfulness compleeeessssssmmpeeeeeeeeee +conciseness. Senompeeteessscccccceeesssss The best scores are marked in bold. denotes a statistically significant difference (p 0.05) based on a paired bootstrap test.
5 Evaluation
5.1 Experiment Setup
Test Dataset. We compare ReFeed with four baselines (P1-P4) on UniSumEval (Lee al., 2024), which features challenging source documents in nine domains, with summaries from nine summarizers (both non-LLMs and LLMs) prone to factual errors, missing information, and unnecessary content.
Additionally, since refinement tasks require feedback labels aligned with the input sum- mary, we collect initial feedback labels for UniSumEval's summaries using FineSurE (Song et al., 2024) (as a "detector" module). This generates feedback, including sentence-level labels for faithfulness and conciseness and key fact-level labels for completeness for each <document, summary> pair. Note that two types of feedback are used to adjust the qual- ity in FineSurE by employing different backbones: quality feedback, respectively. ReFeed and all baselines receive the same input for re- finement: <document, summary, feedback>. The goal is to refine the initial summary using LLM-generated feedback and produce revised one.
1.MMA--.1-8B/700 for low- and high-
Evaluation Metric. We assess the quality of the refined summaries using FineSurE again, this time with GPT-4o as the backbone. GIT--oo is chosen for its high accuracy in evaluation, ensuring reliable assessments, unlike input feedback, which may contain noise. We assess the revised summary on faithfulness, completeness, and conciseness. We also provide results with humans and another automated evaluator, G-Eval, in Appendices H and G, along with the metric details.
Compared Methods. Since existing methods focus on single-aspect refinement, we pri- marily compare the multi-diimensional pipelines, P1-P4, in Section 3.3. For complete evaluation, we also evaluate two recent methods, DCR (Wadhwa et al., 2024) and ACUÉval (Wan et al., 2024) in Section 5.6, though both designed only for single dimension, i.e., faithfulness.
Main Questions. Our experiment is mainly designed to address the following three research questions.
RQ1. We compare refinement pipelines based on their capacity for dimensions (single vs. multi), feedback exposure (sequential vs. simultaneous), and reasoning policy (receptive vs. reflective).
7

---

Published as a conference paper at COLM 2025
Random
Last-Faith
Comp
Last-Cone
Max-Min
Pipeline
FA
CM
CN
FA
CM
CN
FA
CM
CN
FA
CM
CN
FA
CM
CN
P2
80.8
59.1
85.3
80.9
59.0
85.0
81.1
63.2
83.9
79.6
55.3
85.1
1.5
7.9
1.4
80.7
59.3
82.8
80.8
57.7
82.5
80.4
62.5
82.4
79.5
58.4
83.0
1.3
4.8
0.6
P4
83.8
61.3
85.1
82.9
60.6
84.9
83.6
60.7
83.8
81.9
59.8
84.8
1.9
1.5
1.3
ReFeed
84.2
62.9
84.2
83.6
84.6
83.4
62.4
84.5
84.3
62.9
84.4
0.9
0.5
0.4
ReFeed(-)
84.2
61.1
85.2
83.3
60.1
84.2
82.7
60.1
84.5
83.1
61.6
84.5
1.5
1.5
0.7
Table 2: Summary quality after refinement with multi-dimennsionll refinement pipelines in four shuffling setups. FA, CM, and CN denote faithfulness, completeness, and conciseness. "Max-Min" is the score gap between the highest and lowest values for each dimension, with smaller ones indicating better robustness (the smallest value in bold).
RQ2. We examine how the order of feedback dimensions affects the refinement quality, evaluating how robustly the refinement process incorporates multi-dimensional feedback, regardless of order.
RO3. We examine how noisy feedback affects the quality of refined summaries. This high- lights how detection errors cascade into refinement outcomes, emphasizing the impor- tance of reflective reasoning over receptive one.
The following subsections provide an in-depth analysis of the three main research questions.
5.2 RQ1: Refinement Pipeline Comparison
Table presents the revised summary q to P4 and ReFeed (Ours), along with the quality of the original summaries in UniSumEval before refinement. All refinement pipelines use the LLaMA-3.1-80 backbone and take low- quality feedback, along with the document and sunmary, as input.
across five refinement pipelines, namely P1
Highlight. ReFeed outperforms other pipelines in the average score ("Avg. "), composite measure across three dimensions. This indicates that ReFeed effectively balances trade-offs, accounts for feedback dependencies, and remains robust to detector inaccuracies. Also, this trend is consistent human evaluation; refer to Appendix H.
5.2.1 Detailed Analysis
Feedback Dependency: While P1-Faith and P1-Conc achieve significant improvements in faithfulness and conciseness, respectively, they reduce completeness or faithfulness. This confirms that focusing solely on one dimension compromises others due to the dependen- cies on multi-feedback.
Single- vs. Multi-turn: In the sequential refinement scheme, the single-turn method (P2) exhibits large disparity in quality improvements across dimensions, it resets previous sessions during refinement. Specifically, P22 maximizes conciseness (the last dimension addressed) over the other two dimensions. In contrast, the multi-turn approach (P3) achieves more balanced improvements, though the gains remain modest, i.e., a similar composite score for P2 and P3.
Sequential vs. Simultaneous: Employing multi-dimension feedback sequentially simul- taneously (P3-P4) leads to reasonable improvements across all three dimensions. But, the simultaneous approach (P4) outperforms sequential strategies (P3) on their composite scores. Therefore, refining in a single prompt is more effective than splitting the process across multiple turns, it prevents information loss with each turn change.
Receptive vs. Reflective: ReFeed (reflective reasoning) shows a notable improvement over P4 (receptive reasoning), increasing faithfulness by 2.6 and completeness by 4.0. This large gain confirms that reflective reasoning is essential for summary refinement to mitigate multi-faceted risks and manage feedback interdependencies.
8

---

Published as a conference paper at COLM 2025
5.3 RQ2: Robustness against Ordering Bias
Another key aspect of a robust refinement pipeline is resilience to feedback order variations, ensuring consistent refinement quality. Thus, unlike the previous setup in Section 5.2, we design four shuffling variants, each following a different policy: Random, where feedback order is fully shuffled across three dimensions at the sample level, ensuring each input has a random order; and Last-Dim, where one dimension is fixed in the final position while the first two are randomly shuffled, resulting in three setups: Last-(Faith,,Compp Cons).
Table 2 shows the summary quality's 's variation over the four setups. In this experiment, use high-quality feedback input for summary refinement to minimize bias from noisy feedback. Note that ReFeed(-) is a variant of ReFeed fine-tuned on non-shuffled SuoFeed-CoT data to assess the impact of feedback dimension shuffling in training.
ReFeed demonstrates greater stability due to its refinement strategy and training method. In sequential pipelines (P2 and P3), completeness scores fluctuate significantly, reflected in the high Max-Min values of 7.9 and 4.8, respectively. This variability likely stems from order bias, i.e., later dimensions overly influence refinement performance, as evidenced by P2's superior scores on its final dimension. Additionally, the independent reflection strategy in P2 may amplify this fluctuation.
Lastly, shuffling the dimension order in the training data mitigates ordering bias, leading to improved overall performance and reduced variance, as shown by the performance gap between ReFeed and ReFeed(-).
5.4
RQ3: Robustness against Feedback Quality
An ideal refinement pipeline should im- prove the summary to high quality through reasoning, regardless of the feedback qual- ity. To validate this, we vary the feedback be- tween low and high quality, enabling us to evaluate performance consistency, as sum- marized in Table 3.
Faith.
Comp.
Conc.
Pipeline
High| Low
High
Low
High Low
Before.
78.0
46.4
76.4
P1-Fas Fl-Faith
82.2'
80.7"
47.1"
45,9
79.7"
80.4"
P1- P1-Comp
82.7'
80.4'
66.2
62.0"
80.9'
79.0"
ReFeed achieves the most consistent scores, regardless of the feedback quality. Specif- ically, ReFeed shows score drop of 0.06- 0.25 across all three dimensions, while even the second-best method (P4) shows a drop of 0.12-4.0. In particular, ReFeed, even with low-quality feedback, outperforms P1- Faith, which focuses solely on faithfulness, despite P1 receiving high-quality feedback. Even worse, the P1 series struggles even in their targeted dimensions, making them vulnerable feedback quality. That is, the receptive reasoning in P1 and P4 does not ensure robustness to feedback quality. Therefore, ReFeed's reflective reasoning achieves high robustness feedback quality and produces high-quality summaries.
P1-Cone
78.9
76.7
45.5"
44.2"
86.9*
86.3"
P4
83.3*
80.1"
60.0"
56.0"
84.8'
83.6"
ReFeed
84.6"
82.7"
62.5"
60.0"
84.0"
83.4"
Table 3: Summary quality of revised sum- maries under low- and high-quality feedback. "Before." indicates scores before refinement. denotes a statistically significant difference (p < 0.05) based on paired bootstrap test.
Further Study on Reflective Reasoning
We conduct an in-depth investigation to demonstrate the effectiveness of reflective reasoning over receptive one. To do so, we compare two student models using the LLaMA-3.1-4 backbone distilled from the same teacher (QwQ-32B-previee reasoning strategies, i.e., P4FT (receptive), which a variant of P4 fine-tuned, and ReFeed (reflective). To fine-tune P4, create a dataset following the same process SumFeed-Coi with P4's prompt for receptive reasoning. Refer to Appendix for details training data and prompt for P4.
but trained with different

---

Published as a conference paper at COLM 2025
Composite
Pipeline
Faith.
Comp.
Conc.
Model
Pipeline
High
Low
Before
78.0
46.4
76.4
P4
57.0"
54.9"
ACUEval
80.8'
43.8'
81.9*
(+2.8)
(-2.6)
(+5.5)
P44k
55.3
54.3*
LLMMA-311 8B
ReFeed4k
57.1
55.7'
DCR
81.2
42.8"
76.7
ReFeedsk
57.8"
56.5"
(+3.2)
(-3.6)
(+0.3)
ReFeed
82.7"
60.0"
83.4"
QwQ 32B (Teacher)
58.1"
57.3"
(+4.7)
(+13.6)
(+7.0)
Table 4: Comparison of reflective reasoning over receptive one on low- and high-quality feedback setups. P44k denotes P4 fine-tuned on 4K data; 4K/8K: training dataset size.
Table 5: Comparison with the previous pipelines, which focused only on faithful- ness. indicates a statistically significant dif- ference (p < 0.05, paired bootstrap test).
Table presents the improvement of P4 after fine-tuning compared ReFeed. Since P4's final dataset size is 5K, we train two models, P4FT and ReFeed, using only the 4K overlapping data, where the latter denoted as ReFeed4k-
It is of interest to see that refinement performance worsens after fine-tuning P4, likely due to the absence of explicit goals and critique guidelines, such as best summaries and factual error types in ReFeedd' training process. Moreover, we observe that ReFeed improves as the increase of the data size from 4K to 8K. These results highlight the importance of both carefully crafted reasoning guidelines and large data volumes for boosting reasoning capabilities, consistent with prior works (Jaech et al., 2024; Xu et al., 2025b).
From distillation perspective, ReFeed with those of the teacher, which is prompted with ReFeed's prompt without training. ReFeed achieves nearly equivalent refinement performance to the teacher, while achieving inference speeds approximately 4x faster (see latency comparison in Appendix 1.3).
compare the refinement performance and latency of
Furthermore, in Appendix 1.5, we compares refinement model trained on the revised sum- maries in SumFeed-CoT, which directly refines summaries without using the detection and reasoning modules. While this direct refinement model optimizes heavily for one dimension and shows only minor gains in others, ReFeed achieves more balanced improvements across all aspects. This underscores that reflective reasoning on multiple feedback contribute to balanced refinement quality beyond what training on revised summaries can achieve.
5.6 Comparison with Existing Pipeline
For a complete evaluation, we compare ReFeed with two recent summary refinement methods, DCR (Wadhwa et al., 2024) and ACUEval (Wan et al., 2024), despite their original focus solely on enhancing faithfulness. To ensure a reasonably fair comparison, we use the same LLaMA-3.1-8B backbone for refinement and provide input feedback generated by the same model used in ReFeed.
ReFeed demonstrates improvements across all dimensions without sacrificing perfor- mance in any dimensions via reflective reasoning. On the other hand, DCR and ACUEval incur an alignment tax in completeness, despite improving faithfulness. This faithful- ness-completeness trade-off aligns with the findings in Song et al. (2025). These results highlight the necessity of balancing trade-offs for multi-dimensional feedback refinement.
6 Conclusion
We propose ReFeed for enhancing summary refinement across multiple dimensions. Our experiments show that incorporating reflective reasoning into the refinement process out- performs four alternatives and existing approaches, by achieving a better balance across dimensions while keeping robustness against order bias and noise in feedback. Also, we observe that ReFeed delivers refinement performance comparable to its teacher model while significantly reducing inference time.
10

---

Published as a conference paper at COLM 2025
Ethics Statement
Our work primarily focuses on reasoning capabilities based on diverse LLM-generated feedback, which does not raise any ethical concerns during the model training phase.
Acknowledgements
This research was supported by KISTI in 2025 (No.(KISTI) K25LIMIC1), aimed at develop- ing KONI (KISTI Open Neural Intelligence, a large language model specialized in science and technology), and by the NRF funded by MSIT (No. RS-2022-NR068758) For GPU infras- tructure, our work was supported by the ITTP grant funded by MSIT (No. RS-2025-02653113, High-Performance Research AT Computing Infrastructure Support at the PFLOPS Scale).
References
Jason Cai, Hang Su, Monica Sunkara, Igor Shalyminov, and Saab Mansour. Ceret: Cost-
effective extrinsic refinement for text generation. In NAACL, 2024.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. DialogSum: A real-life scenario dialogue summarization dataset. In ACL, 2021.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summa- rization of long documents. NAACL, 2018.
Alexander Fabbri, Wojciech Kryšciñiski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409 2021.
Alexander R Fabbri, Prafulla Kumar Choubey, Jesse Vig, Chien-Sheng Wu, and Caiming Xiong. Improving factual consistency in summarization with compression-based post- editing. In EMNLP, 2022.
Huawen Feng, Yan Fan, Xiong Liu, Ting-En Lin, Zekun Yao, Yuchuan Wu, Fei Huang, Yongbin Li, and Qianli Ma. Improving factual consistency of news summarization by contrastive preference optimization. In EMNLP, 2024.
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, et al. Controllable preference optimization: Toward controllable multi-objective alignment. arxiv preprint ar drXio:2402.19085 2024.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
Yebowen Hu, Timothy Ganter, Hanieh Deilamsalehy, Franck Dernoncourt, Hassan Foroosh, and Fei Liu. Meetingbank: A benchmark dataset for meeting sunmarization. In ACL, 2023.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In ACL, 2021.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arviv preprint ar xrXio=2412.16770, 2024.
Mahnaz Koupaee and William Yang Wang. Wikihow: large scale text summarization dataset. arXiv preprint 1xXi:18800.099 2018.
Yuho Lee, Taewon Yun, Hang Su, Jason Cai, and Hwanjun Song. UniSumEval: Towards unified, fine-grained, multi diittsidmmssinall summarization evaluation for llms. EMNLP, 2024.
11

---

Published as a conference paper at COLM 2025
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 2023a.
Conference on Empirical Methods in Natural Language Processing, pp.
Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Halfaker, Dragomir Radev, and Ahmed Hassan. On improving summarization factual consistency from natural language feed- back. In ACL, 2023b.
Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, and Kaiqi Huang. Spo: Multi-dimensional preference sequential alignment with implicit reward modeling. arXiv preprint Xiv:2405.122335 2024.
Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. In ACL, 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. In NeurIPS, 2024.
Shikib Mehri and Maxine Eskenazi. Usr: An unsupervised and reference free evaluation metric for dialog generation. In ACL, 2020.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.094133 2024.
Prakamya Mishra, Zonghai Yao, Parth Vashisht, Feiyun Ouyang, Beining Wang, Vidhi Dhaval Mody, and Hong Yu. SYNFAC-EDIT: Synthetic imitation edit feedback for factual alignment in clinical summarization. In EMNLP, 2024.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text sum- marization using srquence-to-sqqueer rnns and beyond. ar preprint ar xxx::1602.00003,
2016.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics, 12:484-000 2024.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations. In EACL, 2024.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In KDD, 2020.
Sangwon Ryu, Heejin Do, Yunsu Kim, Gary Geunbae Lee, and Jungseul Ok. Multi- dimensional optimization for text summarization via reinforcement learning. In ACL, 2024.
Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ramakanth Pa- sunuru, Mrinmaya Sachan, Jason Weston, and Asli Celikyilmaz. The art of 1lm refinement: Ask, refine, and trust. In N024,,2024.
Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. FineSurE: Fine- grained summarization evaluation using 1lms. In ACI,, 2024.
Hwanjun Song, Taewon Yun, Yuho Lee, Jihwan Oh, Gihun Lee, Jason Cai, and Hang Su. Learning to summarize from -lm-generated feedback. In NAACI, 2025.
12

---

Published as a conference paper at COLM 2025
Shichao Sun, Ruifeng Yuan, Ziqiang Cao, Wenjie Li, and Pengfei Liu. Prompt chaining or stepwise prompt? refinement in text summarization. In ACL,, 2024.
Liyan Tang, Philippe Laban, and Greg Durrett. MiniCheck: Efficient fact-checking of Ilms on grounding documents. Xiv preprint ar arXiv:2404.10774 2024a.
Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, et al. TofuEval: Evaluating hallucinations of Ilms on topic-focused dialogue summarization. In NAACL, 2024b.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don't always say what they think: unfaithful explanations in thain-of-thought prompting. In NeurlPS, 2024.
Manya Wadhwa, Xinyu Zhao, Junyi Jessy Li, and Greg Durrett. Learning to refine with fine-grained natural language feedback. In EMNLP, 2024.
David Wan, Koustuv Sinha, Srini Iyer, Asli Celikyilmaz, Mohit Bansal, and Ramakanth Pasunuru. Acueval: Fine-grained hallucination evaluation and correction for abstractive summarization. In ACI, 2024.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In ACL, 2024.
Siwei Wu, Zhongyuan Peng, Xinrun Du, Tuney Zheng, Minghao Liu, Jialong Wu, Jiachen Ma, Yizhi Li, Jian Yang, Wangchunshu Zhou, et al. A comparative study on reasoning patterns of openai's ol model. arXiv preprint Xro:224400.13: 2024.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. NeurIPS, 2023.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiao chong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. Xiv preprint arXiv:2501.09666 2025a.
Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, et al. Redstar: Does scaling long-cot data unlock better slow-reasoning systems? ar preprint ar Xix::2501.11284 2025b.
Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, and Markus Freitag. LImrefine: Pinpointing and refining large language models via fine-grained actionable feedback. In NAACL, 2024.
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. In ICLR, 2024.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. Summit: Iterative text summarization via chatgpt. In EMNLP, 2023.
Kaiyi Zhang, Ang Lv, Yuhan Chen, Hansen Ha, Tao Xu, and Rui Yan. Batch-icl: Effective, efficient, and order-agnostic in-context learning. In ACL,, 2024.
Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al. Evaluation of openai ol: Opportunities and challenges of agi. arxiv preprint arXiv:2409.18486, 2024.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. Mediasum: A large-scale media interview dataset for dialogue summarization. In NAACL, 2021.
13

---

Published as a conference paper at COLM 2025
Pipeline
Dim.
Feedback
Refinement
Reasoning
Pl
Single
Independent
Sequential
Receptive
P2
Multi
Independent
Sequential
Receptive
P3
Multi
Dependent
Sequential
Receptive
P4
Multi
Dependent
Simultaneous
Receptive
ReFeed
Multi
Dependent
Simultaneous
Reflective
Table 6: Five different pipelines for refinement on multi-dimennsons.
A Summary of Refinement Pipelines
A.1 Refinement Prompts
Table presents the configurations of five distinct pipelines. Each reason+refine module in these pipelines receives three inputs: a document, summary, and feedback. The instruction to reason feedback across three dimensions is provided in prompt, and the order of feedback and instruction is interchangeable. Tables 17-19 present the prompts for each pipeline. We also provide the previous two refinement pipelines DCR (Wadhwa et al., 2024) and ACUEval (Wan et al., 2024) in Tables 20-21, ensuring that their refinement prompts follow the same format as ours, while the remaining prompts are directly adopted from the original papers.
B Metric for Summary Quality
By recent study (Song et al., 2024), we employ three core petrics-ffithfulneen foleesseess and conciseness evaluate summary quality.
Faithfulness Score. The faithfulness metric determined by examining sentence-level factual correctness. Let = (si,..., sN be a summary composed of N sentences, where each s; is the i-th sentence. Define Stact S the subset of sentences verified "factually correct." The faithfulness score of S with respect to the source document D is given by:
Stact [S]
Faithfulness(D,, (D,)))
(1)
This measures the fraction of sentences in the summary that are deemed factually accurate relative to the total number sentences.
Completeness and Conciseness Score. Let K= kM represent set of key facts, where M denotes the total count of these facts. By leveraging alignments between key facts and summary sentences, we construct bipartite graph M (K, S,,, where the edge set + s k E KAS E S}. Here, k + indicates that the key fact appears within sentence s in the summary. Based on this graph, the completeness and conciseness scores for the summary 5 are computed as:
E = (k,s)
1 (k,s) E}]
Completeness(K,55
(2)
K
Conciseness Ks (k,s) E E}
(3)
S
Here, denotes set cardinality. The completeness score captures the extent to which key facts are included the summary, whereas the conciseness score evaluates how effectively the summary condenses and integrates those key facts.
Composite Score. For scenarios involving multi-dimenssiie feedback, calculate the average of the three percentage scores scores-faile faithfulness, completeness, and conciseness to derive composite score, which is used to assess summary's comprehensive quality.
14

---

Published as a conference paper at COLM 2025
Model Name
Checkpoints
Bart-large
Bttebook/bart-lrggeeee
Pegasus-large
google/
Flan-t5-large
Phi-2
microsoft/phi-
Mistral-7b-inst
mistralai/
Sestrala//MisstaaAAA-sssttttttt
Mixtral-8x7b-inst
misstala//Miittalkk377--stttue
Llama2-13b-chal
meta-llnna/LLama222BBBhhhhha
Mistral-nemo
mistrahi/MMattaa/nnnnnnnnnnnnnn
Gemma2-27b-ing
google//gemmas22277-itt
Llama3-70b
metalllama//Mettllaan--------
Claude-instant
claude-instant (AWS Bedrock)
GPI-3.5turbo
gpt-3.5tturbo-0125 (OpenAl) gpt-4-0125.preview (OpenAI)
GPT-4
Table 7: Checkpoints of the 13 summarizers. For open-source models, we use publicly available checkpoints from Huggingface, while for proprietary models, utilize paid APT services by OpenAl and AWS Bedrock.
Text: (input text]
Instruction: Summarize the Text.
Provide your answer in JSON format. The answer should be dictionary with the key "summary" containing generated summary string: l" I"summary": "your summary"!
JSON Output:
Table 8: Prompt to generate summary
C Initial Feedback Creation Details
C.1 Input Document Sourcing
Acquiring feedback from a diverse range of source documents is essential for thoroughly evaluating the limitations of contemporary text summarizers, particularly regarding varia- tions in input domain, text length, and format (dialogue vs. non-dialogue) (Lee et al., 2024; Song al., 2025). Following Song et al. (2025), we compile input texts from multiple datasets that span 7 distinct domains, covering spectrum from brief to extensive texts and including both non-dialogue and dialogue formats. Specifically, we randomly sample 200 texts from the training set of each dataset, which comprises four non-dialogue sources C000-NNNDDM (news) (Nallapati et al., 2016), Wikihow (lifestyle) (Koupaee Wang, 2018), GovReport (report) (Huang et al., 2021), and PubMed (medical literature) (Cohan et al., 2018) as well (dailylife) (Chen et al., 2021), MediaSum (interview) (Zhu al., 2021), and Meeting (meeting) (Hu et al., 2023).
as three dialogue datasets datasets-DialogSuue
C.2 Summary Generation
We produce summaries of varying quality and distribution by leveraging suite of 13 dif- ferent language models, which are listed in Table 7. To ensure a consistent and reproducible approach to summary generation, employ specific prompts that are detailed in Table 8.
C.3 Initial Feedback Generation
We employ FineSurE (Song et al., 2024) to obtain multi-dimenssonal and fine-grained initial feedback. Specifically, feedback on faithfulness is collected at the sentence level, while completeness and conciseness assessed at the key fact level. FineSurE utilizes LLMs
15

---

Published as a conference paper at COLM 2025
to conduct fact-checking task for faithfulness and key fact alignment task for the other two dimensions. Since source datasets do not include human-annotated key facts, we also extract the list of key facts in document using the prompt introduced in Song et al. (2024). The prompts used for FineSurE. are provided in Table 22-23.
The binary labels generated by this step utilized in the reason+refine module. Addition- ally, these labels are aggregated into three percentage scores each corresponding to one of the evaluated dimensions assess summary quality, described in Appendix B.
D Reasoning Creation Details
We map the initial feedback, provided as binary labels, where a label of 1 indicates that refinement is needed, into simple textual structure corresponding to the three dimensions, as detailed below:
Mapping of feedback
Summary Sentences: [S1, S2, S3] Key Facts: [K1, K2, K2]
Faithfulness Feedback: [1, 0, 0]
Completeness Feedback: [1, 1, 0]
Conciseness Feedback: |0, 0, 1]
Mapping:
Faithfulness Feedback:
These summary sentences are factually inconsistent with the Document:
[S1]
Completeness Feedback:
These key contents are missing in the summary:
-[K]]
-[K2]
Conciseness Feedback:
These summary sentences do not contain key content:
[S3]
Table 24 shows the prompt used to generate Long-CoT data in SumFeed-CoT. this prompt, the inputs consist of the document, summary, and detection labels transformed into verbal form. The output is generated through reflective reasoning and ends with a revised summary presented in boxed format.
E Dataset Details
We present a detailed statistical analysis the SumFeed-CoT dataset, which consists of <document, summary, feedback> triplets as input and <reasoning + revised summary> as output. Each reasoning in the dataset is a Long-CoT with more than 5K tokens. Table presents statistics of the SumFeed-CoT training set, including the average token counts for documents, summaries, and reasoning steps, as well as the distribution feedback labels across faithfulness, completeness, and conciseness dimensions.
16

---

Published as a conference paper at COLM 2025
Document Token
Summary Token
Reasoning
Faith. Feedback
Comp. Feedback
Conc.
# of Documents
Token
Feedback
708.2
76.6
1393.5
3.2
2.6
1.4
7713
(14-4972)
(2-460)
(523-4470)
(1-17)
(0-17)
(0-11)
Table 9: Statistics of the SumFeed-CoT training set, showing the average token counts for documents, summaries, and reasoning based on the LLaMA-3.1 tokenizer, along with the average number of feedback labels across three dimensions: faithfulness, completeness, and conciseness. The corresponding min-max ranges are provided in parentheses below each value.
Parameter
Value
Backbone model
LLaMA-3.1-8B-Imstt
Batch size
32
Epochs
Learning rate
le44
Max sequence length
10,000
LoRA rank/alpha
16 / 32
Warmup ratio
0.03
I.R scheduler type
cosine
Table 10: Hyperparameters of the training configuration.
F
Training and Inference Detail
We fine-tune LLaMA-3.1 10-MAA-3.1-8B-A 1-1-8--IInstuutt using LoRA (Hu et al., 2022) and DeepSpeed (ZeRO-3) (Rasley et al., 2020) on NVIDIA L40S GPUs. Table 10 summarizes the training configu- rations for SFT. We incorporate a system prompt to promote Lon-CCoT-stylee generation following Min et al. (2024). We the same prompt for user prompt excluding the best summary and error types to train student model. In the training phase, the revised summary wrapped with the <answer> tag, while all preceding intermediate reasoning steps are encapsulated in <think tags. Both the best summary and the error type are omitted from the training process, as such details may not available during the inference phase. Table 25 shows an example of our prompt for training.
Table 26 presents the inference prompt for ReFeed. In the inference phase, the model trained on SumFeed-CoT generates a revised summary and reasoning in the same format as in the training phase. The model's reasoning process involves backtracking and filtering out noisy feedback, enabling it to effectively process multi-dimensional feedback while refining its summary towards what it perceives as the best summary.
G Automated Evaluation Details
To evaluate the quality of revised summaries, we use the same FineSurB prompt described in Section C.3. Additionally, perform automated evaluations using G-Eval (Liu et al., 2023a), which assigns a scalar score from to to assess summary quality. We use a customized G-Eval prompt with DAMMA--.1-70D-instruue to assess faithfulness, completeness, and conciseness, following Song et al. (2024)
G.1 Automated Evaluation using G-Eval
Table 11 presents the automated evaluation results using G-Eval. Despite the change in evalu- ation metric, the trade-offs introduced by single-dimension refinement and the bias induced by order remain consistent. Notably, ReFeed demonstrates superior performance compared
17

---

Published as a conference paper at COLM 2025
Pipeline
Dim.
Dep.
Simul.
Reflect.
Faith.
Comp-
Conc.
Avg.
Before Refine
4.22
3.04
3.42
3.35
P1-Faith
Faith.
X No
X No
X No
4.33" (40.11)
3.03 (-0.0n
3.46 (40.04)
3.60" (+0.05)
P1-Comp
Comp.
No
No
X
4.44 (+0.22)
3.69 (+0.65)
3.59' (+0.17)
3.90 (+0.35)
P1-Cons
Conc.
X No
X No
X No
4.42 (+0.20)
2.97 (-0.07)
3.66 (+0.24)
3.68 (+0.13)
P2
Multi
X No
X No
X No
1.46'(40.4)
3,28 (40.24)
3.67"(40.23)
(40.23)
3,80 (+0.25)
P3
Multi
Yes
X No
No
4.39 (+0.17)
3.32 (40.28)
3.58 (40.16)
3.76 (+0.21)
P4
Multi
/ Yes
Yes
X No
4.44 4.44" (40.22)
3,41 ((40.77)
3.64((40.22)
3.64 (40.22)
3.83 (+0.28)
ReFeed
Multi
/
Yes
Yes
/ Yes
4.53 (+0,29)
3.62 (+0.58)
3.68 (+0,26)
3.94 (+0.39)
Table 11: Summary quality of revised summaries on faithfulness, completeness, and concise- ness across seven refinement pipelines using G-Eval. The best scores are marked in bold. denotes statistically significant difference with p < 0.05 according to a paired bootstrap test.
Pipeline
Dim.
Dep.
Simul.
Reflect.
Faith.
Comp.
Conc.
Avg.
Before Refine
75.2
43.4
74.5
64.4
P1-Faith
Faith.
No
No
(+4.1)
42.6' (-0.8)
77.1"
66.3" 66.3'(41.9)
P1-Comp
Comp.
x No
No
X No
74.8 (-0.4)
57.7 (+14.3)
76.0 (+1.5)
69.5 (+4.1)
Pl-Cons
Cone.
X No
No
No
77.1 (+1.9)
40.7" (-2.7)
83.0" (+8.5)
66.9 (+2.5)
P2
Multi
X No
No
No
72.7' (-2.5)
48.3' (+1.9)
81.3" (+6.8)
67.4 (43.0)
P3
Multi
Yes
No
No
73.7" (-1.5)
50.0 (+6.6)
77.4 (+2.9)
67.0" (+2.6)
P4
Multi
Yes
Yes
No
76.5 (+13)
52.1 (+8.7)
80.4 (+5.9)
69.7' (45.3)
ReFeed
Multi
Yes
Yes
Yes
77.2 (42.0)
55.6" (+12.2)
80.5 (+6.0)
71.1 (+6.7)
Table 12: Summary quality of revised summaries on faithfulness, completeness, and con- ciseness across seven refinement pipelines using automated evaluator. The best scores are marked in bold. denotes statistically significant difference with p 0.05 according to a paired bootstrap test.
as the FineSurE's
to other pipelines, observed in Section This demonstrates proficient performance and robustness of ReFeed across different evaluation frameworks.
G.2 Automated Evaluation using Different LLM
We also use another LLM, LLAMA-31-70B-inntruuc for FineSurE, to confirm the performance of each pipeline. Table 12 presents the automated evaluation results for the revised summary obtained by replacing the backbone model for FineSurE with 1.MMA--.1-70B-insti. Although the scores across three dimensions were somewhat lower, the overall trends were consistent with those observed using other evaluation metrics. Moreover, the evaluation reveals more pronounced order bias in P2 and P3 as indicated by the lower faithfulness scores.
H Results by Human Evaluation
We conduct human evaluation in Section 5.2 to determine which multi-dimensional refinement pipeline best aligns with human judgment, excluding pipeline P1, which is pipeline for single dimension. For human evaluation, randomly sample 15% of the documents from the test dataset and assess the revised summary based on the model summary for these documents.
18

---

Published as a conference paper at COLM 2025
Pipeline
Dim.
Dep.
Simul.
Reflect.
Faith.
Comp.
Conc.
Avg.
Before Refine
77.4
42.8
80.6
66.9
P2
Multi
No
X No
No
77.3 (-0.1)
45.8 (+3.0)
84.5 (+3.9)
69.2 (+2.3)
P3
Multi
/ Yes
X No
X No
77.8 (+0.4)
50.8 (+2.8)
79.3 (+2.4)
69.3 (+2.4)
P4
Multi
Yes
/ Yes
No
84.5 (+7.1)
52.0 (+9.2)
81.1 (+0.5)
72.5 (+5.3)
ReFeed
Multi
/ Yes
/ Yes
/ Yes
87.3 (+9.9)
54.2 (+11.4)
88.7 (+8.1)
76.7 (+9.8)
Table 13: (Human evaluation) summary quality of revised summaries on faithfulness, com- pleteness, and conciseness across seven refinement pipelines based on human evaluation. The best scores are marked in bold.
Annotation Protocol. Our human evaluation follows the protocol in (Lee et al., 2024). Specifically, human annotators assess faithfulness through fact verification task, in which they identify the presence of factual errors in each sentence. Completeness and conciseness are evaluated through a key fact alignment task, which involves matching key facts with sunmary sentences. Conciseness is evaluated by verifying whether each sentence contains the essential key facts, while completeness is measured by determining whether every key fact is included in the summary. For the evaluation, we provide the key facts in UniSumEval. The calculation of the human scores across three dimensions follows the formulations in Appendix B.
Annotation Qualification and Compensation. We recruit three postgraduate students with proficient English skills to assess the quality of revised summaries. Only annotator who pass an English proficiency test-which simulates the evaluation of sentence-level errors for assessing faithfulness and verifying that key facts are included are hired. Annotators receive compensation exceeding the U.S. minimum wage.
Human Evaluation Results. The inter-annotator agreement was moderate for both tasks, with Krippendorff's of 0.45 for fact verification and 0.63 for key fact alignment. Table 13 shows human evaluation results across three dimensions. ReFeed achieves significant improvements, aligning with the findings highlighted in Section 5.2. While other pipelines show only marginal gains approaching zero in either faithfulness or conciseness, failing to achieve balanced performance across all dimensions, ReFeed successfully overcomes these shortcomings. The strong alignment between automated evaluators and human evaluation scores provides compelling evidence for the effectiveness of reflective reasoning.
I Additional Experiment Details
L11 Training Details
We trained CLAMMA--.1--B-IInSii on reasoning data generated using the P4's prompt, with QwQ-32B-preview as the teacher model. The training focused on making the model more handling feedback through receptive reasoning, excluding explicit goals and guidelines for reflective reasoning. The reasoning generation prompt was modified to align with the model's inherent reasoning structure as seen in Table 27. We follow the same quality control process as SumFeed-Col and also shuffle the dimension order. The train settings and formats are the same as those in Section 4.2.
I.2 Training Data Size
Table 14 presents statistics on the data sizes resulting from the two data construction pro- cesses. While the initial <document, summary, feedback> triplets are the same, ReFeed achieves more successful reasoning, ultimately retaining approximately 8K instances, whereas P4 retains only 5K. This result suggests that reflective reasoning exhibits higher success rate in reasoning compared to P4.
19

---

Published as a conference paper at COLM 2025
Reasoning Format Filtering
Verification- based Filtering
Ratio of Successful Refinement
Reasoning Strategies
Original Data
Pipeline
Feedback
LLaMA 3.1-70B (High)
14,505
7,382
2,806
38.01%
P4
Receptive
LLaMA 3.1-8B (Low)
14,496
7,860
2,510
31.93%
LLaMA 3.1-70B (High)
14,505
9,179
3,922
42.73%
ReFeed
Reflective
LLaMA 3.1-8B (Low)
14,496
9,066
3,791
41.82%
Table 14: Comparison of the data construction process between P4 and ReFeed. The two feedback generation models use checkpoints obtained through instruction tuning.
1.3 Inference Latency
Table 15 demonstrates that ReFeed not only reaches performance levels comparable to the teacher model, QwQ-32B-previen but also does so with significantly enhanced computa- tional efficiency. Under conditions of high-quality feedback, ReFeed achieves composite scores nearly equivalent to those of the teacher model while achieving inference speeds approximately 4x faster. Moreover, when compared against alternative pipelines such as P4, Our approach offers a robustness advantage in terms of feedback quality.
I.4 Comparison of Receptive and Reflective Reasoning
To examine the observation in Section 5.5,
Model
Pipeline
Composite High
Inference Time
perform qualitative analysis of two distinct reasoning policies. Table 28-29 il- lustrates reasoning examples under recep- tive (P4) and reflective reasoning (ReFeed). Our analysis reveals that reflective reason- ing demonstrates superior robustness in handling inaccurate feedback. For instance, as shown in Table 28, uncritically accept- ing flawed feedback on conciseness may undermine completeness- sesfeness--poeentillly re- sulting in the complete removal sen- tence. In contrast, ReFeed evaluates and fil- ters such feedback, thereby achieving well- rounded improvements in multiple dimen- sions while enhancing completeness.
Low
P44k
55.3*
54.3*
54.5s
L3.1 8B
ReFeed4k
57.1
55.7'
53.9s
ReFeedgk
57.8"
56.5'
40.0s
QwQ 32B (Teacher)
58.1"
57.3"
196.6s
Table 15: Comparison of reflective reasoning over receptive reasoning on low- and high- quality feedback setups as P4fT indicates P4 fine-tuned with 4K data. 4K and 8K refer to training data size. For inference time, use batch size of 1 on two NVIDIA L40S GPUs.
I.5 Comparison with Direct Refinement Approach
We compare ReFeed with the "Direct Re- finement" approach, which uses the same LLaMA-3.1 8B backbone but is trained to generate the revised summary directly from the original summary. This approach ex- cludes any detection or reasoning modules and is designed to take the original sum- mary as input, producing revised sum- mary in single step, specified in the prompt shown Table 30.
Pipeline
Faith.
Comp.
Conc.
Avg.
Before Refine
78.0
46.4
76.4
66.9
85.0
46.9
80.3
70.8
DR
(+7.0)
(+0.5)
(+3.9)
(+3.9)
ReFeed
82.7
60.0
83.4
75.3
(+4.7)
(+13.6)
(+7.0)
(+8.4)
Table 16: Comparison between Direct Refine- ment (DR) and ReFeed. Faith., Comp., and Conc. denote Faithfulness, Completeness, and Conciseness, respectively.
As shown in Table 16, while Direct Refine- ment boosts faithfulness by 7.0 with little change in other metrics, showing gains of only 0.5 in completeness and 3.9 in concise- ness, producing a limited average of 3.9,
20

---

Published as a conference paper at COLM 2025
ReFeed shows balanced improvements across all dimensions, more than doubling the aver- age score to 8.4. This improvement can be attributed to the ReFeed's reflective reasoning process, which processes explicit feedback across multiple dimensions, selectively integrates while filtering out noisy feedback with adverse effects, and employs backtracking to ensure balanced improvement, thereby achieving substantial gains across all dimensions rather than minor gains in isolated ones.
21

---

Published as a conference paper at COLM 2025
Instruction for reasoning
Faithfulness: reason about factual inconsistencies in the summary sentence.
Completeness: reason about why the summary is each missing key content. -Conciseness: reason about why the summary does not contain key content and contains unnecessary details.
User Prompt
Your task is to reason about the provided feedback and to refine the summary based on the provided feedback.
Instruction:
Give reasons about the provided feedback by considering the relevant dimension and provide suggested fix to the summary:
[Instruction for reasoning on the target dimension (single))
2. Revise the summary by incorporating the feedback.
Provide your response in the following format:
Feedback Reasoning:
[Your reasoning feedback and suggested fix]
Revised Summary:
|Your revised summary]
"
Document:
(Document)
Summary:
(Summary)
Feedback:
[Feedback]
Table 17: Prompt for Pl and P2. For P1, a prompt is sent once for a single target dimension, whereas in 122 prompt is sent three times for each target dimension. In P2, "Summary" in inputs can be revised summary in previous step. The prompt includes a single reasoning instruction specific to the target dimension among the three dimensions.
22

---

Published as a conference paper at COLM 2025
Instruction for Reasoning
Faithfulness: reason about factual inconsistencies in the summary sentence.
-oompletnness: reason about why the summary is each missing key content.
-Conciseness; reason about why the summary does not contain key content and contains unnecessary details.
User Prompt (Turn 1)
Your task is to reason about the provided feedback and to refine the summary based on the provided feedback.
Instruction:
1. Give reasons about the provided feedback by considering the relevant dimension and provide a suggested fix to the summary:
[Instruction for reasoning on the dimension A]
[Instruction for reasoning the dimension BJ
[Instruction for reasoning on the dimension C}
2. Revise the summary by incorporating the feedback.
Provide your response in the following format:
Feedback Reasoning:
[Your reasoning on feedback and suggested fix] Revised Summary:
|Your revised summaryl
Document:
[Document]
Summary:
(Summary)
Feedback:
[Feedback on the dimension A]
User Prompt (Turn N)
Refine your refined summary again based the provided feedback. Critique and Refine again based on the provided feedback.
Feedback:
[Feedback on Nth dimension]
Table 18: Prompt for P3. In Turn 1, instruction for reasoning on three dimensions are provided. From the next turn, only feedback is given.
23

---

Published as a conference paper at COLM 2025
Instruction for Reasoning
Faithfulness: reason about factual inconsistencies in the summary sentence.
CCompleteness: reason about why the summary is each missing key content.
-Conciseness: reason about why the summary does not contain key content and contains unnecessary details.
User Prompt
Your task is to reason about the provided feedback and to refine the summary based on the provided feedback.
Instruction:
1. Give reasons about the provided feedback by considering the relevant dimension and provide a suggested fix to the summary:
[Instruction for reasoning on dimension A}
[Instruction for reasoning on dimension BJ
[Instruction for reasoning dimension Ci
Revise the summary by incorporating the feedback.
3. Provide your response in the following format: "4
Feedback Reasoning:
[Your reasoning on feedback and suggested fix]
Revised Summary:
[Your revised summary]
*
Document:
(Document)
Summary:
[Summary]
Feedback:
[Feedback on dimension A]
[Feedback on dimension B)
[Feedback dimension C)
Table 19: Prompt of P4. The prompt includes all reasoning instructions, with their order
aligned to the sequence of dimensions in the feedback.
24

---

Published as a conference paper at COLM 2025
User Prompt (Reasoning Model)
I summarized the following document:
(Document)
Summary of the above document:
(Summary)
Reason about the factually inconsistent span in the sentence. A span is factually inconsistent cannot substantiated by the document. Give reasons for the factual inconsistency, point to the error span by stating "The error span: <span from sentence> and end your answer with suggested fix to the summary.
User Prompt (Refinement Model)
Your task is to refine the summary based on the provided feedback.
Instruction:
Revise the summary by incorporating the feedback.
2. Provide your response in the following format: .44
Revised Summary:
[Your revised summary]
Document:
(Document)
Summary:
(Summary)
Feedback:
[Feedback]
Table 20: Prompt of DCR. This pipeline separates the reasoning and refinement modules. First, the reasoning module generates critique based detected labels. Then, the output from the reasoning module is used as the feedback input for the refinement model.
25

---

Published as a conference paper at COLM 2025
User Prompt
Your task is to refine the summary based on the provided feedback.
Instruction:
1. Revise the summary by incorporating the feedback.
Provide your response in the following format:
Revised Summary:
[Your revised summary]
Document:
(Document)
Summary:
(Summary)
Feedback:
The summary is not consistent with the source text. The source text does not mention the following facts:
[Incorrect Atomic Fact 1)
[Incorrect Atomic Fact 2)
[Incorrect Atomic Fact N]
The summary should not include information that is not present in the article. Please check the document for the correct information and make appropriate edits.
Table 21:
Prompt for ACUEval. ACUEval uses detection labels at the atomic fact level. We follow the detection prompt from Wan et al. (2024) and adjust the refinement prompt to ensure compatibility with other pipelines.
26

---

Published as a conference paper at COLM 2025
You will receive document followed by corresponding summary. Your task is to assess the factuality of each summary sentence across nine categories: no error: the statement aligns explicitly with the content of the document and is factually consistent with it. * out-of-context error: the statement contains information not present in the document.
entity error: the primary arguments (or their attributes) of the predicate wrong. predicate error: the predicate in the summary statement inconsistent with the document.
circumstantial error: the additional information (like location or time) specifying the circumstance around predicate is wrong. grammatical error: the grammar of the sentence is wrong that it becomes meaningless.
coreference error: a pronoun or reference with wrong or non-existing antecedent. linking error: error in how multiple statements are linked together in the discourse (for example temporal ordering causal link). other error: the sta tement contains any factuality error which is not defined here.
Fact Check
Instruction:
First, compare each summary sentence with the document.
Second, provide single sentence explaining which factuality error the sentence has.
Third, answer the classified error category for each sentence in the summary.
Provide your answer JSON format. The answer should be a list of dictionaries whose keys are "sentence", "reason", and "category":
["sentence": "first sentence", "reason": "your reason", "category": "no error", "sentence": "second sentence", "reason": "your reason", "category": "out-of-context error", "sentence": "third sentence", "reason": "your reason", "category"; "entity error".]
Document:
(document)
Summary with (# of sentences] sentences: (sentences)
JSON Output:
Table 22: Prompt of the FineSurE for fact-checking tasks.
27

---

Published as a conference paper at COLM 2025
You will be provided with a transcript. Your task decompose the summary into a set "key facts".
A "key fact" is a single fact written as briefly and clearly as possible, encompassing at most 2-3 entities.
Here are nine examples of key facts to illustrate the desired level of granularity:
* Kevin Carr set off on his journey from Haytor.
*
Kevin Carr set off on his journey from Dartmoor.
Kevin Carr set off on his journey in July 2013.
Kevin Carr is less than 24 hours away from completing his trip.
Kevin Carr
around the world unsupported.
Kevin Carr ran with his tent.
Key Fact Extraction
Kevin Carr is set break the previous record.
Kevin Carr is set to break the record by 24 hours.
Instruction:
First, read the summary carefully. Second, decompose the summary into (at most 16) key facts.
Provide your answer JSON format. The answer should be a dictionary with the key "key facts" containing the key facts as a list: ("key facts":
facts":""firstt ["first key fact", 'second key facts", "third key facts"]]
Summary: (summary)
JSON Output:
You will receive summary and set of key facts for the same document. Your task is to assess if each key fact is inferred from the summary.
Instruction:
First, compare each key fact with the summary.
Second, check if the key fact is inferred from the summary and then response "Yes" or "No" for each key fact. If "Yes", specify the line number(s) of the summary sentence(s) relevant to each key fact.
Provide your answer in JSON format. The answer should be a list of dictionaries whose keys are "key fact", "response", and "line number": ["key fact": "first key fact", "response": "Yes", "line number": [1], "key fact": "second key fact", "response": "No", "line number": [], "key fact": "third key fact", "response": "Yes", "line number": 2, 3]1
Key Fact Alignment
Summary:
(summary)
(# of key facts) key facts: (key facts)
JSON Output:
Table 23: Prompt of the FineSurE for key fact extraction and key fact alignment tasks.
28

---

Published as a conference paper at COLM 2025
Instruction for Reasoning
Faithfulness: Does this feedback accurately identify summary sentences?
Out-of-article Error: Facts, new information or subjective opinions not found or verifi- able by the document.
Incorrect or misreferenced details about key entities such as names, dates, locations, numbers, pronouns, and events in the summary.
Entity Error:
Relation Error: Misrepresented relationships, such as incorrect use of verbs, prepositions, and adjectives.
Sentence Error: the entire sentence entirely contradicts the information in the document.
Completeness: Does this feedback correctly identify missing key content in the sum- mary?
Conciseness: Does the feedback correctly identify sentences that include unnecessary details and lack key content?
User Prompt
Your goal is to deliberate on the provided feedback and propose actionable and specific aggregated feedback based on it.
Provide your response in the following format:
**Final Reviesed Summary**.
NM [[Xooxxed{) tour revised summary]]
Instruction:
Deliberate on the characteristics an ideal summary should achieve.
2. Assess and choose the validity of the given feedback in improving the summary
considering feedback quality criteria:
[Instruction for reasoning on dimension A]
Instruction for reasoning dimension BJ
[Instruction for reasoning on dimension C)
3. Aggregate the valid feedback and Revise summary by incorporating it.
Carefully check whether each feedback and suggestion compromise other quality dimensions. Backtrack your reasoning If you need to.
Document:
(Document)
Summary:
(Summary)
Ideal Summary:
(Best Summary)
Feedback:
[Feedback dimension A)
[Feedback dimension B)
[Feedback on dimension C)
Table 24: Reflective reasoning prompt for generating reasoning data in SumFeed-CoT (ReFeed)
29

---

Published as a conference paper at COLM 2025
<I beginn of text 1>< start header [d| > end header, > Your role as an assistant involves thoroughly exploring questions through systematic long thinking process before providing the final precise and accurate solutions...< eot id_ 1>
System
<I start header_ starhhe id >usere end header id 1> Your goal is to deliberate on the provided feedhack and propose actionable and specific aggregated feedback based on it.
Instructions:
1. Deliberate un the characteristics an ideal summary should achieve.
2. Assess and choose the validity of the given feedback in improving the summary considering feedback
quality criteria:
Conciseness: Does the feedback correctly identify sentences that include unnecessary details and lack key
content?
Faithfulness:: Does this feedback accurately identify summary sentences that align with one the following four factual inconsistencies types?:
Completeness: Does this feedlbuck correctly identify missing key content in the summary?
3. Aggregate the valid feedback and Revise summary by incorporating it. revision is needed, just answer 'no revision needed".
4. Carefully check whether each feedback and suggestion compromise other quality dimensions. Backtrack your reasoning If you need to.
Document:
Personl: Good Morning, What can I do for you? 'n' Person2: Good Moming, I hhve a bad cough, and want to see an internist. 'nn' Personl: Do you have a registration card? 'nn Person2: No, I don't. I'm tourist. Then you need to register age, occupation, address and contacting number. An' Person2: Here you are. 'n' Personl: Thanks. Which kind medicine do you prefer, western medicine traditional Chinese medicine? 'n' Person2: I'd like try traditional Chinese medicine. I heard it really works quite well if you know what you' re doing. 'an Personl: All right. Yes, does. The registration fee is yuan, please.
'nn' Personl:
patient. Can I have look you ID card? need your name,
Input (User)
Summary:
A tourist asks to see an internitt because he has a bad cough. The doctor asks for his name, age, occupation, address and contacting number. The tourist says he would like to try traditional Chinese mediciie.. The internistt agrees to see him for fee of 7 yuan.
Feedback:
"*Conciseness Feedback
These summary sentences do contain key content:
Sentence 2: The doctor asks for his name, age, occupation, address and contacting number.
Sentence 4: The internist agrees to see him for fee 7 yuan.
**Faithfulness Feedback
These summary sentences are factually inconsistent with the Document:
Sentence 4: The internist agrees tu see him for fer of yuan.
""Completeness Feedback t*
These key contents missing the summary:
Missing key content I: The registration fee is yuan.
<leeotid|>
<I begin of text bex istart header, jid bassistante end header_idis <thinks
Alright, I have this task to review summary based on given ducument and some feedback. I need to make sure summary accurate, complete, and concise. Let's break this down step by step.
First, I need to understand what an ideal summary should look like. A good summary should capture the main points of the document without including unnecessary details....
Now, let's look at the feedback provided:
Faithfulness Feedbbckk:
Sentence 4: "The internist agrees to see him for fee of 7 yuan." This feedback suggests that this sentence is factually inconsistent with the document. need to check the document to see if this is accurate.
In document:
Personl (likely a staff member) says, "The registration fee 7 yuan, please." There's 's no explicit statement that internist agrees to see him for that fee. The seems to be for registration, not directly for the internist's service....
Output (Assistant)
Let me try to revise the summary based on this analysis. First, clarify the roles: It's likely that Personl is receptionist or a staff member, not the internist, The internist is probably the doctor the tourist wants to see...
</think>
<answer>>
*FFinal Revised Summary:"
|[abxeed] text/A tourist visits medical facility with a bad cough and requests to see an internist. Since he doesn't have registration card, he needs register a new patient. The staff member asks for his card to record his name, age, occupation, address, and contact number. The tourist expresses preference for yuan.]) A]
traditional Chinese medicine, and the staff informs him that the registration
</answer> leot_ id >>
Table 25: An example of an input (system prompt and user prompt) and its corresponding reasoning and revised summary (assistant prompt) used to train LLaMA-3.1-8binnste
30

---

Published as a conference paper at COLM 2025
Instruction for Reasoning
Faithfulness: Does this feedback accurately identify summary sentences? Completeness: Does this feedback correctly identify missing key content in the sum- mary?
Conciseness: Does the feedback correctly identify sentences that include unnecessary details and lack key content?
System Prompt
Your role as an assistant involves thoroughly exploring questions through systematic long thinking process before providing the final precise and accurate solutions. This requires engaging in comprehensive cycle of analysis, summarizing, exploration, re- assessment, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Think and Answer. In the Think section, detail your reasoning process using the specified format: <think> /thought with steps separated ted with '\n \n') </think> Each step should include detailed consid- erations such as analyzing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Answer section, based on various attempts, explorations, and reflections from the Think section, systematically present the final solution that you deem correct. The solution should remain a logical, accurate, concise expression style and detail necessary step needed reach the conclusion, formatted as follows: <answer> (final formatted, precise, and clear solution] </answer> Now, try to solve the following question through the above guidelines:
User Prompt
Your goal is to deliberate on the provided feedback and propose actionable and specific aggregated feedback based
Instruction:
Deliberate on the characteristics ideal summary should achieve.
2. Assess and choose the validity of the given feedback in improving the summary considering feedback quality criteria:
[Instruction for reasoning on dimension A]
[Instruction for reasoning on dimension B)
[Instruction for reasoning on dimension C/
Aggregate the valid feedback and Revise summary by incorporating it.
4. Carefully check whether each feedback and suggestion compromise other quality dimensions. Backtrack your reasoning you need to.
Document:
(Document)
Summary:
(Summary)
Feedback:
[Feedback on dimension A)
[Feedback on dimension B)
[Feedback on dimension C)
Table 26: Prompt of ReFeed for inference. The prompt includes all reasoning instructions, tailored for reflective reasoning to validate noisy feedback, with their order aligned to the sequence of dimensions the feedback.
31

---

Published as a conference paper at COLM 2025
Instruction for Reasoning
Faithfulness: reason about factual inconsistencies in the summary sentence.
-Completeness: reason about why the summary is each missing key content. -Conciseness: reason about why the summary does not contain key content and contains unnecessary details.
User Prompt
Your task to reason about the provided feedback and to refine the summary based the provided feedback.
Provide your response in the following format:
**Final Reviesed Summary**.
AM \boxed// Ytuxt/Your revised summary]/N]
Instruction:
1. Give reasons about the provided feedback by considering the relevant dimension and provide a suggested fix to the summary:
[Instruction for reasoning on dimension A]
[Instruction for reasoning on dimension B)
[Instruction for reasoning dimension Ci
2. Revise the summary by incorporating the feedback.
Document:
(Document)
Summary:
(Summary)
Feedback:
[Feedback on dimension A)
[Feedback on dimension B)
[Feedback on dimension C₁
Table 27: Prompt for generating reasoning data through receptive reasoning (P4)
32

---

Published as a conference paper at COLM 2025
Attorneys representing 8000+ women who were secretly recorded during pelvic exams by Dr. Nikita Levy at Johns Hopkins-affilitte receive $32 million in legal fees from $190 million settlement. The recordings were discovered in 2013, and Dr. Levy committed suicide ten days later. The settlement one of the largest on record in the U.S. involving sexual misconduct by physician. The case never produced criminal charges, but investigators found 1200 videos and 140 images stored servers in his home. Dr. Levy's patients were traumatized, but their faces were not visible the images and could not be established which patients were recorded. Dr. Levy graduated from Weill Cornell Medical College and completed his internship and residency at Kings County Hospital Center. He began working Hopkins in 1888 and saw roughly 12,600 patients during his 25-year tenure. The attorneys had originally asked for a 35% settlement fee, but Baltimore judge ruled they would only receive 17.25%. The settlement closes a case that threatened the reputation of Johns Hopkins, one of the world's 's leading medical institutions.
affiliaae clinic will
Original Summary
Faithfulness Feedback [0, 0, 0, 0, 0, 0, 1,
Feedback (Ground truth)
Completeness Feedback [0, 0, 0, 0, 0, 1, 0,1.0.0.000000011..0111
Conciseness Feedback [0, 0, 0, 0, 0, 1, 0,
Faithfulness Feedback 0, 1, 0,
Feedback (Machine)
Completeness Feedback [0, 1, 0,
Conciseness Feedback [0, 0, 0, 10,.0000,0000
Faithfulness Score 89%
Original Scores
Completeness Score 71%
(FineSurE)
Conciseness Score : 89%
Feedback Reasoning:
Faithfulness: The summary sentences are factually inconsistent with the Document. The date of Dr. Levy's s start at Hopkins is incorrect (1888 instead of 1988), and the discovery of recordings was not in 2013, but was discovered after co-worker alerted hospital authorities, which led to his firing in February 2013. The sentence about the case never producing criminal charges is also inconsistent with the Document, as it mentions that investigators found 1200 videos and 140 images, but it does not explicitly state that the
never produced criminal charges.
Suggested fix: Revise the summary reflect the correct information.
Completeness: The summary is missing key content, specifically the fact that Dr. Nikita Levy was fired in February 2013 after a co-worker alerted hospital authorities.
P4 (Receptive reasoning)
Suggested fix: Add this key content to the summary.
The summary contains unnecessary details, specificaly sentences 5, 6, and 9, which do not contain key content.
Conciseness:
Suggested fix: Remove these sentences and focus the essential information.
Revised Summary: Attorneys representing more than 8,000 women who were secretly recorded during pelvic exams by Dr. Nikita Levy at a Johns Hopkins-affill affiliated clinic will receive $32 million in legal fees from a $190 million settlement. Dr. Levy was fired in February 2013 after co-worker alerted hospital authorities, and he committed suicide days later. The settlement is one the largest on record the U.S. involving sexual misconduct by a physician. Investigators found 1,200 videos and 140 images stored on servers in his home. The attorneys had originally asked for a 35% settlement fee, but Baltimore judge ruled they would only receive 17.25%.
Revised
Faithfulness Score 100%
Summary Scores
Completeness Score: 57%
Conciseness Score: : 100%
Table 28: An example of refinement through receptive reasoning (P4). Red indicates incorrect feedback or flawed reasoning. Summary scores is evaluated by FineSurE
33

---

Published as a conference paper at COLM 2025
Faithfulness Feedback: : [1. 0, 1, 1, 1,0,1] Completeness Feedback: 1, 0, 0]
Feedback (Machine)
Conciseness Feedback: : |0,0, 0, 1, 1,
<think> "Stop Understanding the Document" First, I need to grasp the main points of the document. It's about settlement involving a gynecologist. Dr. Nikita Levy, who secretly recorded and pholographed thousands of women during pelvic exams at Johns Hopkins-affliated clinie. The settlement for $190 million, with $32 million going the attornys representing over 8,000 women. Dr. Levy was fired in 2013 after a co-worker reported her he later committed suicide. The settlement is one the largest for sexual misconduct by a physician in the U.S., and it closes a case that threatened the reputation of Johns Hopkins.
""Step Evaluating Faithfulless"*
I need to check if these sentences are factually inconsistent with the document.
-Sentence Correct. It accurately states that attorness representing over 8,000 women will receive $32 million from a S190 million settlement.
Sentence 2: Correct. It states that the recordings were discovered in 2013 and that Dr. Levy committed suicide ten days later.
Sentence Correct. It mentions that the case never produced criminal charges and that investigators found 1,200 videos and 140 images servers in his home.
Sentence 5: Correct. It states that Dr. Levy's s patients were traumatized, but their faces were not visible in images, and couldn't it be established which patients were recorded.
Sentence 7: Incorrect. According to the document, Dr. Levy began working at Hopkins in 1988, not 1898, 1888 in the 19th century, and seems like a typo or an erzor in the summaay.
Sentence 9: Correct. It states that the settlement closes case that threatened the reputation of Juhns Hopkins, one of the world's 's leading medical institutions.
So, there's factual inconsistency in sentence regarding the year Dr. Levy started working at Hopkins.
""Step 3: Evaluating Completeness"
Looking at the completeness feedback:
Missing key content 1: 'The gynecologist, Dr. Nikita Levy, was fired in February 2013 after a co-worker alerted hospital authorities." This is indeed a key piece of information that missing from the summary. The document states that Dr. Levy was fired in February 2013 after a co-worker alerted hospital authorities, and then committed suicide ten days later. This important because sets the timeline and explains why Dr. Levy was fired.
""Step 4: Evaluating Conciseness"*
P5 (Reflective reasoning)
need to assess if these sentences contain key content or if they are unnecessary details. Sentence This sentence important because mentions that Dr. Levy's patients traumatized, but their faces were not visible in the images, and it couldn't be established which patients were recorded. This provides context on the impact Dr. Levy's actions.
Sentence 6: This sentence provides Dr. Levy's educational background and where he completed his internship and residency. While factual, it might not be essential for understanding the main points of the summary.
Sentence 7: This sentence is incorrect due to the year being 1888 instead of 1988. It also mentions the number of patients and tenure, which could be considered key content.
Sentence 9. This sentence is important because it highlights the impact of the settlement on Johns Hopkins' reputation.
Given that sentence 7 contains an error and might not be necessary, and sentence is factual but not escentall
"SStep 5: Revising the Summary**
Considering the feedback, I need correct the year Dr. Levy started working at Hopkins, include the missing key content about Dr. Levy's termination, and ensure that the summary remains concise and accurate.
First, correct the year: Instead of "He began working Hopkins 1888," it should be 'He began working at Hopkins in 1988."
'The gynecologist, Dr. Nikita Levy, was fired February 2013 after a co-worker alerted hospital authorities.
Second, include the missing key content:
Third, consider the conciseness of sentences. Sentence 6 provides Dr. Levy's is educational background, which might not be essential for the summary. Sentence contains error and might be unnecessary. Fourth, ensure that all key elements are included. </think>
<answers "Final Revised Summary:
\[bboxed( (ttett/ (ttexx(AAttrreey Attorneys representing over 8,000 wornen who were secretly recurded during pelvic exams by Dr. Nikita Levy at a Johns Hopkins-affiliate clinic will receive $32 million in legal fees from $190 milkoon settlement. Dr. Levy was fired in February 2013 after a co-worker alerted hospital
authorities about her suspicions, and he later committed suicide. The settlement is one of the largest on record in the U.S. involving sexual misconduct by a physician. Investigators found 1,200 videos and 140 images stored on servers in his home, but no evidence that he shared the material with others. He began working at Hopkins 1988 and saw soughly 12,600 patients during 25-year tenure. The attorneys had originally asked for a 35% settlement fee, but a Baltimore judge ruled they would only receive 17.2 25... The settlement closes case that threatened the reputation of Johns Hopkins, one of the world's leading medical institutions. }) \] </ snanswer>
Revised Summary Scores (FincSurE)
Falthfulness Score: 100%
Completeness Scure: 77%
Conciseness Score: : 100%
Table 29: An example of refinement through reflective reasoning (ReFeed). The original sunmary and summary scores are identical in Table 28. Red indicates incorrect feedback or flawed reasoning, while Blue highlights the proper validation of feedback, ensuring that only incorrect feedback is filtered out.
34

---

Published as a conference paper at COLM 2025
User Prompt
Your task is to refine the summary.
Instruction:
1. Revise the summary by considering the relevant dimension:
Faithfulness: Correct factual inconsistencies the summary.
Completeness: Add any missing key information that is essential to the context of the summary.
Revise summary to clearly and efficiently convey key content without unnecessary details.
Conciseness:
Document:
(Document)
Summary:
(Initial Summary)
Table 30: Prompt for training the Direct Refinement Approach. The output uses the revised summary in SumFeed-CoT.
35


Reviews:
[{"review_id": "TFh1tfEKds", "reviewer": "Reviewer_rriG", "summary": "The paper provides a new framework refeed for text/summary refinement, incorporating multiple dimensions of text assessment. The paper provides abalation of variation in the proposed techniques along with a fine-tuning dataset.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and easy to follow. The experiments are sound and all the results are well backed by numbers.\n- Novel approach to incorporate Multi-Dimensional assessment.\n- The approach incorporates reflective reasoning and backtracking during CoT.\n- The experiments and the comparison cover most of the variation that can be thought of in this setting.\n- The final proposed approach is justified with experiments and baselines\n- ReFeed performs better than the existing text refinement framework while improving the generation on multiple dimensions.\n- They do test low- vs. high-quality feedback, but not feedback format variation or dropouts.", "reasons_to_reject": "Important reasons:\n- Potential comparison bias in fine-tuning evaluation (Section 5.6) comparing existing methodologies, DCR, and ACUEval.\n   - The ReFeed model is fine-tuned using its reflective training data (SumFeed-CoT), whereas baselines like DCR and ACUEval are evaluated using this same ReFeed-trained model and not their respective methodologies. This creates a potential bias in ReFeed’s favor and raises questions about the fairness of the comparison.\n\nNot a big concern:\n- ReFeed Requires Teacher Distillation Infrastructure, this is compute intensive\n- No Direct Evaluation on Generalization to Unseen Feedback Styles\n   - Only one dimension’s feedback is available\n   - Different feedback formats are used (e.g., scalar scores or freeform text)\n\nNitpicking:\n- Fine-tuned model comparisons are not done for P1–P3 Baselines", "questions_to_authors": "-  I think there is potential to extend this paper by incorporating more feedback dimensions like style, tense. I think this could have made the paper stronger", "extra_scores": {}}, {"review_id": "YgLDSg5MrD", "reviewer": "Reviewer_iDgk", "summary": "The paper proposes a summarization system that seeks to jointly optimize for faithfulness, completeness, and conciseness.\n\nThe authors started by building a dataset (SummFeed-CoT) that pulls together LLM's reasoning from feedback, and trained a model on this dataset. They described 4 pipelines with varying levels of complexity (P1 to P4) in terms of how they leverage feedback. These 4 pipelines will go on to serve as baselines during subsequent evaluations. The authors then proposed ReFeed - which further builds on the most complex P4 pipeline, by also using reflective reasoning during refinement.\n\nExperiments are conducted on the UniSumEval dataset, and FineSurE (LLM-based evaluator). The experiments are focused on 3 questions the authors posed, that 1/ looks at the overall effectiveness of pipelines P1 to P4 as well as ReFeed along the 3 key dimensions of faithfulness, completeness, and conciseness, 2/ the interplay between these 3 key dimensions and their effect on outcomes, and 3/ the impact due to the quality of the feedback received.\n\nReFeed is showmn to do better than the other baselines in terms of Faithfulness, and an average metric computed out of scores achieved for all 3 dimensions. The experiments also show that ReFeed is fairly robust against the order in which feedback is received.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The dataset that comes with the paper would be interesting, and likely useful to the community.\n2. The proposal builds on up-to-date datasets and evaluation metrics. \n3. The experiments are well-designed to help build a case for the efficacy of the proposed ReFeed system. The authors have put in significant effort to run an extensive set of experiments.", "reasons_to_reject": "1. It is hard to fully reproduce the human evaluations backing Table 1, so it is not possible to fully reason about the relative performance of ReFeed. [Update: The authors reproduction with Claude 2.1 comes close and helps close this gap. It could be useful to be more explicit about this in the paper.]\n2. It is not clear what the motivations behind targeting reflecting reasoning is in this proposal. It would have been more compelling to make an argument of why this approach makes sense, and how it overcomes inherent weaknesses or challenges within current reasoning models.", "questions_to_authors": "Suggestion #1 - Figure 1 shows both the construction of your dataset, as well as ReFeed. I understand editorial space is a premium, but for better readability, it could help to tease these out into two separate diagrams.\n\nSuggestion #2 - While P1 and P4 serves as baseline for your evaluations, I also thought they might be distracting from the proposal you are making. If I understand the paper correctly, you are making a case for the use of better reflective reasoning to produce better summarizers that do well for faithfulness, completeness and conciseness. It will be more direct to explain ReFeed, and then benchmark it against other large reasoning models on your evaluation dataset. I could have missed the point behind it, but it's not clear what P1 and P4 add to the proposal here. [Update: Authors addressed this reasonably well]\n\nSuggestion #3 - There is a lot of good discussion and elaboration in the authors' responses. I know it will not be easy, but hopefully you can incorporate many of the suggestions and additional results if space permits.", "extra_scores": {}}, {"review_id": "qdpREWWOT9", "reviewer": "Reviewer_qsDY", "summary": "The paper proposes a summarization refinement pipeline that distills large reasoning models to perform summarization. The paper first constructs SumFeed-CoT that captures reflective reasoning of the LLMs while giving feedback about summaries along three dimensions ((faithfulness, completeness, conciseness)). Then, a smaller model is trained on this data to perform the reflective reasoning and refine the summary. Automatic metrics show that the proposed approach outperforms the existing summary refiners in terms of quality and robustness.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "New approach for refining summaries. New synthetic dataset to train/refine summarization systems.\n\nThe analysis section is comprehensive and thorough. The section on robustness is especially insightful.", "reasons_to_reject": "Missing baselines:\n\n1) All the baselines in Table 1  are trained on (document, summary, feedback) triples as input and (reasoning, refined summary) as output. Ideally there should be a baseline that is trained directly on the refined summaries in SumFeed-CoT to understand how much value reflective reasoning/feedback is adding.\n2) Ideally, there should also be a couple of baselines that do not perform any distillation - such as using standard zero-shot or few shot prompting techniques.", "questions_to_authors": "Does UniSumEval evaluation dataset  include human-written reference summaries  to facilitate other reference-based automatic metrics?", "extra_scores": {}}, {"review_id": "jjyCKgImUL", "reviewer": "Reviewer_7Nyd", "summary": "This paper proposes ReFeed, a refinement pipeline designed to tackle the challenges of addressing multiple quality dimensions in summarization, specifically faithfulness, completeness, and conciseness. The pipeline performs post-hoc refinement to manage trade-offs across these dimensions, accounting for issues such as ordering bias (sequential vs. simultaneous handling of dimensions) and deficiencies like low accuracy or insufficient robustness.\nThe paper investigates whether improving one quality dimension compromises others, and whether their pipeline can outperform not only pipelines that address a single dimension but also baselines that target multiple dimensions under various settings.\nReFeed uses a newly developed dataset that trains models to enhance summaries through reflective reasoning on feedback. The dataset is constructed in three stages: Goal Specification, Guideline Formulation, and Quality Control. It is used to fine-tune LLaMA-3.1-8B-Instruct with LoRA and DeepSpeed, enabling the model to generate reasoning and refined summaries.\nReFeed is evaluated against four baselines on the UniSumEval dataset, using feedback generated by FineSurE to guide refinement across diverse domains. The refined summaries are assessed for faithfulness, completeness, and conciseness using FineSurE with GPT-4o. A small human evaluation study is also conducted.\nThe results show that ReFeed outperforms the four baselines in terms of average summary quality, demonstrating excellent handling of the three target dimensions. The authors also highlight ReFeed’s robustness to ordering bias and variability in feedback quality.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses important challenges in multi-dimensional summarization, and the proposed pipeline appears to outperform several baselines, particularly in managing multiple quality dimensions with reasonable trade-offs.\n- The dataset developed in the paper could be useful to the research community and may be reused or extended by other researchers.\n- The paper is reasonably well-presented; the experiments consider different baselines, and most design choices regarding the pipeline components and evaluation strategy are well-justified.", "reasons_to_reject": "- While the paper is generally well-structured, the presentation is quite dense, especially in its descriptions of the pipeline and dataset construction. Several important details are only fully explained in the appendix, making it difficult to follow without frequent back-and-forth reading. \n- The evaluation relies heavily on automatic metrics, primarily using FineSurE with GPT-4o. While this is acceptable, I think the paper would benefit from a deeper error analysis and more extensive manual inspection of the refined summaries. \n- While evaluating the end-to-end performance of the pipelines is expected and helpful, the paper would benefit from evaluating intermediate components or substeps, even partially through manual inspection. For example, checking the low- and high-quality feedback or analyzing samples from the <document, summary, feedback> triplets collected during the Initial Feedback Collection could provide deeper insight and help understand the impact of each step/component.", "questions_to_authors": "- Did you control for variables that may affect the quality dimensions? For example, the length of the summary output; I checked the example in the appendices and noticed that the summary produced by ReFeed is around 150 tokens, while the one generated by P4 is only about 105 tokens.\n- What is the rationale behind the specific threshold values used during filtering? For instance, why were thresholds such as faithfulness = 1, completeness ≥ 0.5, and conciseness ≥ 0.5 chosen?\n- Did you attempt to optimize the prompts used in the pipelines, for example, by rephrasing, adding more information, or experimenting with different prompt structures?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper provides a new framework for refining summaries based on reflective reasoning. The authors release a large-scale dataset containing chains of thought reasoning about summaries in terms of faithfulness, completeness and conciseness. They then trained smaller models on this dataset (distillation) to perform reflective reasoning and refine a summary. The experiments investigated several pipeline variants and assessed their effect on qualities such as faithfulness, completeness and conciseness.  

Reviewers agree on a number of strengths, which make this paper a worthwhile contribution to the conference. The paper is well written and focuses on important challenges in refining summaries along multiple dimensions. The proposed approach to refining summaries along these multiple aspects is original and is shown to outperform existing text refinement frameworks and baselines, particularly in trading-off multiple quality dimensions. The experimental analysis is thorough, considering important variations of the method and building on recent datasets and metrics, and makes a strong case for the efficacy of the method. A new dataset is presented in the paper that could be useful for future research.

Some clarifications provided by the authors in the rebuttal phase would be valuable to include in the final version of the paper, if accepted, such as the motivations for targeting reflective reasoning and the shift from passively accepting feedback to reflecting on it. The human evaluation using Claude v2.1, along with the accompanying explanation, will be important for understanding differences in performance metrics to prior work. The results for the additional baseline ‘direct refinement’ provides an interesting point of comparison that further demonstrates the effectiveness of ReFeed. Likewise, the ablation results further support the complete approach and should be added to the camera-ready version. 

Finally, please note some constructive suggestions from reviewers on presentation, which was generally very good, but could be improved in a couple of places, e.g., Figure 1.
Decision: Accept