SummIt: Iterative Text Summarization via ChatGPT
Authors: Haopeng Zhang
Xiao Liu
Jiawei Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: summarization
large language model
text editing
TL;DR: The paper proposes an iterative text summarization framework that enables the model to refine the generated summary iteratively through self-evaluation and feedback.

Abstract: Existing text summarization systems have made significant progress in recent years, but typically generate summaries in a single step. The one-shot summarization setting is sometimes inadequate, however, as the generated summary may contain hallucinations or overlook important details related to the reader's interests. In this paper, we address this limitation by proposing SummIt, an iterative text summarization framework based on large language models like ChatGPT. 
Our framework enables the model to refine the generated summary iteratively through self-evaluation and feedback, closely resembling the iterative process humans undertake when drafting and revising summaries. Furthermore, we explore the potential benefits of integrating knowledge and topic extractors into the framework to enhance summary faithfulness and controllability. We evaluate the performance of our framework on three benchmark summarization datasets through empirical and qualitative analyses. We also conduct a human evaluation to validate the effectiveness of the model's refinements and find a potential issue of over-correction.

SummIt: Iterative Text Summarization via ChatGPT
Haopeng Zhang
Xiao Liu
Jiawei Zhang
IFM Lab, Department of Computer Science, University of California, Davis, CA, USA haopeng, xiao, jiawei@i fmlab..lbbrrr.
Abstract
Summarize the Document:
User
Chelssa's is on loan midferider Ortol Romeu goes up againet Sportamed la Marin Kacwnn as the par pretict the actones the woekend's Premier League and FA Cup matchhs.....
Text summarization systems have made signifi- cant progress in recent years, but typically gen- erate summaries in one single step. However, the one-shot summarization setting is some- times inadequate, as the generated summary may contain ballucinations overlook essen- tial details related to the reader's 's interests. This paper addresses this limitation by proposing Summlt, iterative text summarization frame- work based on large language models like Chat- GPT. Our framework enables the model re- fine the generated summary iteratively through self-evaluation and feedback, resembling hu- mans' iterative process when drafting and re- vising summaries. Furthermore, explore the potential benefits of integrating knowledge and topic extractors into the framework enhance summary faithfulness and controllability. We automatically evaluate the performance of our framework three benchmark summarization datasets. We also conduct a human evaluation validate the effectiveness of the iterative re- finements and identify potential issue of over- correction."
Summary: Cha
loan middieide Oniol Romeu and Sportsmal's is Martin Keown predict the scones for the weekand's i Fremier League and IA Cup manches. between
ChatGPT Summartzer
Mencheeterr United.
Ieeraion
Revision Comments: The su summary is too long and contains urnacessary details. It should be shortened to only inclide main point, which Onai Poomu and Keown predicting the scorss Premier League
Iteratvv Summartzattinn [Run Background)
woekend's
Evalustor
(D.1, matches.
D.1.
Summary: Cheisee's en loan midifalder Oriel Romeu and prodict sccres FA
Spartsmal's
Itertioo
Revision Comments: The inckuden the main paint the document, Ther'ss need further revise. (STP.
Evolaation Soore: (D.8, 0,1,0,0, 0,
Chaisez's 's on loan mi midflelcer On Romou and Sporismal's
prodict soores FA matches.
Figure illustration of the iterative summarization process. The summarizer continuously refines the sum- mary according to self-feedback from the evaluator at cach iteration.
characterized by multiple iterations of drafting and editing (Faltings et al., 2020).
1 Introduction
These end-to-end summarization systems en- counter multiple challenges. Firstly, they fre- quently suffer from the issue of hallucination, re- sulting in the generation of ungrammatical or fac- tually incorrect content (Kryscinski et al., 2020; Zhang et al., 2022b). Secondly, these systems are often optimized using imperfect reference sum- maries, and widely adopted evaluation metrics like ROUGE may not accurately assess summary qual- ity. Thirdly, most of these systems lack controllabil- ity, as they only produce single generic summary conditionally on the input document. In practice, instead of a single condensed version of the entire document, generating summaries that cater to spe- cific aspects or queries would be more beneficial to meet the diverse requirements of users.
Document summarization aims to compress text material while retaining its most salient informa- tion. With the increasing amount of publicly avail- able text data, the significance of automated sum- marization techniques has amplified. Recent ad- vancements in summarization systems, leveraging neural networks and pre-trained language models, have demonstrated notable progress (Cheng and La- pata, 2016; Nallapati al., 2016; Liu and Lapata, 2019; Lewis al., 2019; Zhang al., 2020). The above summarization systems are all built end-to- end and follow a one-shot paradigm that generates summaries in a single step. On the contrary, hu- mans often write text through an evolutionary pro-
We will release our codebase at attps://github com/ hpzhang94/sun 9ummpmnnt/s
The emergence advanced instruction-tume

---

large language models (LLMs), such ChatGPT has presented exciting possibilities for summariza- tion systems by exhibiting strong zero-shot per- formance in various downstream tasks. A recent study by (Goyal et al., 2022) compared GPT-3 with traditional fine-tuning methods and found that de- spite lower ROUGE scores, human annotators pre- ferred the GPT-3 generated summaries. Another comprehensive analysis by (Zhang et al., 2023d) focused on large language models for news sum- marization and revealed that the quality of gener- ated summaries already on par with those cre- ated by humans. Furthermore, Liu et al. (2023) demonstrated the utilization of LLMs like GPT-4 an effective natural language generation eval- uator, showing a higher correlation with humans in the summarization task compared previous reference-based methods.
feedback mechanisms. In addition, we pro- pose incorporate knowledge and topic ex- tractors to further improve the faithfulness and controllability of Summlt.
We conduct experiments on three summariza- tion benchmark datasets, and empirical results from automatic evaluation demonstrate the effectiveness of our proposed framework summary refinement.
A human evaluation is conducted to investi- gate the impact of self-evaluation-guiiee sum- mary refinement. The results revealed a po- tential issue of over-correction: while the large language model effectively refines the summary based on the feedback rationale, it exhibits bias toward its own evaluation cri- teria, rather than aligning closely with human judgment.
The advent of LLMs also introduces new oppor- tunities for summarization beyond the traditional one-shot generation setting. In this paper, we in- troduce Summlt, framework that leverages large language models for iterative text summarization. Instead of generating summaries in single step. our framework enables the model to iteratively re- fine the generated summary through self-evaluation and feedback, resembling the human process of drafting and revising summaries. According to our experiments, the rationale generation and summary refinement in Summit can be guided effectively with in in-context learning, eliminating the need for supervised training reinforcement learning pro- cesses. Additionally, we explore the potential ben- efits of incorporating knowledge and topic extrac- tors to enhance summary faithfulness and control- lability. We instantiate SummIt with ChatGPT as the backbone, and the automatic evaluation results on three benchmark datasets demonstrate the effec- tiveness of Summit in improving summary quality, faithfulness, and controllability within only a few iterations. Furthermore, conduct a human eval- uation validate the iterative refinements quality and identify potential over-correction issue.
Related Work
2.1 Text Summarization
Recent years have witnessed significant advance- ments in text summarization systems with the de- velopment of deep neural networks and pre-trained language models. Automatic summarization meth- ods can be broadly categorized into extractive and abstractive approaches. Extractive summarization involves the direct extraction of sentences from the source text to form summaries (Xu et al., 2019; Liu and Lapata, 2019; Zhong et al., 2020; Zhang al., 2022a, 2023b), while abstractive approaches condi- tionally generate summaries using sequence-to sequence (seq2seq) framework (Lewis et al., 2019; Zhang et al., 2020).
Existing approaches mentioned above generate summaries in one-shot manner, and their out- puts may not always align with user expectations and may contain hallucinated content (Kryscinski et al., 2020). To address the limitation, Liu et al. (2022) proposes to automatically correct factual inconsistencies in generated summaries with gen- erated human feedback. In contrast, our SummIt framework enables iterative summary refinement with self-evaluation and feedback, eliminating the need for costly human annotations. Additionally, we propose the integration of knowledge and topic extractors to further enhance summary faithfulness and controllability.
We summarize the contributions of this paper as follows:
We propose SummIt, a novel framework for iterative text summarization. SummIt enables the itcrative refinement generated sum- maries by incorporating self-evaluation and
Phaps://chat.opeeaiicommmm

---

Refined Summary
Knowledge Extractor
Topic Extractor
You summarizer that follows the output pattern. You revise the summary based on the given instructions. You follow all the instructions without commenting on them.
You summary evaluator that gives scores summarries with revise suggestions.
Refined
Your suggestions can be:
1. Add the information of <insert>
Summary
Remoove the information of <insert>>>
[Revise Suggestions] Revise summary. Follow all the suggestions and you not make more comments, [Format Instructions]
Refine:
Rephrase the information of <innsert>>
Evaluation
4. Shorten the summary
5. Keep the summary unchanged
Rationale
If you think there's further revision is needed, you must "<STOP>" the end.
Summarizer
Evaluator
Source Document
Figure 2: The overall framework of our proposed iterative text summarization system. The evaluator generates evaluation rationale based the current summary, and the summarizer then refines the summary accordingly. The knowledge and topic extractors retrieve information from the source document guide the process.
2.2 Summarization with Large Language Models
big, 2022), and grammar error correction (Awasthi et al., 2019). Recent works also formulate text editing as an interactive task, such as command- based editing systems (Faltings et al., 2020), and interactive editing systems (Schick et al., 2022). Zhang al. (2023a) also proposed benchmark for fine-grained instruction-based editing.
Recent years have seen surge in training large- scale language models (LLM) large amounts of text, such as GPT (Radford et al., 2019; Brown et al., 2020). Several studies have explored the application of LLMs in the context text sum- marization. For instance, Goyal et al. (2022) com- pared the performance of GPT-3-generated sum- maries with traditional fine-tuning methods, find- ing that although the former achieved slightly lower ROUGE scores, human evaluators expressed pref- erence for them. Similarly, Zhang al. (2023d) re- ported that -LM--gnerated summaries were on par with human-writted summaries the news domain. Zhang et al. (2023c) benchmarked the performance of ChatGPT on extractive summarization and pro- poses to improve summary faithfulness with an extract-thenn-genenate pipeline. On the other hand, prior works have also leveraged LLMs for summa- rization evaluation (Liu al., 2023; Fu al., 2023; Luo al., 2023), demonstrating that based metrics outperform all previous evaluation metrics like ROUGE (Lin, 2004) and BertScore (Zhang et al., 2019) by a significant margin in terms of correlation with human evaluations.
Recently, Welleck et al. (2022) introduced a self- corrective Icarning framework that incorporates a corrector into the language model to facilitate self- correction during sequence generation. Akyürek et al. (2023) propose reinforcement learning- based approach generate natural language feed- back for correcting generation errors. Concurrent work Madaan et al. (2023) presents a similar gener- ation pipeline that enhances imitial outputs through iterative feedback using a single LLM for short text generation tasks. In contrast, our SummIt frame- work differs from these approaches as it specifically focuses on the conditional generation task of sum- marization, with an emphasis on improving sum- mary faithfulness and controllability. Additionally, we empirically observe that separating the sum- marizer and evaluator into different LLMs, each employing different in-context guidance leads to improved performance in our framework.
2.3 Text Editing
3 Methods
Our work is also closely related the task of text editing. Traditional editing models are trained to solve specific tasks, such as information updating (Iso et al., 2020), Wikipedia edit (Reid and Neu-
3.1
Iterative Summarization
The overall architecture of our iterative text sum- marization system SummIt shown in Figure 2.

---

The system consists of major components, summarizer that generates and refines the summary, and an evaluator that generates feedback rationale.
Doc #words
Sum
Dataset
#Test
#Sum
#words
XSum
11,334
430.2
23.3
CNN/DM
11,489
766.1
58.2
Summarizer: The summarizer is in charge of generating the initial summary and revising a sum- mary conditioned on the given explanations and source document. We instantiate the summarizer with an instruction-tuned language model S.
NEWTS
600
738.5
70.1
Table I: Detailed statistics of the experimental datasets. Doe # words and Sum # words refer to the average word number in the source document and summary.
Formally, given the input source document x, the initial summary yd generation process can be represented as:
C = U1).,)))...(aaa Bm)), and generates sum- mary y by concatenating the exemplar source- summary pairs and input document as prompt: Parly x, C)
ps(y x) II ps (vl y
x2):x),
(1)
We also use in-context learning to guide our iterative summarization system, where we use "efeennentrrffrrence summary" pairs as the con- text for the summarizer S, and use "document- reference summary-human written explanation" triplets as the context for the evaluator E. We empirically find that in-context learning could im- prove the efficacy of our system.
where yhe denotes the generated tokens, vf refers to the 1-th summary token, and m denotes the sum- mary length.
After obtaining the i-step self-evaluation feed- back e' from the evaluator E, the summarizer will refine the summary accordingly and then generates refined summary (++1) as: (1+() x, ex)
3.3 Summary Faithfulness and Controllability In practical applications, the faithfulness of the generated summary holds significant importance, alongside the overall quality of the summary (Kryscinski et al., 2020). Previous research has demonstrated the effectiveness of leveraging knowl- edge extraction from source documents to enhance the faithfulness of generated summaries (Huang et al., 2020; Zhu et al., 2020). Building upon these insights, we propose integrating a knowledge ex- tractor into our itcrative summarization system.
Evaluator:
The evaluator is another instance of language model E that generates summary quality evaluation and corresponding explanations e' for the i-th iteration as: (e:(e y").
Stopping Criteria: The evaluator gives a qual- ity assessment of the generated summary and then outputs the rationale for the evaluation feed- back. The summarizer receives model evaluation and feedback from the evaluator, subsequently re- fining the summary based on this input.
Knowledge Extractor: In particular, we utilize OpenTE which extracts knowledge the form of triplcts from the source document. During each iteration, the summarizer (S) is guided to refine the summary in accordance with the extracted knowl- edge, represented as: ( (11 x, e k). More- over, the evaluator (E) be directed to factor in faithfulness when delivering feedback, denoted as pa x, y k). as LLMs have shown to be efficient faithfulness evaluators (Luo et al., 2023).
This iterative process can be repeated until 1) the evaluator determines that no further refinement is required or 2) fulfills mlee stopping criteria, such reaching maximum iteration number.
3.2 In-context Learning
Since the summarizer and evaluator in Summit not fine-tuned with supervised data or trained rein- forcement learning rewards, it would be beneficial to guide the explanation and summary generation process with the desired format or template. Re- cent studies have shown that large language mod- els have strong few-shot performance on various downstream tasks, known as in-context learning (ICL) (Brown al., 2020).
Furthermore, real-world applications often re- quire the generation of summaries tailored to spe- cific aspects queries, rather than single generic summary of the entire document. Our iterative summarization framework offers enhanced control- lability for aspect- based summarization tasks.
The standard ICL prompts a language model, M, with a set of exemplar source- source-summen summary pairs,

---

Topic Extractor: Given an aspect-oriented query g, we prompt both summarizer S and evaluator E to initially extract relevant snippets, each contain- ing less than 5 words, from the source document x. Following the extraction, these components then proceed to either generate or assess the summary by taking into account the extracted snippets. The iterative nature of our framework further facilitates the controllable summary generation, allowing for the easy transformation of generic summaries into topic-focused summaries based on the user's pref- erences.
CNN/DailyMail (Hermann al., 2015) is the most widely-adopted summarization dataset that contains news articles and corresponding human- written news highlights as summaries. We use the non-anonymized version in all experiments. 2) XSum (Narayan ct al., 2018) is a one-sentence news summarization dataset with all summaries profes- sionally written by the original authors of the BBC news. 3) NEWTS (Bahrainian et al., 2022) is uspect-focused summarization dataset derived from the CNN/DM dataset and contains two summaries focusing on different topics for the same news.
3.4 Prompt Format
Evaluation metries: For summary quality, we use ROUGE scores (Lin, 2004) and G-Eval (Liu et al., 2023) as the automatic metrics. We re- port ROUGE-I, ROUGE-2, and ROUGE-L scores, which respectively measure the overlap of uni- grams, bigrams, and the longest common sequence between the generated summary and the reference summary. G-Eval an LLM-based matrix with a scale ranging from to 5. G-Eval uses LLM with chain-of-thoughts (CoT) and a form-filligg paradigm to assess the quality of NLG outputs. It shows the highest correlation with humans com- pared to other summarization quality metrics.
We utilize both system prompts and user prompts following the OpenAl API in our system implemen- tations. The full prompts used in the experiments can be found in Table 8 and Table 9. Notably, we empirically find that pre-defining the possible edit operations for the evaluator improves the system performance significantly since it avoids free-form edits the summary by the large language model. Thus, we adopt the five types of text editing opera- tions commonly used in text editing systems (Reid and Neubig, 2022; Faltings et al., 2020). We specif- ically require the evaluator to generate feedback based on the source document and summary at this iteration with the following five types of possible refinement operations:
For summary faithfulness, we use FactCC (Kryscinski et al., 2020) and DAE (Defining are entailment) (Goyal and Durrett, 2020) as our eval- uation metrics. FactCC is weakly supervised BERT-based model metric that verifies factual con- sistency through rule-based transformations ap- plied to source document sentences. It shows high correlation assessing summary faithfulness with human judgments. DAE decomposes entail- ment at the level of dependency ares, examining the semantic relationships within the generated out- put and input. Rather than focusing on aggregate decisions, DAE measures the semantic relationship manifested by individual dependency ares in the generated output supported by the input.
Add: Add the information of <insert>
Remove: Remove the information of <insert> from the summary
Rephrase: Rephrase the information of <in- sert> the summary
Simplify: Shorten the summary
Keep: Keep the summary unchanged
4 Experiments
In this section, we validate our SummIt framework on three benchmark summarization datasets. We employ both automatic metrics and human assess- ment to evaluate the quality 4.2, faithfulness 4.3, and controllability 4.4 of the generated summaries.
For the controllability of query-focused summa- rization, we use BM25 (Robertson et al., 2009) and (ar(karn et al., 2020) to measure the sim- ilarity between the query and the summary with both sparse and dense evaluations. BM25 is a prob- abilistic retrieval function that ranks documents based on query term frequency. DPR leverages dense vector representations for scalable retrieval, embedding both questions and passages into fixed- length vector spaces for nuanced similarity calcula- tions.
4.1 Experiment Settings
We conduct experiments on the follow- ing three publicly available benchmark datasets, as presented in Table 1, ensuring they are con- sistent with previous fine-tuning approaches: 1)
Datasets:

---

CNN/DM
XSum
Model
R1
R2
RL
G-Eval
R1
R2
RL
G-Eval
sero-shot setting
PEGASUS
32.90
13.28
29.38
3.23
19.27
3.00
12.72
3.52
BART zs
32.83
13.30
29.64
3.42
19.26
3.30
14.67
3.49
T528
39.68
17.24
26.28
3.47
19.66
2.91
15,31
3.55
ChatGPT
39.44
16.14
29.83
3.46
21.61
5.98
17.60
3,47
Summlt (ours)
36.50
13.49
26.76
4.33
21.92
5.93
17.62
4.24
Few-shot setting
ChatGPT
40.00
16.39
30.02
3.57
23.96
7.36
19.36
3.57
Summlt (ours)
37.29
13.60
26.87
4.35
22.04
6.20
17.46
4.32
Table 2: This table presents results from experiments conducted on the CNN/DM and XSum datasets under both zero-shot and few-shot settings. A random sample 1, 0.00 data points was taken from each dataset for evaluation. G-Eval represents the score evaluated by the ChatGPT evaluator in our framework.
Model
Coherence
Fluency
Relevance
Consistency
Conciseness
Overall
Human Pref
CNN/DM
BART
3.92
4.16
4.00
3.12
3.64
3.24
0.04
T5
3.72
4.24
4.32
3.52
3.84
3.68
0.10
PEGASUS
3,20
3.53
3.33
2.87
1.85
1.63
0.00
ChatGPT
4.20
4,36
4.28
4.01
3.92
4.01
0.34
Summlt
4.24
4.50
4.29
4.12
3.84
4.09
0.52
XSum
BART
3.97
4.30
4.13
3.30
3.93
3.84
0.30
T5
3.84
4.32
4.02
3.84
3.25
0.08
PEGASUS
3.13
4.10
3.52
2.87
2.03
2.41
0.00
ChatGPT
4.03
4.40
4.30
3.93
3.87
3.92
0.24
Summlt
4.35
4.28
4.05
3.72
3.96
0.38
Table Human study results on generic summary quality. The first five columns include Likert seale ratings and the last column is the human preference results.
In line with previous research findings that have emphasized the inclination of human annotators to- wards summaries generated by LLM models, even in the presence of comparatively lower ROUGE scores (Goyal et al., 2022), we further validate the effectiveness of Summlt through dedicated use 1) five-point Likert scale ratings (Likert, 1932) covering sum- mary coherence, fluency, relevance, consistency, conciseness, and overall evaluation, and 2) human preference test, where annotators are shown sum- maries of the same source document from all five summarization systems and then asked to select their most preferred summary or summaries.
dom samples from CNN/DM and XSum test sets, with seed 101, and the full NEWTS test set. Our prompts were refined with 50-example develop- ment set. The detailed experimental setup is pro- vided in Appendix A.
human study. Specifically,
4.2 Generic Summary Quality Evaluation
The automatic evaluation results for generic sum- marization quality are shown in Table 2. We use previous pre-trained language models, including PEGASUS (Zhang al., 2020), BART (Lewis al., 2019), and T5 (Raffel et al., 2020) as baseline models. We compare our framework Summit with these baseline models under a zero-shot setting for a fair comparison.
We evaluated the performance using 1000 ran-

---

It is observed that Summlt has inferior ROUGE scores compared to fine-tuning approaches on CN- N/DM, while exhibiting significantly higher LLM- based evaluation metric G-Eval. On the other hand, it outperforms all baseline methods on the XSum dataset. Compared to the output of ChatGPT, the summaries of ChatGPT after our iterative refine- ment see consistent improvement in the G-Eval score. The results consistent with the previous conclusions in (Zhang et al., 2023d), where large language model summary outputs receive lower ROUGE scores due to the low quality of reference summaries.
R1
R2
RL.
G-Eval
FacCCC
DAE
ChaIGPT
21.61
17.60
Summlr
21,92
17.62
4.24
36.00
33.02
ChaGGPT-IR
22.01
5.11
17.06
3.85
51.68
93.68
Summli-E
19.72
3.85
15.36
4,95
47.24
90.36
Table 4: Experimental results of incorporating knowl- edge extractor on summary quality and faithfulness on XSum dataset. -IE refers to the model integrated with OpenIE.
R1
R2
RT.
G-Eval
BM25
DPR
ChaiGIT
30.01
8.94
27.03
1.06
33.09
77.22
CharGPT Topic
1.16
36.20
Summlt Topic
Table 5: Experimental results on NEWTS dataset to test the controllability of our framework. -Topic indicates model that is prompted to extract topic-related snippets before generating a summary.
In addition to the zero-shot setting, we investi- gate the effects of in-context learning for Summlt, shown in the lower block of Table 2. The re- sults consistently demonstrate that incorporating in- context learning significantly enhances the model's performance on ROUGE and G-Eval scores. This observation underscores the substantial few-shot capabilities of Summlt, showcasing its ability to adapt effectively and generate high-quality sum- maries in contexts with very few examples.
4.3 Summary Faithfulness Evaluation
To evaluate the efficacy of the SummIt frame- work in enhancing summary faithfulness with the knowledge extractor, we conducted additional ex- periments, as presented in Table 4. The find- ings demonstrate that our framework's iterative ap- proach to refining summaries yields significant im- provements in summary faithfulness, as indicated by both FactCC and DAE results. Furthermore, the integration of knowledge extractor such OpenIE further enhances the level of faithfulness. The LLM-based evaluation score G-Eval also indi- cates a higher level of satisfaction with the refined summaries when guided by the extracted knowl- edge triplets. In conclusion, our study reveals that iterative refinements with the incorporation of the knowledge extractor effectively enhance summary faithfulness without compromising the quality of the summaries.
To further verify the summary quality, we con- duct human study evaluate the overall quality of the summarics as shown in Table 3. According to the five-point Likert scale ratings, the summaries of ChatGPT and Summlt consistently outperform pre-trained language model results. The iterative refinement Summlt also provides consistent im- provements, which align with the G-Eval results obtained from the automatic evaluation. We also conducted a human preference study, where sum- maries from all models were presented to human annotators. They were tasked to select the best sum- mary, without any prior knowledge of the origin of each summary. Consistent with the findings in (Goyal et al.. 2022), the results reveal a clear pref- erence among human annotators for summaries generated by large language models (LLMs) for both CNN (86%) and BBC (62%) style summaries. We also notice the summaries of ChatGPT after our iterative refinement (SummIt) show signifi- cant improvement in human preference, with 18% and 14% percent improvements on CNN/DM and XSum datasets. The results demonstrate the ef- fectiveness of refining generic summaries of our framework.
4.4 Query-focued Summarization Controlability Evaluation
We utilize the query-based summarization dataset NEWTS as our testbed to demonstrate the control- lability ability of Summlt. The results obtained, as depicted in Table 5, highlight the framework' 's capability to align the focus of generic summary with the specific topic of interest or query provided by the user. We also observe improved G-Eval eval- uation scores by directing the summary generation process toward the intended topic.
Furthermore, we evaluate the controllability of the summarization systems by quantifying the sim-

---

R1
R2
RL
G-Eval
100m
Summlt
36.50
13.49
26.76
4.33
-w/o Add
33.01
11.55
24.71
3.98
-w/o Remove
36.46
13.44
26.55
3.64
0.02
-w/o Rephrase
34.71
12.12
26.31
3.82
A23
-W/ Simplify
33.49
12.33
25.76
3.55
-w/o Keep
33.87
13.03
25.70
3.94
Ieeation
Table 6: Ablation Study Iterative Refinement Opera- tions
0.95
CHNON
0.96
6.96
4.96
Summary-Iter Hospitals in Wales may have to choose between emergency care and non-urgent surgery during peak winter months, according to Dr. Andrew Goodall. He suggested that hospitals may de- cide not to schedule surgery in order to focus on "front door pressures." Two hospitals in Swansea and Brid- gend have already cancelled some surgical procedures until after Christmas.
0.95
0.94
0.94
2.02
p.92
2
5
Eeracion
Figure 3: Human evaluation to justify the refinement behavior of Summlt. The top plot refers the human justification of the ratio that the summary is improved each iteration and the bottom plot indicates the ratio that the summarizer follows the evaluator's ' evaluation rationale.
The summary effectively conveys the main point of the article, but it could be shortened for conciseness. Consider removing the specific hos- sentence about
Evaluation-Iterl:
pitals mentioned and rephrasing hospitals having choose between emergency and non-urgent surgery make more concise.
Summary Iter2: Hospitals Wales may have to pri- oritize emergency care over non-urgent surgery during peak winter months, according to Dr. Andrew Goodall. Some surgical procedures have already been cancelled until after Christmas.
ilarity between the query and the generated sum- mary. Both BM25 and DPR are employed as simi- larity metrics, and we consistently observe enhance- ments after the iterative refinement process. This observation serves as evidence that SummIt effec- tively refines the summary to align with the topics specified in the query.
iterations...
An example iterative summary refinement from the XSum dataset. The revision between the iterations and their corresponding comments are pre- sented same color. The blue color refers to the rephrase revision and the orange color refers the move operation.
Table 7:
5 Analysis
5.1
Ablation Studies
Table 6 shows the results of the ablation study by re- moving refinement operations. The ablation study is conducted on the CNN/DM dataset under the zero-shot settings. According to the results, each option contributes to the success of our method, and the add operation affects the ROUGE score most, while the simplify operation affects the GPT- evaluation scores the most. Without adding the operation, the information in the iterative process will only decrease, resulting in less n-gram overlap. On the other hand, without the simplify and re- move operations, the redundant information results in low G-Eval scores.
1) Does the refinement actually improve the sum- mary?
Another potential issue of Summit would be:
2) Does the refinement follow the ratio- nale feedback from the evaluator?
To address these concerns and provide fur- ther validation for the step-wise summary refine- ment in SummIt, we conducted the corresponding human evaluations. Specifically. we asked expert human annotators to label 1) whether these edits resulted in improvements to the summary based on human judgment and 2) whether the edits made by the summarizer align with the feedback provided in the last step by the evaluator.
5.2 Over-correction Issue
A recent work (Liu et al., 2023) highlights the po- tential issue LLM- evaluators having a bias towards the LLM outputs, which raises the doubt:
The results of the human evaluation, presented in Figure 3, indicate that approximately 90% the edits performed by the summarizer adhered to

---

the provided feedback as intended on both datasets, However, only around 50 60% of these edits af- ter or more iterations were deemed beneficial according to human judgment, whereas the eval- uator Summlt still asks to perform the refine- ments. We also notice a clear trend that the per- centage of beneficial refinements decreases as the iteration number gocs up. The finding shows an Over-correction problem: the LLM may demand itself to continuously refine the summary based on its own evaluation criteria, rather than adhering to the true evaluation criteria of good summaries by humans.
Limitations
Instead of conducting experiments on the entire test set, we randomly sample 1000 examples from each dataset test set due to budget limits. Previous research efforts (Goyal et al., 2022; Zhang et al., 2023d) have also been limited in their testing of GPT-3 on a small number of instances.
We only use gpt-3.5-ture model from openAl API as an instance of large language models. The focus of the paper is to explore the iterative sum- marization framework with LLM, but not compare different open and closed LLMs.
This finding highlights the need for better stop- ping criteria in developing iterative summarization systems, and we argue that incorporating human- in- toothe-lop may be potential solution. We leave this for future work.
Acknowledgement
This work is partially supported by NSF through grants 1IS-1763365 and IIS-2106972. We express gratitude to the anonymous viewers for their valuable reviews and feedback.
5.3 Case Study
References
We show an example of iterative summary refine- ment in Table 7. The evaluator provides a detailed rationale for the summary and the summarizer can refine the summary accordingly. The full example can be found in Appendix C.
Afra Feyza Akyürrek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. 2023. RI4f: Generating natural language feedback with reinforcement learning for repairing model outputs. arXiv preprint ar 2305v:2305.088
Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019. Par- allel iterative edit models for local sequence trans- duction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP--JCNN) pages 2260-4270, Hong Kong. China. Association for Com- putational Linguistics.
6 Conclusion
In this paper, we propose new framework for text summarization by iteratively refining sum- maries with model feedback. Our framework is fully built upon large language models and docsn't require supervised training or reinforcement learn- ing alignment. We also demonstrate that the sys- tem improves faithfulness and controllability by incorporating knowledge and topic extractors. We conduct extensive experiments and analyses three benchmark datasets and experimental results show that our iterative summarization system out- performs the one-shot generation setting systems with LLM, which demonstrates the effectiveness of our method. Our human evaluation finds that the summary refinement by our framework can clearly follow the self-evaluation feedback, but is highly biased toward its own evaluation criteria, rather than human judgment. We believe the potential issue could be addressed with human-in-the-loon feedback. We hope the insights gained from this work can guide future research in building more powerful LLM LLM--ased summarization systems.
Seyed Ali Bahrainian, Sheridan Feucht, and Carsten Eickhoff. 2022. Newts: A corpus for news topic-focused summarization. arXiv preprint arXiv:2205. 15661.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Jianpeng Cheng and Mirella Lapata. 2016. Neural sum- marization hy extracting sentences and words. arXiv preprint arxiv:1603.07252
Felix Faltings, Michel Galley, Gerold Hintz, Chris Brockett, Chris Quirk, Jianfeng Gao, and Bill Dolan. 2020. Text editing by command. In North Ameri- can Chapter of the Association for Computational Linguistics.

---

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire.
Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron L Halfaker, Dragomir R. Radev, and Ahmed Hassan Awadallah. 2022. On improving summarization fac- tual consistency from natural language feedback. ArXiv, abs/2212.099688.
Tanya Goyal and Greg Durrett. 2020. Evaluating factu- ality in generation with dependency-level entailment. In Findings of the Association for Computatioonl Linguistics: EMNLP 2020.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. arXiv preprint wrXiv:2303.15021
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the cra of gpt-3, arXiv preprint arXiv:2209. 12356.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651.
Luyang Huang. Lingfei Wu, and Lu Wang. 2020. Knowledge graph-augmented abstractive summa- rization with semantic-driven cloze reward. arXiv preprint arXiv:2055.00 1599.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequeence-o--sequene mns and beyond. arXiv preprint arXiv:1602.06023.
Hayate Iso, Chao Qiao, and Hang Li. 2020. Fact-based Text Editing. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 171-182, Online. Association for Computational Linguistics.
Shashi Narayan, Shay B Cohen, and Mirella Lap- ata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural net- works for extreme summarization. arXiv preprint 1rXiv:1808..08
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-553
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332 9346, Online. Association for Computa- tional Linguistics.
Machel Reid and Graham Neubig. 2022. Learn- ing to model editing processes. arXiv preprint arXiv:2205.12374
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- pre-training for natural language generation, translation, and comprehension. arXiv preprini arXiv: 13461.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- trieval, 33):333--38
noising
Timo Schick, Jane Dwivedi-YY Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebas- tian Riedel. 2022. Peer: A collaborative language model. ArXiv, abs/2208.
Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology.
Chin-Yew Lin. 2004. Rouge:
A package for automatic
evaluation of summaries. In Text summarization branches out, pages 74-81.
Sean Welleck, Ximing Lu, Peter West, Faeze Brah- man, Tianiiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.0003:
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nlg evaluation using gpt-4 with better human align- ment. arXiv preprint ar03v:2303. 16634.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Discourse-aware neural extractive model for text summarization. arXiv preprint 191iv: 14142.
Yang Liu and Mirella Lapata. 2019. Text summa- rization with pretrained encoders. arXiv preprint arXiv:7908. 0tfi::908,083440 08345.

---

Haopeng Zhang, Hayate Iso, Sairam Gurajada, and Nikita Bhutani. 2023a. Xatu: A fine-graing greigea instruction-baset benchmark for explainable text up- dates. arXiv preprint ar0vv:2309. 1063.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2022a. HEGEL: Hypergraph transformer for long document summarization. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pages 10167-10176, Abu Dhabi, United Arab Emirates, Association for Computational Lin- guistics.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang, 2023b. DiffuSum: Generation enhanced extractive summa- rization with diffusion. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pages 13089-13100, Toronto, Canada. Association for Computational Linguistics.
Haopeng Zhang. Xiao Liu, and Jiawei Zhang. 2023c. Extractive summarization chatgpt for faithful summary generation. arXiv preprint arXiv:2304.04193.
Haopeng Zhang, Semih Yavuz, Wojciech Kryscinski, Kazuma Hashimoto, and Yingbo Zhou. 2022b. proving the faithfulness of abstractive summarization via entity coverage control. In Findings of the Asso- ciation for Computational Linguistics: NAACL 2022, pages 528-535, Seattle, United States, Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Ahmad Saleh, Peter Liu. 2020. Pegasus: Pretraining with ex- tracted gap- sentences for abstractive summarization by sequeenc-to--eqquence models.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint 17X1v:190.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang. Kathleen McKeown, and Tatsunori Hashimoto. 2023d. Benchmarking large language models for summarization. arXiv preprint arXiv:2301.1384.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang. Xipeng Qiu, and Xuanjing Huang. 2020. Extrac- tive summarization text matching, arXiv preprint arXiv:2004.08795.
Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang. and Meng Jiang. 2020. Enhancing factual consis- tency of abstractive summarization. arXiv preprint arXiv:2003.08612.

---

Appendix
A Experimental Setup
We use the official checkpoints of the baseline models BART, T5, and PEGASUS from Hugginglace. We use gpt-3.5-turbo model* as the backbone LLM for both the generation and evaluation of summaries, keeping the temperature parameter at 0 to ensure reproducibility.
As for the datasets, randomly sample 1000 samples with random seed 101 from the test set for both CNN/DM and XSum datasets and the full test set for the NEWTS dataset. We also tune the LLM optimal prompt and hyperparameters on dev set of 50 examples. Each discovery experiment was run three times, and the average result was used to mitigate the instability small datasets.
B Prompts
Here list prompts used in experiments for extracted and generated summaries in Table 8 and Table 9. Note that according to OpenAl's document, the model could receive categories of prompts: system prompt and user prompt, where the system prompt functions as the global instruction initialize the model and the user prompt as the question proposed by users. our experiment, we leverage both prompts to guide the model and select the best prompts on a dev set of 50 examples.
Model
System Prompt
Summarizer
You are summarizer that follows the output pattern. You revise summary based on given instructions. follow the instructions without commenting on them. Make sure summary concise and accurate.
Evaluator
You summary evaluator that follows the output pattern. You give for the summaries well revise suggestions. Your should be corresponding to your suggestions. You suggestions can be:
1. Add the information of []
Remove the information of
Rephrase the information of
4. Shorten the summary.
Do nothing.
Only ask for the information that appeared in the document, If you find the summary is too long, ask for shorter summary. Keep the summary short and concise. If you think there's further revision is needed, you must add "<STOP>" the end of your output at the end of the comment. Give precise and clear suggestions.
Table 8: System prompts of the summarizer and the evaluator for all settings.
C Example summaries
Here we show the full example of the iterative summarization and rationale feedback in Table 10 from CNN/DM dataset, together with their golden references.

---

Setting
Model
User Prompt
Summarizer
Summarize:
[In-context Examples] Please summarize the following document.
Quality
[Document Content] [Format Instructions]
Refine: [Revise Suggestions] Revise the summary. Follow all the suggestions and you can not make more comments. [Format Instructions]
Evaluator
Evaluate: [In-context Examples] Pleasse evaluate the summary for the docu- ment. Document Content] [Summary Content]. The output should be prohahility distribution of assigning the between 1-5 well as its justification. Please give revise comments if you think this summary is good enough. [Formar Instructions]
Summarizer
Summarize:
[In-context Examples] Please summarize the following document based on the given lopic sentence. [Docunent Content] [Topic Sentence) [Formut Instructions]
Control
Refine: [Revise Suggestionsl Revise the summary. Follow all the suggestions and you can not make more comments. [Format Instructions]
Evaluator
Evaluate: [In-context Examples] Please evaluate the summary the document check if the summary follows the given topic sentence. DDocument sentence.[Deeument Contenr]
[Summary Content] [Topic Sentence] T The output should he prohability dis- tribution assigning the score between 1-5 well as justification. Please give revise comments if you think this summary is not good enough. Instructions]
Summarizer
Summarize:
[In-context Examples) Please summarize the following document based the given relationships. [Document Content] [OpenIE Relationships) [Format Instructions]
Faithfulness
Refine: [Revise Suggestions] Revise the summary. Follow all the suggestions and can not make more comments. [Format Instructions]
Evaluator
Evaluate: [Dn-context Examples] Please evaluate the summary
document
to check if the summary follows the given relationships. DDocument (Deecumn Contenr] [Summarry Content] [OpenIE Relationships]. The output should a probability distribution of assigning the score between as well as its justification. Please give revise comments if you think this summary is not good enough.[Forme Instructions]
Table 9: User prompts of summarizer and evaluator for different settings.

---

Document
Media playback unsupported on your d december 2014 last updated at 08:58 gmtdr andrew goodall said a choice may have to be made between emergency care and non-urgent surgery .in an extended interview with bbe wales health correspondent owain clarke said hospitals might make the decision not to schedule surgery the first place to focus on " front door pressures during the peak winter months ,wwo hospitals, morriston in swansca and the princess of wales bridgend, have already cancelled some surgical procedures until after christmas.
gondall
Reference
The new chief executive of the welsh nhs hospitals to cancel operations.
bbc wales winter pressures could force
Summary-Iter
Hospitals in Wales may have choose between emergency and non-urgent surgery during peak winter months, according to Andrew Goodall. He suggested that hospitals may decide not to schedule surgery in order focus front door pressures.Tww hospitals in Swansea and Bridgend have already cancelled some surgical procedures until after Christmas.
Evaluation-Ieer
The summary effectively conveys the main point of the article, but it could be shortened for conciseness. Consider removing the specitic hospitals mentioned and rephrasing the sentence about hospitals having to choose between emergency and non-urgent surgery to make more concise.
Summary-Iter2
Hospitals in Wales may have to prioritize emergency care over non-urgent surgery during peak winter months, according to Dr. Andrew Goodall. Some surgical procedures have already been cancelled until after Christmas.
Evaluation-Iter2
The revised summary effectively conveys the main point of the article a concise manner. However, it could benefit from adding more specific information about the reason for the cancellations and the potential impact on patients. Consider adding information about the reason for the cancellations and the potential impact on patients to make the summary more informative.
Summary-
Andrew Goodall warns that hospitals Wales may have to prioritize emergency care over non-urgent surgery during peak winter months, which has already to some surgical procedures heing cancelled until after Christmas. This decision is made to focus on front door pressuresänd ensure that emergency care is not compromised. The cancellations may cause inconvenience to patients, but is necessary to ensure that emergency care is not overwhelmed.
Evaluation-Iter:
The revised summary effectively conveys main point of the article a concise and informative manner. includes specific information about reason for the cancellations and potential impact on patients, as well as the importance of prioritizing emergency care. The only suggestion would consider shortening the summary slightly for even greater conciseness.
Summary-Ire Itter4
Hospitals Wales may have prioritize emergency peak winter months, leading to some cancellations until after Christmas. Dr. Andrew Goodall emphasizes the importance of focusing front door pressuresio ensure that emergency not compromised. While the cancellations may cause inconvenience patients, is necessary to prevent emergency care from being overwhelmed.
over non-urgent surgery during
Evaluation- Iter4
The revised summary effectively conveys the main point of the article in concise and informative manner. It includes specific information about the reason for the cancellations and the potential impact patients, well the importance prioritizing emergency care. The summary is well-written written and does not require any further revision.
Table 10: Case study


Reviews:
[{"review_id": "qLL0yJJSB7", "reviewer": "Reviewer_dwUH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review of our paper and the effort you've put into providing valuable feedback. We have carefully considered your concern, and we would like to address comprehensively:\n\n**Question: You used 5 refinement operations, but we don't know the effects of each operation. More analysis for this is necessary. For example, if removing \"Keep the summary unchanged\", would the results be different?**\n\n**Answer:** Thank you for bringing up this point. We agree that including an ablation study involving removing options would more effectively showcase the performance of our system. The table below shows the ablation study results. We will also include it in the final version of the paper.\n\n\n**Table Refinement Operations Ablation Analysis**\n\n|              | R1    | R2    | RL    | G-Eval |\n| ------------ | ----- | ----- | ----- | ------ |\n| SummIt       | 36.50 | 13.49 | 26.76 | 4.33   |\n| w/o Add      | 33.01 | 11.55 | 24.71 | 3.98   |\n| w/o Remove   | 36.46 | 13.44 | 26.55 | 3.64   |\n| w/o Rephrase | 34.71 | 12.12 | 26.31 | 3.82   |\n| w/o Simplify | 33.49 | 12.33 | 25.76 | 3.55   |\n| w/o Keep     | 33.87 | 13.03 | 25.70 | 3.94   |\n\nThe ablation study is conducted on the CNN/DM dataset under the zero-shot settings. According to the results, each option contributes to the success of our method, and the add operation affects the ROUGE score most, while the simplify operation affects the GPT-evaluation scores the most. Without adding the operation, the information in the iterative process will only decrease, resulting in less n-gram overlap. On the other hand, without the simplify and remove operations, redundant information, resulting in low G-Eval scores.\n\nWe hope the results could address your concern about our paper. Thanks again for the helpful feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow.\n2. They conduct experiments on three summarization benchmark datasets. Although their proposed approach is not complicated, it exceeds SOTA models in some aspects.\n3. This approach can be further developed and improved for text summarizations.", "reasons_to_reject": "you used 5 refinement operations, but we don't know the effects of each operation. More analysis for this are necessary. For example, if removing \"Keep the summary unchanged\", would the results be different?", "questions_to_authors": "", "paper_topic_and_main_contributions": "They introduce \"SummIt,\" an iterative text summarization framework leveraging large language models, such as ChatGPT. Their approach allows the model to continuously refine its summaries by employing self-assessment and feedback, mirroring the human method of drafting and revising summaries. Additionally, they investigate the advantages of incorporating knowledge and topic extractors into their framework.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LaRBEpW5eI", "reviewer": "Reviewer_Ceeh", "summary": "", "questions": "A. For the L235, have you considered using y^i as input since \"Rephrase: Rephrase the information of <insert> in the summary\" is rephrasing the corresponding content in the summary?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the meticulous review of our paper and the dedicated effort you've invested in providing invaluable feedback. We have conscientiously examined each of your concerns and would like to offer comprehensive responses to address them:\n\n**Question: The model has a lower ROUGE score compared to other models. And the model only compares to the model with zero-shot and few-shot learning. The performance is much lower compared to the fine-tuning model on the tasks. I am wondering what's the upper bound performance of the model if it can be fine-tuned.**\n\n**Answer:** We agree the zero-shot and few-shot results are much lower compared to the fine-tuned models like PEGASUS and BART. This discrepancy is probably attributed to the fact that training with similar data during fine-tuning may hint the model about the style, length, and coverage of the summaries. Unfortunately, our backbone is ChatGPT, which is not open-sourced, so we can't directly fine-tune the model. We will explore our method in fine-tuning open LLMs like LLaMa in the future.\n\n**Question: The experiments are done on limited data points. Not sure if the method can adapt to more complex scenarios other than CNN and XSum.**\n\n**Answer:** Thanks for pointing this out. CNN and XSum are both news domain summarization benchmarks and we agree testing our framework on other domains and longer documents would better validate our approach.\n\nBelow is the Pubmed dataset (209.5 words/summary) results of ChatGPT and our Summit framework.\n\n**PubMed Experimental results**\n\n| Setting   | R1    | R2    | RL    | G-Eval |\n| --------- | ----- | ----- | ----- | ------ |\n| ChatGPT   | 31.92 | 12.01 | 26.44 | 3.18   |\n| Summit(Zero-shot) | 26.56 | 10.32 | 21.07 | 3.39   |\n| Summit(Few-shot) | 27.33 | 10.01 | 22.56 | 3.58   |\n\nWe notice similar trends as in the news domain, that our Summit consistently improves the GPT-evaluation scores while showing lower ROUGE scores compared to ChatGPT. On the other hand, few-shot examples benefit the model in terms of both metrics on long summarization generation on the PubMed dataset.\n\n**Question: For the L235, have you considered using y^i as input since \"Rephrase: Rephrase the information of <insert> in the summary\" is rephrasing the corresponding content in the summary?**\n\n**Answer:** Thanks for bringing up this interesting idea. We tried this option in our experiment and empirically found the performance is worse than our current setting. Using y^i in generating the next round y^{i+1} will confuse ChatGPT and make the output more similar. We will include this analysis in our final version.\n\n**Question: G-Eval, FactCC, and DAE are not very common metrics for summarization. Can you give some explanations in the appendix?**\n\n**Answer:** Thanks for pointing this out. We briefly introduce these metrics in the Evaluation metrics in Section 4.1 due to space limit. We will add more explanations and details in the appendix in our final version. \n\nSpecifically, G-Eval is a very recent summary quality evaluation matrix based on large language models. G-Eval uses LLM with chain-of-thoughts (CoT) and a form-filling paradigm to assess the quality of NLG outputs. It shows the highest correlation with humans compared to other summarization quality metrics.\n\nFactCC is a weakly-supervised BERT-based model metric that verifies factual consistency through rule-based transformations applied to source document sentences. It shows a high correlation in assessing summary faithfulness with human judgments.\n\nDAE (Defining arc entailment) decomposes entailment at the level of dependency arcs, examining the semantic relationships within the generated output and input. Rather than focusing on aggregate decisions, DAE measures the\nsemantic relationship manifested by individual dependency arcs in the generated output is\nsupported by the input.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and thorough experiments are made with several meaningful metrics, not just ROUGE score. The proposed model exceeds SOTA models in several aspects.\n2. The direction is promising and can be used in text summarization.", "reasons_to_reject": "1. The model has a lower ROUGE score compared to other models. And the model only compares to the model with zero-shot and few-shot learning. The performance is much lower compared to the fine-tuning model on the tasks. I am wondering what's the upper bound performance of the model if it can be fine-tuned.\n2. The experiments are done on limited data points. Not sure if the method can adapt to more complex scenarios other than CNN and XSum.", "questions_to_authors": "A. For the L235, have you considered using y^i as input since \"Rephrase: Rephrase the information of <insert> in the summary\" is rephrasing the corresponding content in the summary?", "paper_topic_and_main_contributions": "The paper proposes a new model SummIt which iteratively refines its summarization by the prompt generated by its evaluator. And the paper shows that knowledge and topic extractors improve faithfulness and controllability. SummIt exceeds SOTA models in some metrics.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. G-Eval, FactCC, and DAE are not very common metrics for summarization. Can you give some explanations in the appendix"}, "extra_text": {}}, {"review_id": "vqzVZLlMiM", "reviewer": "Reviewer_h2TX", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely value the priceless evaluation of our paper and the devoted time you've dedicated to offering valuable feedback. We have thoroughly assessed each of your points and are eager to provide thorough responses that address them comprehensively:\n\n**Question: knowledge extractor part shows mixed results in the faithfulness automatic evaluation**\n\n**Answer** We agree with this point. Our stance is that there might exist certain **trade-offs** between the faithfulness of the summary and the overall quality of the summary. During the iterative refinements, the LM may sacrifice some faithfulness to improve the overall summary quality. However, compared to the case without a knowledge extractor, we did notice substantial faithfulness performance gaps, indicating the advantages of guiding summary generation with extracted knowledge. Overall, SummIt with the knowledge extractors could generate high-quality and faithful summaries.\n\n**Question: important details of human evaluations left unknown: were there multiple passes, what is inter-annotator agreement on those**\n\n**Answer:** Thanks for pointing this out. We will definitely add more human evaluation details in the appendix in our final version. We conducted the human evaluation in a single pass with multiple annotators, and below is a table summarizing the inter-annotator agreement in terms of different evaluation aspects during our annotation process. We used **Pearson Correlation** to evaluate the inter-annotator agreement\n\n**Inter-annotator agreement**\n| Aspect         |  correlation    |\n| --------------- | ----- |\n| **Coherence**   | 0.658 |\n| **Fluency**     | 0.851 |\n| **Relevance**   | 0.796 |\n| **Consistency** | 0.621 |\n| **Conciseness** | 0.747 |\n| **Overall**     | 0.742 |\n\nWe will release our annotations to better improve reproducibility.\n\n\n**Question: Given ChatGPT's inherent stochasticity, it would be essential to have public access to the summaries for reproducibility, which I didn't see offered in this work**\n\n**Answer:** We totally agree with the point. We are committed to reproducibility and will definitely open-source all the source codes, generation outputs, and human studies upon acceptance. We included our detailed experimental Setup, prompts, and example outputs in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* a simple and efficient technique SummIt for iterative summarization improvement with an LLM is presented\n* SummIt attains strong humans' preference in the corresponding evaluation round, performs competitively on automatic metrics", "reasons_to_reject": "* knowledge extractor part shows mixed results in the faithfulness automatic evaluation", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes SummIt, an LLM (specifically, ChatGPT) based summarization framework with iterative summary generation as opposed to single-pass adopted in previous work. For the subsequent pass, the authors employ LLM self-evaluation and feedback for the model to improve the output.\nThe feedback is limited to certain edit actions, and the key document content to edit is provided by a separate knowledge extraction model.\n\nThe authors compare SummIt to a series of LM and LLM-based approaches working in a traditional 1-pass setting in terms of human evaluation along several qualitative dimensions. On CNN/DM, in SummIt elicits the best performance on most metrics (including Overall); on XSum, ChatGPT is more fluent, but Overall, SummIt outperforms it as well. Human preference is given to SummIt on both datasets.\n\nIn further evaluation of the factuality of the model equipped with the knowledge extractor (G-Eval, FactCC, DAE metrics), ChatGPT without iterative refinements performed better in terms of FactCC and DAE, SummIt being better on G-Eval. In the controllability evaluation of query-focused summarization on NEWTS dataset, SummIt-Topic (the model with additional topic-related content in the prompt), outperforms 1-pass ChatGPT on G-Eval, BM25, and DPR metrics.\n\nFinally, a separate human evaluation shows that the summaries improve with subsequent iterations, and the summarizer follows the evaluator's rationale.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces SummIt, which iteratively refines its outputs using LLM self-evaluation and feedback. The paper is well-written with thorough experiments, the approach performs competitively on automatic metrics and also receives positive feedback during human evaluation. While the approach shows promise, reviewers highlighted some concerns about the model's adaptability beyond CNN and XSum, its comparison to the fine-tuning approach, and mixed results on the faithfulness evaluation.
Decision: Accept-Findings