Extractive Summarization via ChatGPT for Faithful Summary Generation
Authors: Haopeng Zhang
Xiao Liu
Jiawei Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: summarization
large language model
faithfulness
TL;DR: This paper explores ChatGPT for extractive summarization and propose an extract-then-generate pipeline for faithful summary generation.

Abstract: Extractive summarization is a crucial task in natural language processing that aims to condense long documents into shorter versions by directly extracting sentences. The recent introduction of large language models has attracted significant interest in the NLP community due to its remarkable performance on a wide range of downstream tasks. This paper first presents a thorough evaluation of ChatGPT's performance on extractive summarization and compares it with traditional fine-tuning methods on various benchmark datasets. Our experimental analysis reveals that ChatGPT exhibits inferior extractive summarization performance in terms of ROUGE scores compared to existing supervised systems, while achieving higher performance based on LLM-based evaluation metrics. In addition, we explore the effectiveness of in-context learning and chain-of-thought reasoning for enhancing its performance. Furthermore, we find that applying an extract-then-generate pipeline with ChatGPT yields significant performance improvements over abstractive baselines in terms of summary faithfulness. These observations highlight potential directions for enhancing ChatGPT's capabilities in faithful summarization using two-stage approaches.

Extractive Summarization via ChatGPT for Faithful Summary Generation
Haopeng Zhang
Xiao Liu
Jiawei Zhang
IFM Lab, Department of Computer Science, University of California, Davis, CA, USA haopeng, , flawei@ifmlab.oo
Abstract
extractive summarization directly selects sentences from the source document form the summary, re- sulting in summaries that are grammatically correct and faithful to the original text.
Extractive summarization is a crucial task in natural language processing that aims to con- dense long documents into shorter versions by directly extracting sentences. The recent intro- duction of large language models has attracted significant interest the NI.P community due wide range
The growing interest in applying advanced large language models (LLM) such as ChatGPT for text summarization tasks has sparked significant attention. A recent study by (Goyal et al., 2022) compared GPT-3 with traditional fine-tuning meth- ods and found that, despite lower Rouge scores, hu- man annotators preferred the GPT-3 generated text. Another study by (Zhang et al., 2023d) conducted a comprehensive analysis of large language models for news summarization and found that the gener- ated summaries were comparable to those produced by humans. However, existing rescarch (Yang al., 2023; Luo et al., 2023) has only focused on abstrac- tive summary approaches, and the performance of ChatGPT for extractive summarization remains an open question. Morcover, the hallucination prob- lem has dramatically hindered the practical use of abstractive summarization systems, highlighting the need to explore extractive summarization with LLMs for faithful summaries.
its remarkable performance of downstream tasks. This paper first presents thorough evaluation of ChatGPT's performance extractive summarization and compares it with traditional fine-tuning methods on various benchmark datasets. Our experimental analy- sis reveals that ChatGPT exhibits inferior ex- tractive summarization performance in terms of ROUGE scores compared to existing super- vised systems, while achieving higher perfor- mance based on LLM-based evaluation met- rics. In addition, explore the effectiveness of in-context learning and chain-of-thoughe rea- soning for enhancing its performance. Further- more, find that applying extract-thern generate pipeline with ChatGPT yields signifi- cant performance improvements over abstrac- tive baselines in terms of summary faithful- ness. These observations highlight potential directions for enhancing ChatGPT's capabili- ties in faithful summarization using two-stage approaches.
In this study, comprehensively evaluate Chat- GPT's performance on extractive summarization and investigate the effectiveness of in-context learn- ing and chain-of-thougle explanation approaches. Our experimental analysis demonstrates that Chat- GPT exhibits inferior extractive summarization per- formance in terms of ROUGE scores compared to existing supervised systems, while achieving higher performance based LLM-based evalua- tion metrics. Additionally, observe that using an extract-thenngeenraae pipeline with ChatGPT yields large performance improvements over ab- stractive baselines in terms of summary faithful- ness.
1 Introduction
Document summarization aims to compress text material while retaining its most salient infor- mation. With the increasing amount of pub- licly available text data, automatic summarization approaches have become increasingly important. These approaches can be broadly classified into two categories: abstractive and extractive sum- marization. While abstractive methods (Nallapati al., 2016; Gupta and Gupta, 2019) have the ad- vantage of producing flexible and less redundant summaries, they often struggle with generating un- grammatical or even nonfactual contents (Kryš- ciñski et al., 2019; Zhang et al., 2022b). In contrast,
The main contributions of this paper are: 1) This study represents the first attempt to extend the ap-
hhtnps//chat.oppaaiccm/chh

---

plication of ChatGPT to extractive summarization and evaluate its performance. We investigate the effectiveness of in-context learning and chain-of- thought reasoning approaches for extractive sum- marization using ChatGPT. 3) We further extend the extraction step to abstractive summarization and find that the extract-then-genere framework could improve the generated summary faithfulness by large margin compared to abstractive-only baselines without hurting summary qualities.
to produce accurate and trustworthy summaries de- void of grammatical inconsistencies. Formally, given a document consisting n sentences, the goal of an extractive summarization system is to produce a summary 8 comprising of m(m < n) sentences, by directly extracting rel- evant sentences from the source document. Most existing work formulates it as sequence label- ing problem, where the sentences are selected by model M based on the probability of whether it should be included in the summary 8:
2 Related Work
Most extractive summarization works formulate the task as a sequence classification problem and use sequential neural models with diverse encoders such recurrent neural networks (Cheng and La- pata, 2016; Nallapati al., 2016) and pre-trained language models (Liu and Lapata, 2019; Zhang al., 2023b). Another group of works formu- lated extractive summarization as a node classifica- tion problem and upplied graph neural networks to model inter-sentence dependencies (Xu et al., 2019; Wang et al., 2020; Zhang et al., 2022a, 2023a). Several studies also explored the use of large language models (Brown et al., 2020) for summa- rization. Goyal et al. (2022) found that while the former obtained slightly lower Rouge scores, hu- man evaluators preferred them. Likewise, Zhang et al. (2023d) reported that large language model- generated summaries were on par with human- written summaries in the news domain. In addition. Yang et al. (2023) explored the limits of ChatGPT on query-based summarization other than generic summarization. Luo et al. (2023) explored the use of ChatGPT as factual inconsistency evaluator for abstractive summarization. Zhang al. (2023c) proposed a self-evaluation and revisement framework with ChatGPT. While most of the exist- ing research has focused on abstractive summariza- tion, this work aims to investigate the applicability of ChatGPT to extractive summarization and ex- amine whether extractive methods could enhance abstractive summarization faithfulness.
arg max par d).
(1)
In the training of supervised summarization mod- els, it is common employ greedy algorithm, as described in (Nallapati et al., 2017). generate extractive ground-truti labels (ORACLE) by select- ing multiple sentences that maximize the ROUGE score compared to the gold summary.
3.2 In-context Learning
Recent studies have shown that large language mod- els have strong few-shot performance on various downstream tasks, known in-context learning (ICL) (Brown al., 2020). The standard ICL prompts a large language model, M, with a of pairs and predicts
exemplar document-summen a summary & for the document by:
arg (M d, (dd st).,s))...(d
(d. s
Besides simple input-output pairs, previous works also show that including explanations and th-thugh (COT) reasoning in prompts (Nye et al., 2021; Wei et al., 2022) also benefits language models, represented as:
-rr max ( d)C),
(3)
where C = {(d) c² sd). ch, **} is the set of
triplets in prompts. Be- sides zero-shot setting, this study also investigates the impact of in-context learning on extractive sum- marization, with and without explanations.
3 Methods
3.1
Task Formulation
3.3 Extract-abstract Summarization
Extractive summarization systems form a sum- mary by identifying and concatenating the most salient sentences from a given document. These approaches have gained widespread traction in var- ious real-world applications owing to their ability
It not new use extractive summaries guide abstractive summary generations (Dou et al., 2020: Wang al., 2022). Here we also propose to use LLM in a two-stage manner: extract salient sen- tences to form extractive summaries (s*) first, and

---

CNN/DM
XSum
Models
R1
R2
RL
G-EVAL
R1
R2
RL
G-EVAL
SOTA-Fxt
44.41
20,86
40.55
3.28
24,86
4.66
18.41
2.60
ChatGPT-Ext
39.25
17.09
25.64
3.24
19.85
2.96
13.29
2.67
context
42.38
17.27
28.41
3.30
17.49
3.86
12.94
2.69
42.26
17.02
27.42
3.10
20.37
4.78
14.21
2.89
SOTA-Abs
47.78
23.55
44.63
3.25
49.07
25.13
40.40
2.79
ChaGGTT-ABs
38.48
14.46
28.39
3.46
26.30
7.53
20.21
3.47
Reddit
PubMed
Models
RI
R2
RL
G-EVAL
R1
R2
RL
G-EVAL
SOTA-Ext
25.09
6.17
1.82
41.21
14.91
36.75
ChatGPT-Ext
21.40
4.69
14.62
1.87
36.15
11.94
25.30
2.12
+ context
22.32
4.86
14.63
1.83
36.78
11.86
25.19
2.14
reason
21.87
4.52
14.65
1.83
37.52
12.78
26.36
SOTA-Abs
32.03
11.13
25.51
1.87
45.09
16.72
41.32
2.78
ChaGPTT-Abs
24.64
5.86
18.54
2.43
36.05
12.11
28.46
2.70
Table Summarization results on four benchmark datasets. '+context' and "+reason' refer to ChatGPT with three in-context examples and human reasoning. The best results in both extractive and abstractive settings are in bold.
Dataset
Domain
Doc #words
Sum fwords
#Ext
et al., 2018) is a one-sentence news summarization dataset with professionally written summaries. We follow the common splits (204,045/1 1,33//111,//11,3343 PubMed (Cohan et al., 2018) is scientific paper summarization dataset and we use the introduc- tion section the article and the abstract section as the summary following (Zhong et al.. 2020) with common splits (83,233/4,946/5,0255 Red- dit (Kim et al., 2018) is highly abstractive dataset collected from social media platforms with split (41,675/645/645)
Reddit
Media
482.2
28.0
2
XSum
News
430.2
23.3
2
CNN/DM
News
766.1
58.2
PuhMed
Paper
444
209.5
Table 2: Detailed statistics of the datasets. Doc words and Sum words refer to the average word number in the source document and summary. Ext refers the number of sentences to extract.
then ask the LLM to generate summaries guided by the extractive summaries, represented as:
Evaluation: We conducted evaluation of CharGPT's summarization performance utilizing ROUGE (Lin and Hovy, 2003) following previous studies. We also employ GPT-based evaluation metric G-EVAL (Liu et al., 2023). To investigate the faithfulness of the summarics, we employed common metrics FactCC (Kryšciński et al., 2019) and QuestEvvl (Scialom et al., 2021).
p(s d) II p (st d.))
(4)
where 8şr denotes the previous generated tokens before step 1. We explore the extract-then--eneraae pipeline in this study, aiming to alleviate the hallu- cination problems in LLM summary generation.
We selected the best prompts on dev set 50 examples and randomly sampled 1000 examples from each test set of the original dataset for evalua- tion. The detailed prompts used in the experiments and more details about the experimental setup be found in Table and Appendix B.
4 Experiments and Analysis
4.1 Experiment Settings
We chose four publicly available bench- mark datasets as listed in Table 2, ensuring that they are consistent with previous fine-tuning approaches. CNN/DailyMMail (Hermann et al., 2015) is the most widely-adopted summarization dataset that con- tains news articles and corresponding highlights as summaries. We use the non-anonymized version and follow the common training/validationnesinng splits 1,499. XSum (Narayan
Datasets:
4.2 Experiments Results
The overall results are shown in Table 1. The upper block includes extractive results and SOTA scores from MatchSum (Zhong et al., 2020). The lower block includes abstractive results and SOTA scores from BRIO (Liu et al., 2022) for CNN/DM and XSum, SummaReranker (Ravaut et al., 2022) for

---

Dataset
Setting
G-EVAL
FactCC
QuestEval
Abs
18.54
2.43
9.46
40.79
Reddit
Ext-Abs
18.26
2.60
60.40
49,45
Oracle-Ahs
19.37
2.64
59.75
48.93
Abs
20.21
2.67
5.42
46.14
XSum
Ext-Abs
18.55
2.28
55.73
53.25
Oracle-Abs
21.10
2.72
55.03
53.21
Abs
28.46
2.70
8.37
42.83
PubMed
Ext-Ahs
26.50
2.81
26.38
44.32
Oracle-Abs
26.51
2.83
27.35
44.50
Abs
28.39
3.24
6.35
45.32
CNN/DM
Ext-Abs
29.16
3.50
51.65
51.72
Oracle-Abs
33.32
3.51
53.67
52.46
Table 3: Summarization results of the extract-then-genen pipeline. Abs, Ext-Abs, and Oracle-Abs refer to the theneeerate pipeline, and generation based on ORACLE, respectively.
generate-only baseline, the extract-then generate
Reddit, and GSum (Dou et al., 2020) for PubMed. is observed that ChatGPT generally achieves lower ROUGE scores in comparison to previous fine-tuning methods for all datasets under both extractive and abstractive settings, but achieves higher scores in terms of LLM-based evaluation metric G-EVAL. The findings are consistent with the previous conclusion in (Goyal et al., 2022; Zhang et al., 2023d). We also observe that ChatGPT-Ext outperforms ChatGPT-Abs in two ex- tractive datasets CNN/DM and PubMed while per- forming worse in the other two abstractive datasets. We argue the results are due to the bias within the reference summaries of the dataset and the limit of ROUGE scores. Nonetheless, notice that de- spite being primarily designed for generation tasks, ChatGPT achieves impressive results extractive summarization, which requires comprehension of the documents. The decoder-only structure of Chat- GPT doesn't degrade its comprehension capability compared to encoder models like BERT. We also find that the ROUGE score gap between ChatGPT and SOTA fine-tuned baselines are smaller in the extractive setting than in the abstractive setting,
in Table 5 in Appendix.
4.3 Extract Then Generate
We conduct further experiments to examine the ef- fectiveness of the extract-then-generae framework as presented in Table
The results show large improvements in sum- mary factual consistency across all four datasets with the extract-then then-generate framework. Notably, the FactCC scores extremely low for generate- only baselines (less than 10 percent), highlighting the hallucination problems of CharGPTTbbseed sum- marization, where ChatGPT tends make up new content the summary. Nevertheless, the extract- then-generate framework effectively alleviates the hallucination problem of abstractive summaries by guiding the summary generation process with ex- tracted salient sentences from the documents. We also find that guiding ChatGPT summary gener- ation with its own extracted summaries leads similar summary faithfulness improvements com- pared to guiding generation with ORACLE.
The results also indicate that in-context learning and reasoning are generally beneficial for the ex- tractive summarization task across four datasets in different domains. We only observe performance degradation for in-context learning on the XSum dataset. We argue that the degradation comes from the short ORACLE of XSum, which brings more confusion with few ORACLE examples. How- ever, with chain-of-thought reasoning explanations, ChatGPT can better understand the pattern and thus shows improvements with in-context reasoning. More in-context learning results could be found
terms of summary quality, the results demon- strate that the performance of ChatGPT improves largely in terms of ROUGE scores when grounded with the ORACLE summaries. However, the ROUGE score performance of the extract-then- generate framework relies heavily on the extrac- tive performance when grounded with its ex- tractive summaries. In summary, the extract-then- generate framework could effectively improve the summary faithfulness with similar or even better summary quality.

---

4.4 Positional Bias
emphasis of the paper is explore extractive sum- marization and Aurractthen-generaa pipeline with LLM, but not compare different open and closed LLMs.
Lead bias is common phenomenon in extrac- tive summarization, especially in the news domain, where early parts of article often contain the most salient information. As shown in Figure find that the position distribution of the ChatGPT extracted summary sentences is skewed towards a higher position bias than the ORACLE sentences. In addition, in-context learning brings more posi- tional bias to the summaries. The results indicate that LLMs may rely on superficial features like sentence positions for extractive summarization.
Acknowledgement
This work partially supported by NSF through grants ITS-1763365 and IIS-2106972.
We express our gratitude to the anonymous re- viewers for their valuable reviews and feedback.
References
0.8
Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901.
07
Incontext
No contesl
0.6
0.5
0.4
0.1
Jianpeng Cheng and Mirella Lapata. 2016. Neural sum- marization by extracting sentences and words. arXiv preprint arXiv: 1603.07252.
0.0
0.20%
20% 42%
40% 60% Iaiitionn
60% 30%
80% 100%
Figure I: Position distribution of extracted sentences on 1000 random samples from the CNN/DM test set.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discouree attention model for abstractive summarization of long documents. arXiv preprint arXiv:
5
Conclusion
This paper presents thorough evaluation of Chat- GPT's performance on extractive summarization across four benchmark datasets. The results indi- cate ChatGPT's strong potential for the task and the possibility of generating factual summaries using the extract-generate framework. Overall, this study suggests that ChatGPT is powerful tool for text summarization, and we hope the insights gained from this work can guide future research in this area.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2020. Gsum: A general framework for guided neural abstractive summariza- tion. arXiv preprint arXiv:2010.0800
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022.
News summarization and evaluation in era gpt-3. arXiv preprint arXiv:2209.12356.
Som Gupta and Sanjai Kumar Gupta. 2019. Abstractive summarization: An overview of the state the art. Expert Systems with Applications, 121:49-65.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances neural information processing systems, pages 1693-1701.
Limitations
Instead of conducting experiments the entire test set, we randomly sample 1000 examples from each dataset test set due to budget limits. Previous research efforts (Goyal al., 2022; Zhang al., 2023d) have also been limited in their testing of GPT-3 on small number instances.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2018. Abstractive summarization of reddit posts multi-level memory networks. arXiv preprint arXiv:1811.0077
Our experimental results are mainly evaluated with various automatic metrics (summary quality and faithfulness). We plan to include human study further verify the conclusions in the future.
Wojciech Kryácińskii. Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv: /9t0./99901110000 12840.
We only use 3.5-3.55turbb model from openAI API as an instance of large language models. The
Chin-Yer Lin and Eduard Hovy. 2003. Automatic
evaluation of summaries using n-gram co-occurrence

---

statistics. In Proceedings of the 2003 Human Lan- guage Technology Conference of the North American Chapter of the Association for Computational Lin- guistics, pages 150-157.
Fei Wang, Kaiqiang Song, Hongming Zhang, Lifeng Jin, Sangwoo Cho, Wenlin Yao. Xiaoyang Wang. Muhao Chen, and Dong Yu. 2022. Salience alloca- tion as guidance for abstractive summarization. arXiv preprint arXiv:2210. 12330.
Yang Liu, Dan Iter. Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nlg evaluation using gpt-4 with better human align- ment. arXiv preprint arXiv:2303. 16634.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Edd Chi, Quoe Le, and Denny Zhou. 2022.
Chain of thought prompting elicits reasoning in large language models. arXiv preprint urXiv:2201.11903
Yang Liu and Mirella Lapata. 2019. Text summa- rization with pretrained encoders. arXiv preprint 3xXiv:9908008345
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Discourse-aware neural extractive model for text summarization. arXiv preprint arXiv:1910. 1414422.1..
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. BRIO: Bringing order to abstractive summarization, In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2890-2903, Dublin, Ireland. Association for Computational Lin- guistics.
Xianjun Yang. Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.0808)
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2022a. HEGEL: Hypergraph transformer for long document summarization. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 10167-10176. Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as factual inconsistency evaluator for abstractive text summarization. arXiv preprint 1xXi::3333116211.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: recurrent neural network based sequence model for extractive summarization of doc- uments. In Thirty-first AAA/ conference arrificial intelligence.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023a. Contrastive hierarchical discourse graph for scien- tific document summarization. In Proceedings of the 4th Workshop on Computational Approaches to Dis- course (CODI 2023), pages 37-47, Toronto, Canada. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence mns and beyond. arXiv preprint arXiv:1602.06023.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023b. DiffuSum: Generation enhanced extractive summa- rization with diffusion. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pages 13089-13100 Toronto, Canada. Association for Computational Linguistics.
Shashi Narayan, Shay Cohen, and Mirella Lap- ata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural net- works for extreme summarization. arXiv preprint arXiv:1808.08745
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023c. Summit: Iterative text summarization via chatgpt.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz. Maarten Bosma, David Luan, et al. 2021. Show your work: Scratch- pads for intermediate computation with language models. arXiv preprint arXiv:2112.0011
Haopeng Zhang, Semih Yavuz, Wojciech Kryscinski, Kazuma Hashimoto, and Yingbo Zhou. 2022b. Im- proving the faithfulness of abstractive summarization via entity coverage control. In Findings of the Asso- ciation for Computational Linguistics: NAACI 2022, pages 528-535, Seattle, United States, Association for Computational Linguistics.
Mathieu Ravaut, Shafiq Joty, and Nancy Chen. 2022. Summareranker: A multi-task mixture-of-experts re- ranking framework for abstractive summarization. arXiv preprint arXiv:2203. 066669699...
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023d. Benchmarking large language models for news summarization. arXiv preprint 2001.33301.33848
Thomas Scialom, Paul-Alexis Dray, Gallinari Patrick. Lamprier Sylvain, Piwowarski Benjamin, Staiano Ja- copo, and Wang Alex. 2021. Questeval: Summariza- tion asks for fact-based evaluation. arXiv preprint arXiv:2103. 12693.
Ming Zhong, Pengfei Liu, Yiran Chen, Danging Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extrac- tive summarization as text matching. arXiv preprint arXiv:2004.08795.
Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang. 2020. Heterogeneous graph neural networks for extractive document summariza- tion. arXiv preprint 2004v 2009.3333.........

---

Appendix
Setting Extractive
Prompt
System: You are an extractive summa- rizer that follows output pattern. Please extract sentences as summary. The summary should con- tain m sentences. Document: [Tesr Document) |Format Instruction]. System: You are an abstractive sum- marize that follows the output pattern. User: Please write a summary for the document. Document: [Test Docu- ment] [Format Instruction]
A Prompts
User:
Here we list prompts used in our experiments for extracted and generated summaries Table Note that according to OpenAI's document, the model could receive two categories of prompts: system prompt and user prompt, where the system prompt functions the global instruction to initial- ize the model and the prompt the question proposed by users.
Abstractive
In-context
System: You an extractive summa- rizer that follows the output pattern. The following examples are successful extractive summarization instances: Document- Sunmary Pairs). Please summarize the follow- ing document. Document: [Test Doc- ument]. The summary should contain sentences. [Format Instruction System: You an extractive summa- rizer that follows output pattern. User: The following examples are successful extractive summarization instances: [n Document-S -aammeer----aa-------- Reason Triads). Please summarize the following document and give the reason. Document: [Test Documenr). The summary should contain m sen- tences, [Format Instruction].
User:
B Experimental Setup
We employed the 3pt-3.5-turbb model? for the gen- eration and assessment summaries, maintaining a temperature setting of to ensure reproducibility. Regarding the datasets, random sampling method was adopted where 1000 samples were chosen for each dataset for experimental purposes. Furthermore, smaller subset of 50 samples was utilized for the discovery of optimal prompts and hyperparameters. The random seed was established at 101 to promote consistency.
Explanation
Extract- abstract
System: You an abstractive sum- marizer that follows the output pat- tern.
In accordance with established research, the ROUGE' F-1 scores were implemented as the au- tomatic evaluation metrics (Lin and Hovy, 2003). To be specific, the ROUGE-1/2 1/2 scores serve as measures of summary informativeness, while the ROUGE-L score gauges the fluency of the sum- mary. In addition to these measures, GPT model was integrated as a summary evaluator to mimic human evaluation processes. This evaluator was designed to assess the summary based on a compre- hensive analysis of coherence, consistency, fluency, and efficiency. The findings from each experiment are reported single-run results.
User: Please revise extracted sum- mary based on the document. The revised summary should include information in extracted summary. Document: [Test Docuemor] Extrac- tive Summary: [Extractive Summary] Forrat Instruction]
Evaluator
System: You are summary evaluator that follows the output pattern. You give scores for the summaries bused on the comprehensive considcration following criteria:
(I) Coherence: "the collective quality of all sentences":
(2) Consistency: "the factual align- ment between the summary and the reference":
The experiments involving each dataset, which includes 1000 examples, will run for 1.5 hours to perform both inference and evaluation.
(3) Fluency: " "hee quality of individ- ual sentences":
(4) Efficiency: "If the summary is con-
cise"
C In-context Learning Results
User: Please evaluate the mary based the reference Reffnar.Referenceeff.. Summaryl Summary: Summaryl Instruction).
The detailed in-context learning results are shown in T 5
Document length Analysis
Table 4: Prompts used for both extractive and abstrac- tive summarization. m the number of extracted sen- tences defined in Table 2. Document-sumen document- document-summary--ae is the number of context instances.
We further investigate the influence of document length on the summarization performance, pre-
pairs and triads are the input contexts.
completions-upi
'ROUGE: hhtps//hrpi.arggppiccttouggoooo

---

CNN/DM
XSum
Context
R1
R2
RL
R1
R2
RL
39.25 0.23
15.36 1.10
25.90 0.97
19.85 2.59
2.96 + 2.59
13.29 + 1.30
40.62 0.70
17.00 1.06
26.44 0.84
15.33 0.50
2,48 + 0.19
11.48 + 0.13
1w/R
38.83
14.94
25.36
17.86 1.73
3.29 + 0.85
12.55 : 1.29
40.91 0.69
15.68 + 0.61
26. 0.83
18.61 0.39
4.42 +: 0.97
14.06 + 2.01
2w/R
41.70 1 0.70
13.95 1 0.92
26.98 1.33
17.95 1 3.03
4.11 1: 1.01
13.46 1: 1.76
3
42.38 1 0.13
17.27 T1 0.23
28.41 1 0.31
17.49 1: 1.87
3.86 1 1.55
12.94 1 2.16
3w/R
42.26 + 1.38
17.02 + 1.60
27.42 + 1.62
20.37 + 1.61
4.78 + 0.44
14.21 + 1.07
42.26 0.50
17.41 0.83
27.96 0.83
16.68 1.56
3.72 0.20
0.20
12.12 + 1.19
4w/R
41.23
17.08
18.17
4.05 + 0.38
12.74 0.94
40.71 1.92
16.96 0.91
27.42 1.26
17.43 1.08
3.53 + 0.96
12.33 + 0.51
5w/R
40.18 0.83
15.15 1.44
25.98
1.91
19.55 0.64
4.29 + 0.46
13.13 + 0.68
Table 5: In-context learning experimental results CNN/DM and XSum datasets. For each dataset, randomly sampled 50 data from the test set. In each section, w/R means we provide human written reasons for each context document. For test document, we also the system to generate the reason why it choose selected sentences.
sented in Figure 2. Our findings suggest that Chat- GPT maintains consistent performance across doc- uments of different lengths, indicating the model's robustness in the context of extractive summariza- tion.
In-comest
No context
40
ROUGE:
02
Figure 2: ROUGE-1 Score distribution over document of different lengths.
E
Case Study
Here show the ChatGPT-generater summaries with different prompt settings in Table 6 for one example from the CNNDM dataset.

---

Document
Duredevil Nik Wallenda says he ll walk untethered top of 400-foot observation wheel in Orlando, Florida, this month. Wallenda said Monday a New York City news conference that the Orlando Eye will be moving when he attempts his feat on April 29. The Orlando Eye, part of new entertainment complex, will offer views of contral Florida from inside 30 enclosed, air-conditioned glass capsules when opens to the public on May Eyes on the prize: high-wire performer Nik Wallenda announces his latest stunt at the 400-foot Orlando Eye, during news conference, in New York on Monday. Tough challenge: the 36-year-old daredevil will walk atop the Orlando Eye as it turns on April 29. The Orlando Eye team issued a statement saying it's excited have Wallenda attempt the 'amazing stunt,' No distance set yet. but Wallenda, 36, said he was not likely walk the entire minutes or so that takes the wheel to go a full circle. Wallenda previously walked stop Ferris wheel in Santa Cruz, California, but said the size of the much larger Orlando wheel and the fact that he will not use pole sets this stunt apart. The seventh- -eeenthgnnneaiionn member of the "Flying Wallenda' family of acrobats has walked across the Grand Canyon and Niagara Falls, In November, he walked twice between two Chicago skyscrapers without net or harness, doing the second walk hlindfolded. Wallenda is the great- grandson of Karl Wallenda, who fell to his death in Puerto Rico at 73.
the performance has been
Reference
The 36-year-old will stage his next stunt on April 29. In November, Wallenda walked back and forth between two Chicago skyscrapers in a live television event. His great-grande graat-grandfather Karl Wallenda died in a tightrope walk in Puerto Rico in 1978. Wallenda has also tightrope walked across Niagara Falls and the Grand Canyon.
ORACLE
Tough challenge: the yeaycoo-old daredevil will walk atop the Orlando Eye as it turns April 29. The seventh-generatie
member of the 'Flying Wallenda' family of acrobats has walked across the Grand Canyon and Niagara Falls. In November, he walked twice hetween two Chicago skyscrapers without a net hamness, doing the second walk blindfolded.
Abstractive
Daredevil Nik Wallenda plans walk unlethered atop the 400-foot Orlando Eye observation wheel on April 29th. The 36-year-old high- performer, a seventh-generation member of the "Flying Wallenda" acrobat family, has previously completed daring walks across the Grand Canyon, Niagara Falls, and between Chicago skyscrapers. Although the exact distance for this stunt hasn't it been determined, Wallenda stated he's unlikely to walk the entire 20 minutes takes wheel to complete a full circle.
Extractive
Daredevil Nik Wallenda says he' Il walk untethered on top of a 400-foot observation wheel in Orlando, Florida, this month. Wallenda said Monday at a New York City news conference that the Orlando Eye will be moving when he attempts his feat on April 29, The 36-ycar-old daredevil will walk atop the Orlando Eye as it turns on April 29.
In-Context
Daredevil Nik Wallenda says he' II walk untethered on top of a 400-foot observation wheel in Orlando, Florida. this month. Wallenda suid Monday at a New York City news conference that the Orlando Eye will be moving when he attempts his feat on April 29. Wallenda is the great-grandson of Karl Wallenda, who fell to his death in Puerto Rico at 73.
Extract-abstract
Nik Wallenda, a daredevil and geenth-geneettonn member of the "Flying Wallenda family of acrohats, will attempt to walk untethered on top of a 400-foot observation wheel in Orlando, Florida, on April 29. The Orlando Eye, part of new entertainment complex, will offer views of central Florida from inside 30 enclosed, air-conditioned glass capsules when it opens to the public on May 4. Wallenda previously walked atop Ferris wheel in Santa Cruz, California, but he said the size of the much larger Orlando wheel and the fact that he will not pole sets this stunt apart. Wallenda is the great- grand-gr of Karl Wallenda, who fell to his death in Puerto Rico at 73.
Table 6: Case study of different settings


Reviews:
[{"review_id": "bwt1mNi88Z", "reviewer": "Reviewer_dGKd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely value the priceless evaluation of our paper and the devoted time you've dedicated to offering valuable feedback. We have thoroughly assessed each of your points and are eager to provide thorough responses that address them comprehensively:\n\n**Question:The paper lacking fair baseline methods to compare with. The authors say their results on evaluated on randomly selected 1000 results. However, some the SOTA baselines as Table 1. are based on the full dataset. The paper did not do human examination to better validate the improvement in factuality or overall quality. Also based on GPT-based metrics, GPT4 will provide more accurate evaluation then ChatGPT. ChatGPT is essentially a generative method, it may be not accurate enough in copying the original sentences from the articles, I did not see how the authors are measuring and addressing this problem.**\n\n**Answer:** Thanks for pointing this out. Unfortunately due to budget and expense limits, we couldn't test CHATGPT on the full test sets on the large scale datasets. We expected using 1000 sampel test exmaples would also give a good approximation of the summarization performacne of ChatGPT.\n\n\n We agree that **human examination** will better demonstrate the performance of different systems. The human annotation results are summarized in the following table on both CNNDM and XSUM datasets under the zero-shot settings. The aspect-based ratings are from a scale of 1 to 5, and the human preference is the percentage of the summary being the best system.\n\n\n**Human Eval for CNN/DM datasets**\n\n| Model | Coherence | Fluency | Relevance | Consistency | Conciseness | Overall | Human Pref |\n|--------------|--------------------|------------------|--------------------|----------------------|----------------------|------------------|------------------------------------|\n| BART         | 3.92               | 4.16             | 4.00               | 3.12                 | 3.64                 | 3.24             | 0.04                        |\n| T5           | 3.72               | 4.24             | 4.32      | 3.52                 | 3.84                 | 3.68             | 0.10                               |\n| PEGASUS      | 3.20               | 3.53             | 3.33               | 2.87                 | 1.85                 | 1.63             | 0.00                               |\n| ChatGPT      | 4.20               | 4.36             | 4.28               | 4.01                 | 3.92        | 4.01             | 0.34                               |\n\n\n\n**Human Eval for XSum datasets**\n\n| Model | Coherence | Fluency | Relevance | Consistency | Conciseness | Overall | Human Pref |\n|--------------|--------------------|------------------|--------------------|----------------------|----------------------|------------------|------------------------------------|\n| BART         | 3.97               | 4.30             | 4.13               | 3.30                 | 3.93        | 3.84             | 0.30                       |\n| T5           | 3.84               | 4.32             | 4.02               | 3.63                 | 3.84                 | 3.25             | 0.08                               |\n| PEGASUS      | 3.13               | 4.10             | 3.52               | 2.87                 | 2.03                 | 2.41             | 0.00                               |\n| ChatGPT      | 4.03               | 4.40    | 4.30      | 3.93                 | 3.87                 | 3.92             | 0.24                               |\n\nAccording to the results, human evaluation generally aligns with the GPT evaluation and ChatGPT outperforms other models by a large margin.\n\n We used Pearson Correlation to evaluate the inter-annotator agreement:\n|                 |       |\n| --------------- | ----- |\n| **Coherence**   | 0.658 |\n| **Fluency**     | 0.851 |\n| **Relevance**   | 0.796 |\n| **Consistency** | 0.621 |\n| **Conciseness** | 0.747 |\n| **Overall**     | 0.742 |\n\nWe will add all these human studies in our final version.\n\nIn terms of **extractive summarization of CHATGPT**, it has a strong instruction following capabilities. In the generation prompt, we use \"Please extract sentences as the\nsummary. The summary should contain\nm sentences. Document: [Test Document] [Format Instruction]. \" and found CHATGPT could extract sentences accurately without error during the generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written and provide enough details like prompt for reproducibility.\n2. The authors have conducted through experiments and analysis. For example, conduct experiments on 4 benchmark datasets and evaluated using different automatic metrics. \n3. The insights gained in the paper can be helpful to further research. For example, extract-then-abstract can be used to improve factuality.", "reasons_to_reject": "The paper lacking fair baseline methods to compare with. The authors say their results on evaluated on randomly selected 1000 results. However, some the SOTA baselines as Table 1. are based on the full dataset.\nThe paper did not do human examination to better validate the improvement in factuality or overall quality. Also based on GPT-based metrics, GPT4 will provide more accurate evaluation then ChatGPT. \nChatGPT is essentially a generative method, it may be not accurate enough in copying the original sentences from the articles, I did not see how the authors are measuring and addressing this problem.", "questions_to_authors": "", "paper_topic_and_main_contributions": "1) This study represents the first attempt to extend the application of ChatGPT to extractive summarization and evaluate its performance. \n2) they investigate the effectiveness of in-context learning and chain-of thought reasoning approaches for extractive summarization using ChatGPT. \n3) they further extend the extraction step to abstractive summarization and find that the extract-then-generate framework could improve the generated summary faithfulness by a large margin compared to abstractive-only baselines without hurting summary qualities.\n\n\n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6utVZJOgq8", "reviewer": "Reviewer_6bTC", "summary": "", "questions": "A.\tIn table 2, how the number of sentences to extract is computed？\n\nB.\tWhen prompt ChatGPT to extract sentences, does all the output are original sentences from the document without any change?\n\nC.\tWhy CoT doesn't work for summarization (Table 1) and sometimes hurt the performance?\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review of our paper and the effort you've put into providing valuable feedback. We have carefully considered your concern, and we would like to address comprehensively:\n\n**Question: In table 2, how the number of sentences to extract is computed？**\n\n**Answer:** The number are calculated based on the average number of sentences in the reference summaries in the development set. We use the same number with previous extractive summarization baseline papers like [1][2]. We will add these details in our final version.\n\n\n**Question: The experiment results are not very strong. The paper claims the advance and potential extract-then-generate pipeline with ChatGPT. But in Table 3, (1) the scores show limited improvement compared to Table 1; (2) the factual metrics only compare to abstractive methods but not extractive methods. This method requires more in-depth analysis.**\n\n**Question: Table 4 shows the prompts used for both extractive and abstractive summarization. (1) The extract-then-generate (Line 6) pipeline uses an extractive summary, which is generated by sentence-level extraction (Line 1). This can be a very coarse granularity and hurts flexibility. (2) The prompts ask ChatGPT to ‘extract sentences’ or ‘write a summary’ to implement extractive/abstractive summarization. But actually, ChatGPT directly generates all summaries token by token. This method may not meet the strict, classic definition of extractive summarization. (3) The evaluator is the same ChatGPT. There may be bias in evaluating the performance of models with the same model. This paper gives no more explanation.**\n\n\n**Answer:** ChatGPT has a strong instruction following capabilities. In the generation prompt, we use \"Please extract sentences as the\nsummary. The summary should contain\nm sentences. Document: [Test Document] [Format Instruction]. \" and surprsingly found CHATGPT could extract sentences accurately without error during the generation, so it doesn't violate the strict, classic definition of extractive summarization. \n\nWe agree that ChatGPT as evaluator may cause some self-bias, so we also conducted a **human study.** The human annotation results are summarized in the following table on both CNNDM and XSUM datasets under the zero-shot settings. The aspect-based ratings are from a scale of 1 to 5, and the human preference is the percentage of the summary being the best system.\n\n\n**Human Eval for CNN/DM datasets**\n\n| Model | Coherence | Fluency | Relevance | Consistency | Conciseness | Overall | Human Pref |\n|--------------|--------------------|------------------|--------------------|----------------------|----------------------|------------------|------------------------------------|\n| BART         | 3.92               | 4.16             | 4.00               | 3.12                 | 3.64                 | 3.24             | 0.04                        |\n| T5           | 3.72               | 4.24             | 4.32      | 3.52                 | 3.84                 | 3.68             | 0.10                               |\n| PEGASUS      | 3.20               | 3.53             | 3.33               | 2.87                 | 1.85                 | 1.63             | 0.00                               |\n| ChatGPT      | 4.20               | 4.36             | 4.28               | 4.01                 | 3.92        | 4.01             | 0.34                               |\n\n\n\n**Human Eval for XSum datasets**\n\n| Model | Coherence | Fluency | Relevance | Consistency | Conciseness | Overall | Human Pref |\n|--------------|--------------------|------------------|--------------------|----------------------|----------------------|------------------|------------------------------------|\n| BART         | 3.97               | 4.30             | 4.13               | 3.30                 | 3.93        | 3.84             | 0.30                       |\n| T5           | 3.84               | 4.32             | 4.02               | 3.63                 | 3.84                 | 3.25             | 0.08                               |\n| PEGASUS      | 3.13               | 4.10             | 3.52               | 2.87                 | 2.03                 | 2.41             | 0.00                               |\n| ChatGPT      | 4.03               | 4.40    | 4.30      | 3.93                 | 3.87                 | 3.92             | 0.24                               |\n\nAccording to the results, human evaluation generally aligns with the GPT evaluation and ChatGPT outperforms other models by a large margin.\n\nWe used Pearson Correlation to evaluate the inter-annotator agreement:\n\n|                 |       |\n| --------------- | ----- |\n| **Coherence**   | 0.658 |\n| **Fluency**     | 0.851 |\n| **Relevance**   | 0.796 |\n| **Consistency** | 0.621 |\n| **Conciseness** | 0.747 |\n| **Overall**     | 0.742 |\n\nWe will add all these human studies in our final version.\n\n\n**Question: When prompt ChatGPT to extract sentences, does all the output are original sentences from the document without any change?**\n\n**Answer:** As mentioned above, ChatGPT has a strong instruction following capabilities. Surprsingly we found that CHATGPT could extract sentences accurately without error during the generation.\n\n\n**Why CoT doesn't work for summarization (Table 1) and sometimes hurt the performance?**\n\n**Answer:** This is a fairly hard question to answer given ChatGPT is a black-box model. We noticed CoT brings very small degragation on CNNDM and larger on Reddit compared to in-context learning. One possible reason is our human reasoning reference is not ideal and brings more confusion to the model.\n\n\n\n[1]Text Summarization with Pretrained Encoders\n\n[2]Extractive Summarization as Text Matching", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper focus on extractive summarization analyses with ChatGPT, which hasn't been discussed much yet. Extractive summarization is an important research direction that will contribute to factuality and simplification and has many application scenarios in real life.\n2.\tThe observation that using an extract-then-generate pipeline with ChatGPT improves is consistent with intuition, providing practical insights for few-shot summarization.\n", "reasons_to_reject": "1.\tThis analytical paper makes some contributions but lacks novelty. Combining extractive and abstractive methods has been proven effective in previous work. This has also been explored based on LLM. \n2.\tThe experiment results are not very strong. The paper claims the advance and potential extract-then-generate pipeline with ChatGPT. But in Table 3, (1) the scores show limited improvement compared to Table 1; (2) the factual metrics only compare to abstractive methods but not extractive methods.  This method requires more in-depth analysis.\n3.\tTable 4 shows the prompts used for both extractive and abstractive summarization. (1) The extract-then-generate (Line 6) pipeline uses an extractive summary, which is generated by sentence-level extraction (Line 1). This can be a very coarse granularity and hurts flexibility.\n(2) The prompts ask ChatGPT to ‘extract sentences’ or ‘write a summary’ to implement extractive/abstractive summarization. But actually, ChatGPT directly generates all summaries token by token. This method may not meet the strict, classic definition of extractive summarization.\n(3) The evaluator is the same ChatGPT. There may be bias in evaluating the performance of models with the same model. This paper gives no more explanation.\n", "questions_to_authors": "A.\tIn table 2, how the number of sentences to extract is computed？\n\nB.\tWhen prompt ChatGPT to extract sentences, does all the output are original sentences from the document without any change?\n\nC.\tWhy CoT doesn't work for summarization (Table 1) and sometimes hurt the performance?\n", "paper_topic_and_main_contributions": "This paper presents an evaluation of ChatGPT's performance on extractive summarization on benchmark datasets. The paper also explores the effectiveness of in-context learning and chain-of-thought for enhancing the performance of ChatGPT in extractive summarization tasks. The results show that ChatGPT exhibits inferior extractive summarization performance in terms of ROUGE scores, but achieves higher performance on advance metrics, like LLM evaluation (G-EVAL). The authors also show that using an extract-then-generate pipeline with ChatGPT improves summary faithfulness compared to abstractive baselines.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Bottom-Up Abstractive Summarization\n\n2. Guiding Large Language Models via Directional Stimulus Prompting"}, "extra_text": {}}, {"review_id": "BwxGScAdws", "reviewer": "Reviewer_jH6Y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review of our paper and the effort you've put into providing valuable feedback. We have carefully considered your concern, and we would like to address comprehensively:\n\n**Question: Given the complexity of summary quality assessment, it is still necessary and recommended to conduct the manual evaluation.**\n\n**Answer:** We agree that human analysis will better demonstrate the performance of different systems. The human annotation results are summarized in the following table on both CNNDM and XSUM datasets under the zero-shot settings. The aspect-based ratings are from a scale of 1 to 5, and the human preference is the percentage of the summary being the best system.\n\n\n**Human Eval for CNN/DM datasets**\n\n| Model | Coherence | Fluency | Relevance | Consistency | Conciseness | Overall | Human Pref |\n|--------------|--------------------|------------------|--------------------|----------------------|----------------------|------------------|------------------------------------|\n| BART         | 3.92               | 4.16             | 4.00               | 3.12                 | 3.64                 | 3.24             | 0.04                        |\n| T5           | 3.72               | 4.24             | 4.32      | 3.52                 | 3.84                 | 3.68             | 0.10                               |\n| PEGASUS      | 3.20               | 3.53             | 3.33               | 2.87                 | 1.85                 | 1.63             | 0.00                               |\n| ChatGPT      | 4.20               | 4.36             | 4.28               | 4.01                 | 3.92        | 4.01             | 0.34                               |\n\n\n\n**Human Eval for XSum datasets**\n\n| Model | Coherence | Fluency | Relevance | Consistency | Conciseness | Overall | Human Pref |\n|--------------|--------------------|------------------|--------------------|----------------------|----------------------|------------------|------------------------------------|\n| BART         | 3.97               | 4.30             | 4.13               | 3.30                 | 3.93        | 3.84             | 0.30                       |\n| T5           | 3.84               | 4.32             | 4.02               | 3.63                 | 3.84                 | 3.25             | 0.08                               |\n| PEGASUS      | 3.13               | 4.10             | 3.52               | 2.87                 | 2.03                 | 2.41             | 0.00                               |\n| ChatGPT      | 4.03               | 4.40    | 4.30      | 3.93                 | 3.87                 | 3.92             | 0.24                               |\n\nAccording to the results, human evaluation generally aligns with the GPT evaluation and ChatGPT outperforms other models by a large margin.\n\n We used Pearson Correlation to evaluate the inter-annotator agreement:\n\n|                 |       |\n| --------------- | ----- |\n| **Coherence**   | 0.658 |\n| **Fluency**     | 0.851 |\n| **Relevance**   | 0.796 |\n| **Consistency** | 0.621 |\n| **Conciseness** | 0.747 |\n| **Overall**     | 0.742 |\n\nWe will add all these human studies in our final version.\n\n\n**Question: Evaluating the effectiveness of large models on extractive summarization tasks using only ChatGPT as a representative seems insufficient.**\n\n**Answer:** Thanks for pointing this out. By the time we prepared this draft for submission, ChatGPT is still one of the strongest large language models. We will add more recent LLM like GPT-4 and LLaMa in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is generally well-written and the evaluation of ChatGPT’s performance on extractive summarization is interesting and well explained.", "reasons_to_reject": "1. Evaluating the effectiveness of large models on extractive summarization tasks using only ChatGPT as a representative seems insufficient.\n2. It is not meaningful to compare the factual consistency of the generated and extracted summaries in the experiments (e.g., Table 3), because all the content of the extracted summaries comes from the original text, so it is obvious that the factual consistency is better, and the use of the joint extraction and generation model can significantly improve the quality of summary generation, which is one of the hot topics in recent years, and this paper's work on the \"extract-generate framework\" is not significantly innovative. \n3. Given the complexity of summary quality assessment, it is still necessary and recommended to conduct the manual evaluation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper provides a preliminary experimental test of the effectiveness of ChatGPT on the task of extractive summarization and verifies that the factual consistency of ChatGPT-generated summaries can be further improved by adopting the extract-generate framework.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FHXFO9wVQT", "reviewer": "Reviewer_hwbk", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review of our paper and the effort you've put into providing valuable feedback. We have carefully considered your concern, and we would like to address comprehensively:\n\n**Question: Task formulation section is not needed as we all know what extractive summarization is, let alone the paper is short and has limited space to share the findings. I hence suggest removing it to have more space for other important sections such as discussion or introducing \"in-context\" and \"chain-of-thought\" concepts.**\n\n**Answer:** Thanks for pointing this out. Chain-of-thought is a recent prompting technique to improve the reasoning capabilities of LLM by generating a chain of thought -- a series of intermediate reasoning steps. It is widely used in prompting LLM and we will definitely add more information in the final version of the paper.\n\n\n**Question: While the authors have mentioned that ChatGPT underperforms the fine-tuning methods on the ROUGE metric and outperforms them on LM-based metrics, they have not provided the underlying reasons for this inconsistency.**\n\n**Answer:** The major reason for this inconsistency is the fine-tuned models learn data-specific patterns during the fine-tuning process. In contrast, ChatGPT is under zero/few-shot setting. We will clarify this in the final draft of the paper.\n\n**Question: The paper could make a more substantial contribution with a human analysis on a subset of the results, as automatic metrics may not be sufficient.**\n\n\n**Answer:** We agree that human analysis will better demonstrate the performance of different systems. The human annotation results are summarized in the following table on both CNNDM and XSUM datasets under the zero-shot settings. The aspect-based ratings are from a scale of 1 to 5, and the human preference is the percentage of the summary being the best system.\n\n\n**Human Eval for CNN/DM datasets**\n\n| Model | Coherence | Fluency | Relevance | Consistency | Conciseness | Overall | Human Pref |\n|--------------|--------------------|------------------|--------------------|----------------------|----------------------|------------------|------------------------------------|\n| BART         | 3.92               | 4.16             | 4.00               | 3.12                 | 3.64                 | 3.24             | 0.04                        |\n| T5           | 3.72               | 4.24             | 4.32      | 3.52                 | 3.84                 | 3.68             | 0.10                               |\n| PEGASUS      | 3.20               | 3.53             | 3.33               | 2.87                 | 1.85                 | 1.63             | 0.00                               |\n| ChatGPT      | 4.20               | 4.36             | 4.28               | 4.01                 | 3.92        | 4.01             | 0.34                               |\n\n\n\n**Human Eval for XSum datasets**\n\n| Model | Coherence | Fluency | Relevance | Consistency | Conciseness | Overall | Human Pref |\n|--------------|--------------------|------------------|--------------------|----------------------|----------------------|------------------|------------------------------------|\n| BART         | 3.97               | 4.30             | 4.13               | 3.30                 | 3.93        | 3.84             | 0.30                       |\n| T5           | 3.84               | 4.32             | 4.02               | 3.63                 | 3.84                 | 3.25             | 0.08                               |\n| PEGASUS      | 3.13               | 4.10             | 3.52               | 2.87                 | 2.03                 | 2.41             | 0.00                               |\n| ChatGPT      | 4.03               | 4.40    | 4.30      | 3.93                 | 3.87                 | 3.92             | 0.24                               |\n\nAccording to the results, human evaluation generally aligns with the GPT evaluation and ChatGPT outperforms other models by a large margin.\n\n We used Pearson Correlation to evaluate the inter-annotator agreement:\n\n|                 |       |\n| --------------- | ----- |\n| **Coherence**   | 0.658 |\n| **Fluency**     | 0.851 |\n| **Relevance**   | 0.796 |\n| **Consistency** | 0.621 |\n| **Conciseness** | 0.747 |\n| **Overall**     | 0.742 |\n\nWe will add all these human studies in our final version.\n\n**Question: The reader would like to see a discussion of choosing ChatGPT over the fine-tuning methods. Although this may involve a trade-off, having a discussion on this topic would add to the contributions of the paper.**\n\n**Answer:** As mentioned above, the major limitation of ChatGPT is that we could not fine-tune it with large-scale in-domain data. We experimented with LLaMa [1], one open LLM model on the Reddit dataset, and got ROUGE numbers of 26.24/6.82/21.58. We will add more results in our final draft.\n\n[1]LLaMA: Open and Efficient Foundation Language Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper touches on an interesting and significant topic in the context of text summarization, which could make a substantial contribution to the field.\n- The findings, specifically regarding the potential use of ChatGPT in producing faithful summaries when prompted with an extract-then-generate approach, are particularly interesting.\n- The experiments have been conducted on a handful of datasets from diverse domains (general news, scientific, and social media), indicating the strengths of the proposed arguments in different domains.", "reasons_to_reject": "- The paper must be re-structured. The authors have introduced some concepts such as \"in-context\" and \"chain-of-thought\" learning/reasoning in the introduction for the first time but haven't provided any explanation of what these are. While an attempt has been made to explain \"in-context learning\" in section 3.2, the reader still gets confused about the \"chain-of-thought\" approach as it has never been explained in the paper. This adds to the lack of clarity of the paper, as a significant part of the results are based on these learning approaches as they stand.\n\n- While the authors have mentioned that ChatGPT underperforms the fine-tuning methods on the ROUGE metric and outperforms them on LM-based metrics, they have not provided the underlying reasons for this inconsistency.\n\n- The paper could make a more substantial contribution with a human analysis on a subset of the results, as automatic metrics may not be sufficient.\n\n- The reader would like to see a discussion of choosing ChatGPT over the fine-tuning methods. Although this may involve a trade-off, having a discussion on this topic would add to the contributions of the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This study explores the potential of ChatGPT for text summarization, specifically in an extractive setting. The findings suggest that while ChatGPT achieves comparably lower ROUGE scores (compared to fine-tuning methods), it performs better in LM-based automatic evaluation scores. In the follow-up discussion, the authors found that ChatGPT exhibits higher performance improvement in terms of faithfulness when prompted to perform an extract-then-generate procedure for generating summaries. The experiments on four different and diverse datasets indicate the high potential of ChatGPT in text summarization, as the authors suggest.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Task formulation section is not needed as we all know what extractive summarization is, let alone the paper is short and has limited space to share the findings. I hence suggest removing it to have more space for other important sections such as discussion or introducing \"in-context\" and \"chain-of-thought\" concepts."}}]

Author rebuttal:

Meta-review: The paper provides a thorough analysis of ChatGPT's performance on extractive summarization and contrasts it with conventional fine-tuning techniques. The study is performed on multiple benchmark datasets.  The experimental results show that ChatGPT performs better on LLM-based assessment measures than existing supervised systems, but performs worse on extractive summarization tasks as measured by ROUGE scores. The authors also explore the effectiveness of in-context learning and chain-of-thought reasoning for enhancing chatGPT's performance on extractive summarization. Finally, the research reveals that an extract-then-generate pipeline with ChatGPT significantly outperforms abstractive baselines in terms of summary faithfulness. 

The following pros were revealed by reviewers: 
1. It is the first attempt to extend the application of ChatGPT to extractive summarization and evaluate its performance.
2. Investigation of the effectiveness of in-context learning and chain-of-thought reasoning for extractive summarization using ChatGPT.
3. Experiments with the extract-then-generate framework that significantly outperforms the generated summary faithfulness compared to abstractive baselines. 
4. The paper is well-written and the experiments with ChatGPT for extractive summarization are well organized and explained.

Some drawbacks were also noticed, such as using only one LLM and a limited number of documents in the experiments. The authors addressed all the comments and justified their choices.
Decision: Accept-Findings