Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method
Year: 2023

Abstract:
Automatic summarization generates concise summaries that contain key ideas of source documents. As the most mainstream datasets for the news sub-domain, CNN/DailyMail and BBC XSum have been widely used for performance benchmarking. However, the reference summaries of those datasets turn out to be noisy, mainly in terms of factual hallucination and information redundancy. To address this challenge, we first annotate new expertwriting Element-aware test sets following the “Lasswell Communication Model” proposed by Lasswell (1948), allowing reference summaries to focus on more fine-grained news elements objectively and comprehensively. Utilizing the new test sets, we observe the surprising zero-shot summary ability of LLMs, which addresses the issue of the inconsistent results between human preference and automatic evaluation metrics of LLMs’ zero-shot summaries in prior work. Further, we propose a Summary Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step by step, which helps them integrate more finegrained details of source documents into the final summaries that correlate with the human writing mindset. Experimental results show our method outperforms state-of-the-art fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L on the two datasets, respectively. Dataset and code are publicly available at https://github.com/Alsace08/SumCoT.

heading: 1 Introduction
text: On March 8, Marcin Wasniewski crashed into the back of a lorry on the A444 in Coventry when driving a car. Incredibly, he was injured with just cuts and bruises and cheated death by just millimeters. Paramedics were shocked because of serious damage to the car. This father firmly believed that Jesus saved him.
Element-aware Summary (Ours)
Entity Date Event Result
Trailer 'embedded' into car windscreen in smash on A444 in Coventry. Impact would have 'certainly been fatal' if a couple of inches closer to driver. Paramedics were shocked when Marcin Wasniewski walked out unaided.
Dataset-specific Summary (Original)
Element Extraction
Summarization
Article: The 69-year-old's yamaha collided with a Nissan car between handley's corner and barre garroo crossroads at about 17:00 bst on 4 June. Mr. Baker, who was from the island, was airlifted to noble's hospital, where he later died. (…) He added that investigations are ongoing in relation to the crash. The car driver, who police say was northern irish, was treated in hospital but has been discharged. Another motorcyclist who was injured after the crash has also been released from hospital. (…)
What are the important entities in this document? What are the important dates in this document? What events are happening in this document? What is the result of these events? Please Answer the above questions:
On 4 June, Mr. Baker’s motorcycle collided with a car resulting in his death. The car driver and motorcyclist were injured.
1. The important entities in this document are Mr. Baker, the car driver, and the motorcyclist who was injured. 2. The important dates in this document are 4 June and the present day. 3. The events happening in this document are a collision between Mr. Baker’s motorcycle and a car, and the investigation into the collision. 4. The result of these events is that Mr. Baker died and the car driver and motorcyclist were injured.
Source Document Guiding Questions
for Prompting
LLM
LLM
2

heading: 2.1 Data Construction

heading: 2 Element-aware Summary Test Set
text: is representative of single-document datasets. We use the standard splits (Hermann et al., 2015) for test sets; (ii) BBC XSum (Narayan et al., 2018) provides a highly abstracted news collection. It has one-sentence summaries and is more abstractive than the CNN/DailyMail dataset.
For both datasets, we ask three news experts to independently write professional summaries for 200 randomly sampled source documents according to a complete writing protocol (introduced in §2.2), ensuring comprehensiveness, objectivity, and uniformity of writing style. Different from crowd-sourcing, the involvement of professional writers allows higher inter-annotator agreement. Also, to ensure the uniformity of writing style, we require one of the experts to lead the writing, and the other two to judge the completed summary in four dimensions from the protocol. If there exist inconsistent opinions, they will revise the summary after internal discussion until all pass this annotation. Statistically, the annotation duration of one summary is approximately proportional to the length of source documents. For CNN/DailyMail, a summary is written in 25-30 minutes on average, and for BBC XSum, in 15-20 minutes on average.

heading: 2.2 Writing Protocols
text: Annotators must follow a comprehensive protocol when writing. Specifically, we divide the protocol into micro demands and macro demands. The former emphasizes our targets, namely element awareness, and the latter guarantees the professionalism and objectivity of the overall writing quality, which alleviates the simple stacking of elements. The two demands complement each other.
Micro Demands. All news summaries should have four essential core elements — Entity, Date, Event, and Result — following the “Lasswell Communication Model” (Lasswell, 1948), and these elements must be faithful to the source document. For example, when there is no date in the source document, writers can not add dates to the final summary by force.
Macro Demands. All news summaries should focus on the four dimensions (Gehrmann et al., 2018; Kryscinski et al., 2019). (i) Fluency: No spelling, grammatical, or syntactic errors within sentences; (ii) Coherence: The summary should not be a heap of events, and linguistic transition must be smooth and logically correct; (iii) Consistency: No hallucinated facts — neither facts that do not appear in
Reference Summary
CNN/DaliyMail
% of novel uni/bi/trigram Avg. summary length of words/sentences
Dataset-specific 17.00/53.91/71.98 50.14/3.59 Element-aware 20.31/49.72/62.14 51.08/2.71
Reference Summary
BBC XSum
% of novel uni/bi/trigram Avg. summary length of words/sentences
Dataset-specific 39.39/87.86/96.95 22.18/1.00 Element-aware 36.28/70.56/82.36 23.33/1.00

heading: 2.3 Overall Quality
text: dimensions, respectively, or none if they consider both to be not good.
Figure 3 shows the vote results. It is clear that element-aware summaries are significantly more popularly accepted in both subjective dimensions by the public, demonstrating that our summaries are more human-favored.

heading: 2.4 Element-aware Characteristic
text: In this part, we will demonstrate that our annotated summaries have more obvious element-aware characteristic than the dataset-specific summaries.
We ask three annotators to evaluate every document-summary pair. For each sample, and for i-th annotator (i = 1, 2, 3) and j-th element in the writing protocol (j = 1, 2, 3, 4), we ask this annotator to release two sets that separately contain all j-th elements in the source document they consider important and all j-th elements appearing in the summary. The annotator-released sets for the source document and summary are denoted as Aji and A′i
j , respectively. Then, we compute the Precision and Recall,
they separately reflect the accuracy of the core elements embedded in the summary and the hit rate of the core elements in the source document. Precisionj and Recallj are formulated as:6
6In extreme situations, when Aji is empty, i.e., the annota-
CNN/DailyMail
Core Element
Element-aware Dataset-specific
P R F1 P R F1
Entity 0.98 0.96 0.98 0.75 0.63 0.68 Date 0.89 0.91 0.90 0.74 0.65 0.69 Event 0.96 0.95 0.95 0.66 0.55 0.60 Result 0.95 0.95 0.95 0.49 0.42 0.45
BBC XSum
Core Element
Element-aware Dataset-specific
P R F1 P R F1
Entity 0.97 0.87 0.92 0.76 0.54 0.63 Date 0.97 0.95 0.96 0.52 0.45 0.48 Event 0.93 0.93 0.93 0.80 0.48 0.60 Result 0.96 0.98 0.97 0.23 0.18 0.20

heading: 3 Preliminary Comparison: Zero-shot
text: CNN/DaliyMail
Model Ref Element-aware (ours) Dataset-specific (original)
ROUGE-1 ROUGE-2 ROUGE-L BERTSCORE ROUGE-1 ROUGE-2 ROUGE-L BERTSCORE
BART-BASE 36.06 15.93 33.09 0.8762 38.55 17.57 36.05 0.8779 BART-LARGE 37.98 18.16 34.30 0.8860 39.01 18.26 37.15 0.8868 T5-LARGE 37.47 17.66 34.34 0.8768 38.84 18.39 37.01 0.8802 PEGASUS-LARGE 36.65 17.58 33.84 0.8710 39.11 17.82 36.86 0.8798
175B GPT-3 37.75 15.20 34.25 0.8905 30.10 8.98 27.51 0.8718
BBC XSum
Model Ref Element-aware (ours) Dataset-specific (original)
ROUGE-1 ROUGE-2 ROUGE-L BERTSCORE ROUGE-1 ROUGE-2 ROUGE-L BERTSCORE
BART-BASE 21.89 5.13 17.19 0.8663 29.67 10.09 24.46 0.8779 BART-LARGE 23.79 5.02 17.93 0.8710 33.95 11.29 26.78 0.8880 T5-LARGE 24.98 6.89 19.46 0.8728 30.79 9.61 24.73 0.8792 PEGASUS-LARGE 21.35 4.87 17.03 0.8662 35.16 13.21 29.30 0.8888
175B GPT-3 31.74 10.95 25.42 0.8933 19.99 3.69 15.86 0.8654

heading: Longitudinal Comparison: Language Models.

heading: 3.2 Main Results

heading: 3.1 Experimental Setup
text: BBC XSum. In contrast, GPT-3 obtains dramatic improvements on element-aware test sets. Compared with the performances on dataset-specific test sets, ROUGE-1/2/L increases by +7.65/+6.22/+6.74 points on CNN/DailyMail and +11.75/+7.26/+9.56 points on BBC XSum. These contrasting results demonstrate that our annotated test sets pose a challenge for PLMs fine-tuned with standard datasets, but LLMs can perform well due to their more finegrained writing capabilities.

heading: 3.3 Human Study
text: Human studies are conducted as an overall quality assessment of human preferences. We use a 7-point Likert scale (Likert, 1932) to ask annotators to evaluate four dimensions: Fluency, Coherence, Consistency, and Relevance (equivalent to macro demands in §2.2). Different from baseline-free human studies, we set the element-aware summaries as the baseline (score 0) and set the scoring range to -3~3. A more positive score means higher quality than the element-aware summary and vice versa. For each sample, we present the dataset-specific (original), BART-LARGE, T5-LARGE, PEGASU-LARGE and 175B GPT-3 summaries to the annotators and ask them to score one by one.
As is shown in Figure 4, GPT-3 summaries outperform almost all other dataset-specific or modelgenerated summaries in each dimension, although not yet achieved the level of element-aware summaries. All of these results can fully demonstrate that LLMs have great potential for summarization, and a higher-quality dataset is key for evaluation.

heading: 4 Towards Element-oriented Summary: Chain-of-Thought Method
text: We have analyzed the summary writing ability of zero-shot GPT-3 and other fine-tuned PLMs in §3. We see that GPT-3 performs surprisingly well on our element-aware test sets. The results compellingly show that GPT-3 has great potential for fine-grained zero-shot summary writing. Inspired by the prevalence of the chain-of-thought (CoT) technique in LLMs (Wei et al., 2022b; Kojima et al., 2022; Zhang et al., 2022), we can further enhance the summarization ability of LLMs by leveraging a CoT-based method (SumCoT). SumCoT elicits LLMs to focus on news core elements, thereby generating element-aware summaries step by step. The pipeline and example have been illustrated in Figure 2.

heading: 4.1 Two-stage Pipeline
text: We first ask the LLMs to extract core news elements in the source document by manually-set guiding questions, and later integrate the information based on the extracted elements and more details from the source documents. Pipeline details are as follows.
• Stage 1: Core element extraction. In the first stage, we create guiding-question prompts to elicit the LLMs to extract four core elements: Entity, Date, Event, Result. For the i-th element, we set a simple question qi to guide the model for extracting it (shown in Figure 2), and then concatenate these questions into Q = [q1, q2, q3, q4]. Let the source document be S, then the LLMs input in this stage is formulated as [S;Q].
• Stage 2: Multiple information integration and summarization. We obtain an extraction answer A from the LLMs in Stage 1. Next, we integrate the extracted elements and more detailed information from the source document. We concatenate the source document, questions, answer, and a simple prompt [p′]="Let’s integrate the above information and summarize the article:" to prompt the LLMs for summary generation.7 The input in this stage is formulated as [S;Q;A; [p′]], and the output is the final summary.
7Similarly, for XSum, [p′]="Let’s integrate the above information and summarize the article in one sentence:"

heading: GPT-3 Zero-shot Summary (Standard)

heading: Summarize the above article:
text: The article discusses how Rod Blagojevich, a former Illinois governor, has let his hair go while serving his prison sentence. The photos show the former governor with white hair, rather than the black hair that was his trademark as a politician. Blagojevich is currently serving a 14-year sentence at the Federal Correctional Institution Englewood near Denver.

heading: GPT-3 Zero-shot-cot Summary (Using SumCoT)
text: Let’s integrate the above information and summarize the article: In 2008, Rod Blagojevich was arrested for trying to sell Barack Obama’s U.S. Senate seat. He was impeached and removed from office in 2009, and sentenced to 14 years in prison. He is currently serving his sentence at the Federal Correctional Institution Englewood near Denver. In 2014, his lawyers attempted to appeal his conviction, arguing that a recent Supreme Court ruling supported their view that Blagojevich was engaged in legal, run-of-the-mill political horse trading, not corruption.
Table 4: Case comparisons between GPT-3 zero-shot summaries before and after using SumCoT. Spans of Entity, Date, Event and Result are separately highlighted in red, yellow, blue and green. Prompts are presented in italics.
Model CNN/DaliyMail
ROUGE-1 ROUGE-2 ROUGE-L BERTSCORE
Previous SOTA in Table 3 37.98 18.16 34.34 0.8905 175B GPT-3 37.75 15.20 34.25 0.8905 175B GPT-3 w/ SumCoT 43.03 (↑ 5.05) 19.51 (↑ 1.35) 38.67 (↑ 4.33) 0.9023 (↑ 0.0118)
Model BBC XSum
ROUGE-1 ROUGE-2 ROUGE-L BERTSCORE
Previous SOTA in Table 3 31.74 10.95 25.42 0.8933 175B GPT-3 31.74 10.95 25.42 0.8933 175B GPT-3 w/ SumCoT 35.70 (↑ 3.96) 15.31 (↑ 4.36) 30.19 (↑ 4.77) 0.9018 (↑ 0.0085)
Table 5: Performance comparisons upon element-aware test sets of our method (GPT-3 with SumCoT), standard GPT-3, and previous state-of-the-art (SOTA) results in Table 3 over each metric. The ↑ and corresponding numbers on the right of each result of our method represent the increase after comparing with the previous SOTA.

heading: 4.2 Comprehensive Evaluation
text: First, we visually compare the quality of summaries generated by GPT-3 before and after using SumCoT. As shown in Table 4, it is clear that the summary generated under SumCoT contains more abundant fine-grained elements, saturating the summary text with more key information.
Next, we perform quantitative evaluations over the same metrics as in §3.1. We mainly compare our method (GPT-3 with SumCoT), standard GPT-3, and previous state-of-the-art (SOTA) results in Table 3, and updated results are shown in Table 5. Compared with the standard GPT-3
and previous SOTA, GPT-3 with SumCoT obtains salient improvement in all metrics when compared with the element-aware summaries, where ROUGE1/2/L increases by +5.05/+1.35/+4.33 points on CNN/DailyMail and +3.96/+4.36/+4.77 points on BBC XSum, demonstrating that GPT-3 successfully focuses on more core elements through SumCoT and further fits the element-aware writing pattern.
Finally, we also conduct human studies to compare summaries of GPT-3 w/o and w/ SumCoT. Results (as shown in Table 6) indicate that the SumCoT technique further improves the performance of the standard zero-shot paradigm in all dimensions, particularly coherence and relevance.

heading: 4.3 Better Understanding SumCoT
text: How does SumCoT affect summary writing? First, we explore the extent to which SumCoT affects the final summary generation. We compute the coverage, the fraction of extracted elements in Stage 1 actually appearing in the final summary generated in Stage 2. Table 7 shows the results (see Appendix C.1 for examples), and we observe
CNN/DailyMail BBC XSum
Entity Date Event Result Entity Date Event Result 0.89 0.55 0.93 0.95 0.80 0.48 0.87 0.66
Does the model size limit SumCoT? We compare the performance of GPT-3 with different versions of element extraction. We compute the F1 score (shown in Figure 5) for all the elements. We find that when the model size is small, element extraction is almost invalid. As the model size increases, GPT-3 can extract one by one for all types of elements, but the extraction itself has many errors or redundancies. Only when the model size is the largest, the element extraction is humanapproved (See Appendix C.3 for examples). This indicates that the SumCoT technique is also an emergent ability of model scale (Wei et al., 2022a), and is effective only when the model size is larger.

heading: 5 Related Work and Discussion

heading: 5.1 Summarization: Dataset and Evaluation
text: In the data-driven deep learning era, large-scale corpus crawled from websites for summarization is rich, especially the news domain. They can be divided into the single-document setting (Harman and Over, 2004; Sandhaus, 2008; Napoles et al., 2012; Nallapati et al., 2016; Narayan et al., 2018; Koupaee and Wang, 2018; Grusky et al., 2018) and the multi-document setting (Owczarzak and Dang, 2011; Li et al., 2017; Fabbri et al., 2019) according to the source numbers of document clusters. However, some studies pointed out various noises within them, such as poor coherence, information redundancy, and factual hallucination (Kryscinski et al., 2019; Maynez et al., 2020; Fabbri et al., 2021). Several other studies also corroborated this with human assessments (Stiennon et al., 2020; Fabbri et al., 2021).
Summarization systems are first purely trained (Vinyals et al., 2015; Vaswani et al., 2017; Liu et al., 2022b; Chen et al., 2022) or fine-tuned (Zhang et al., 2019; Liu, 2019; Zhang et al., 2020a; Raffel
et al., 2020; Wang et al., 2022b; Mao et al., 2022) with standard datasets, and then evaluated. The most mainstream automatic evaluation metrics for summarization are reference-based methods, i.e., directly comparing the similarity of generated and dataset-specific summaries. They can be split into lexical overlap methods (Papineni et al., 2002; Lin, 2004; Banerjee and Lavie, 2005) and semantic similarity methods (Ng and Abrecht, 2015; Zhang et al., 2020b; Zhao et al., 2019; Sellam et al., 2020; Rei et al., 2020). Such evaluation is essentially a test of the fit degree to standard datasets. In recent years, the advanced zero-shot paradigm of LLMs makes text generation free of standard datasets (Brown et al., 2020; Chowdhery et al., 2022; Thoppilan et al., 2022) but rely on massive pre-trained data, many researchers tend to revisit the quality assessment of summaries generated by LLMs (Liu et al., 2022a; Zhang et al., 2023a). However, some studies demonstrate that automatic evaluation results do not align with human preference in summarization tasks (Goyal et al., 2022), similar counter-intuitive observations may pose new challenges for the evaluation in the era of LLMs.

heading: 5.2 Chain-of-Thought Prompting for LLMs
text: Recently, intriguing chain-of-thought techniques have greatly improved both the reasoning performance and interpretability of LLMs by decomposing multi-step problems into intermediate steps (Nye et al., 2022; Wei et al., 2022b; Kojima et al., 2022; Zhang et al., 2022; Wang et al., 2022a; Zhang et al., 2023b; Shi et al., 2022; Zhou et al., 2022). However, no prior work has studied CoT in the scenario of automatic summarization. To the best of our knowledge, we are the first to study chainof-thought prompting for summarization, eliciting LLMs to leverage more fine-grained elements from source documents to generate effective summaries.

heading: 6 Conclusion
text: In this work, we construct expert-writing elementaware summary test sets for CNN/DailyMail and BBC XSum, they are specifically designed to assess the generic summarization capabilities of diverse, powerful language models more thoroughly. Upon the fine-grained test sets, we preliminarily conduct experiments on zero-shot LLMs and finetuned PLMs, demonstrating the surprising zeroshot summary writing ability of LLMs. Further, we propose a CoT-based method, which elicits LLMs
to focus on core news elements and generate summaries step by step. In the future, we hope that our work will inspire further research into harnessing LLMs’ potential to mimic human writing processes across various open-ended generative tasks.

heading: Limitations
text: In terms of the test sets, due to time, labor, and financial limitations, we are unable to construct large-scale test sets of the same size as the original, so the domain balance in the test sets is not fully considered, but the uniformity of writing style might have slightly alleviated this issue. In terms of the method, we empirically explore the possibility of chain-of-thought application in text generation. However, due to the stronger openness of generative tasks compared to pure reasoning tasks, generated summaries might be more sensitive to the form of chain-of-thought, which is a key point worth further optimization.

heading: Ethics Statement
text: We use publicly available source documents from existing general datasets for annotations, so the ethics issues of the source texts are non-existent. For the generated contents with LLMs, e.g. GPT3, prior work (Brown et al., 2020; Chan, 2022) has elaborated on their inevitable potential toxicity, such as issues of bias and fairness. Moreover, this is the first work to apply the chain-of-thought technique to open-end generation tasks, so we completely keep the prompts neutral and task-specific to avoid toxic language generation, and there were no toxic texts that appeared in our experiments.

heading: Acknowledgements
text: Yiming and Rui are with MT-Lab, Department of Computer Science and Engineering, School of Electronic Information and Electrical Engineering, and also with the MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200204, China. Rui is supported by the General Program of National Natural Science Foundation of China (6217020129), Shanghai Pujiang Program (21PJ1406800), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), Beijing Academy of Artificial Intelligence (BAAI) (No. 4), CCF-Baidu Open Fund (F2022018), and the Alibaba-AIR Program (22088682). We also thank the computational resource from the SJTU student innovation center.

heading: A Details of Experimental Setup

heading: A.1 Main Experiment
text: Table 9 report the sources and licenses of artifacts and packages we used in this paper.

heading: A.2 Human Study
text: We randomly select 50 samples for each dataset and ask three annotators for these tasks following the setting of most human studies. However, considering the unprofessionalism of crowd-sourcing evaluations (Usually hiring workers from Amazon Mechanical Turk platform with a set hourly salary. Actually, many workers will not work as you expected, their levels vary widely and uncontrollably. He et al. (2020) have encountered such a situation), we privately contact three reliable annotators to conduct the human studies. The first is a Ph.D. candidate in Computer Science, the second is a Master in Film Study, and the last is a graduate in Journalism and Communication. Our human studies are conducted in full compliance with the willingness of the invitees and are fully open about the use of the data they annotated. They have been paid slightly more than the crowd-sourced hourly rate for their work. We use the same configuration for all human studies in this paper, thanks for their participation!

heading: B Abstraction and Faithfulness analysis for Summaries
text: Abstraction and Faithfulness are normally two opposing properties. For dataset-specific summaries, despite their novel n-grams being higher than element-aware summaries in many cases, they sacrifice factual correctness to some extent, which is a fake high-abstraction. Case comparisons are shown in Table 10-11.

heading: C Better Understanding Summary Chain-of-Thought: Case Study

heading: C.1 Learn How SumCoT Works
text: Table 12-14 presents some cases to visually show how SumCoT affects the final generated summary. We compare GPT-3 zero-shot summaries before and after using SumCoT. Core elements have been highlighted in the table. It is clear that summaries using SumCoT Cover a large number of fine-grained elements extracted by GPT-3 in Stage 1 that are not in the standard zero-shot summaries.
Model URL License
BART (Lewis et al., 2020)
https://huggingface.co/ainize/bart-base-cnn MIT licensehttps://huggingface.co/morenolq/bart-base-xsum https://huggingface.co/facebook/bart-large-cnn https://huggingface.co/facebook/bart-large-xsum
PEGASUS (Zhang et al., 2020a) https://huggingface.co/google/pegasus-cnn_dailymail Apache-2.0 license https://huggingface.co/google/pegasus-xsum
GPT-3 (Zhang et al., 2020a) https://openai.com/api/ N/A
Evaluation Metric URL License
ROUGE (Lin, 2004) https://github.com/pltrdy/rouge Apache-2.0 license
BERTSCORE (Zhang et al., 2020b) https://github.com/Tiiiger/bert_score MIT license

heading: D Random Sample Presentation

heading: C.3 Ablation Study of GPT-3 Model Size

heading: C.2 Error Analysis for Element Extractions
text: Source Document (CNN/DailyMail)
(The Hollywood Reporter) Add another fan-favorite character to the cast of next year’s "X-Men: Apocalypse" with director Bryan Singer announcing via Instagram that Olivia Munn will play the telepathic Psylocke in the follow-up to "X-Men: Days of Future Past." Singer revealed that the "Newsroom" actress would play Betsy Braddock in the movie (presumably before the confusing and complicated plot twist that saw Psylocke change from a Caucasian former supermodel to a Japanese ninja for no immediately obvious reason). Äpocalypseïs currently in production for a summer 2016 release. More: "X-Men: Apocalypse" casts fan favorite Jubilee. The comic book’s Psylocke was created by Chris Claremont and Herb Trimpe for the British "Captain Britain" series, where she appeared throughout the 1970s and ’80s, before joining the X-Men in 1987’s "Uncanny X-Men" No. 213. Since that time, she has been a mainstay both of the main team and spin-off series including "Exiles" and "X-Force." More: What newcomers need to know about Marvel’s "Secret Wars". Munn will join a cast that includes James McAvoy, Michael Fassbender and Jennifer Lawrence in the movie, which hits theaters May 27, 2016. Munn is repped by Creative Artists Agency and Atlas Artists. More: Does the big plot twist in "Terminator Genisys" blow up the franchise? @The Hollywood Reporter. All rights reserved."

heading: Dataset-specific Summary Element-aware Summary
text: Olivia Munn will play Psylocke in "x-men: apocalypse" film. Psylocke trended for hours on twitter after director Bryan Singer announced casting. Olivia Munn will play the telepathic Psylocke -created by Chris Claremont and Herb Trimpe for the C̈aptain Britains̈eries - in the Ẍ-Men: Apocalypse.̈ The movie will be released in May 27, 2016.
Source Document (BBC XSum)
More than 350 roma people had lived in the camp on la petite ceinture since mid-2015. Activists said many left early ahead of the police action. The site belongs to the national rail authority sncf. France has one of europe’s toughest policies towards roma. Most live in camps that are regularly demolished and every year thousands are deported. Amnesty international urged city authorities to find a lasting housing solution for those evicted in paris - saying they would become homeless in mid-winter. Hundreds of thousands of roma - mostly from romania and bulgaria - have moved to western europe since the 1990s. The council of europe, the region ’s main human rights body, warned that evictions were " counter-productive" because they disrupted education and healthcare for roma children. Council of europe secretary general thorbjorn jagland said it was crucial for french authorities to provide "adequate alternative accommodation" for those evicted, particularly as they have decided to take this action during winter.

heading: Dataset-specific Summary Element-aware Summary
text: Police have cleared hundreds of roma people from a slumlike camp built on a disused rail line in north paris. Every year thousands of Roma people are deported by France, and the region’s main human rights body urges France to provide alternative accommodation for those evicted.
Source Document (CNN/DailyMail)
Once famed for his mop of blacker than black hair, disgraced Democrat Rod Blagojevich, 58, has really let his haircare regime go while he serves his prison time. The former Illinois governor has return to his roots while inside and has been photographed with his still full head of hair a shocking white color rather than the boot polish black that was his trademark as a politician. Blagojevich was infamously caught trying to sell Barack Obama’s U.S. Senate seat when he was elected president in 2008. Fade to gray: Once famed for his mop of blacker than black hair, disgraced Democrat Rod Blagojevich, 58, has really let his haircare regime go while he serves his prison time. Back in his days as governor of Illinois, Blagojevich was famed for his boot polish black hair. He was impeached and removed from office by the state Legislature in early 2009 following his arrest on federal corruption charges. Blagojevich is currently serving a 14-year sentence at the Federal Correctional Institution Englewood near Denver. The photos, the first of Blagojevich in prison, were first published by the National Enquirer. The photos show the former governor sitting at a desk, reading a book and wearing glasses and sitting outside in shorts and a wife beater shirt. While he continues to awaiting word on a last-ditch appeal, Blagojevich has daily contact with his family and teaches the history of war battles to other inmates, his attorney said. Last July lawyers attempted to argue that a high court ruling in McCutcheon v. the Federal Election Commission in April 2014 bolstered their view that Blagojevich was engaged in legal, run-of-the-mill political horse trading, not corruption. The McCutcheon ruling, they argued, found that soliciting contributions crosses the line into corruption only when a politician makes a clear, explicit promise to take official action in return for a donation. Blagojevich is currently serving a 14-year sentence at the Federal Correctional Institution Englewood near Denver after he was caught trying to sell Barack Obama’s U.S. Senate seat when he was elected president. In that context, defense attorneys said the trial judge was wrong to set a lower standard for jurors by telling them, to convict, they only needed to find that Blagojevich sought donations ’knowing or believing’ it would be given for some official act. But government attorneys responded saying that Blagojevich’s legal team hadn’t correctly characterized the Supreme Court’s findings in McCutcheon. ’The decision provides no support for Blagojevich’s argument on appeal,’ the two-page filing said. ’Nothing in the (McCutcheon) decision suggests that an exchange of contributions for specific official acts is quid pro quo corruption only if the arrangement is stated ’explicitly’ or ’expressly.’ The Chicago-based appellate court has been considering Blagojevich’s appeal for more than a year months. It’s unclear when it might issue a decision.

heading: GPT-3 Zero-shot Summary (Standard)

heading: Summarize the above article:
text: The article discusses how Rod Blagojevich, a former Illinois governor, has let his hair go while serving his prison sentence. The photos show the former governor with white hair, rather than the black hair that was his trademark as a politician. Blagojevich is currently serving a 14-year sentence at the Federal Correctional Institution Englewood near Denver.

heading: Summary Chain-of-Thought (Stage 1, Element Extraction)
text: What are the important entities in this document? What are the important dates in this document? What events are happening in this document? What is the result of these events? Please answer the above questions: 1. The important entities in this document are Rod Blagojevich, the Federal Correctional Institution Englewood, and the National Enquirer. 2. The important dates in this document are 2008, 2009, and 2014. 3. The events in this document are Blagojevich’s arrest and imprisonment, and his appeal of his conviction. 4. The result of these events is that Blagojevich is currently serving a 14-year sentence in prison.

heading: GPT-3 Zero-shot-cot Summary (Stage 2, Summary Generation)
text: Let’s integrate the above information and summarize the article: In 2008, Rod Blagojevich was arrested for trying to sell Barack Obama’s U.S. Senate seat. He was impeached and removed from office in 2009, and sentenced to 14 years in prison. He is currently serving his sentence at the Federal Correctional Institution Englewood near Denver. In 2014, his lawyers attempted to appeal his conviction, arguing that a recent Supreme Court ruling supported their view that Blagojevich was engaged in legal, run-of-the-mill political horse trading, not corruption.
Table 12: More comparisons between GPT-3 zero-shot summaries before and after using SumCoT. Spans of Entity, Date, Event, and Result are separately highlighted in red, yellow, blue and green. Prompts are presented in italics (Appendix C.1).
Source Document (BBC XSum)
The 69-year-old’s yamaha collided with a Nissan car between handley’s corner and barre garroo crossroads at about 17:00 bst on 4 June. Mr. Baker, who was from the island, was airlifted to noble’s hospital, where he later died. Insp dave dobbie said "thoughts and condolences are with Mr. baker’s family". He added that investigations are ongoing in relation to the crash. The car driver, who police say was northern irish, was treated in hospital but has been discharged. Another motorcyclist who was injured after the crash has also been released from hospital. The collision happened outside of the TT racing, which is taking place on the island.

heading: GPT-3 Zero-shot Summary (Standard)
text: Summarize the above article in one sentence: The article discusses a fatal motorcycle collision that occurred on the Isle of Man, and the investigation that is ongoing.

heading: Summary Chain-of-Thought (Stage 1, Element Extraction)
text: What are important entities in this document? What are important dates in this document? What events are happening in this document? What is the result of these events? Please answer the above questions: 1. The important entities in this document are Mr. Baker, the car driver, and the motorcyclist who was injured. 2. The important dates in this document are 4 June and the present day. 3. The events happening in this document are a collision between Mr. Baker’s motorcycle and a car, and the investigation into the collision. 4. The result of these events is that Mr. Baker died and the car driver and motorcyclist were injured.

heading: GPT-3 Zero-shot-cot Summary (Stage 2, Summary Generation)
text: Let’s integrate the above information and summarize the article in one sentence: On 4 June, Mr. Baker’s motorcycle collided with a car, resulting in his death. The car driver and motorcyclist were injured.
Table 13: More comparisons between GPT-3 zero-shot summaries before and after using SumCoT. Spans of Entity, Date, Event, and Result are separately highlighted in red, yellow, blue and green. Prompts are presented in italics (Appendix C.1).
Source Document (BBC XSum)
West midlands police said the man had been detained in reading and the woman in wokingham by counter-terrorism officers at about 19:30 bst. The pair, believed to be friends, were taken to a police station in the west midlands for questioning. The force said the arrests had been intelligence-led. the suspects did not pose any imminent threat, a spokesperson added. Supt robin rickard from thames valley police said the arrests "were part of an ongoing investigation by west midlands counter terrorism unit and were pre-planned". "Officers from reading and wokingham are providing support and there are searches taking place at the suspects’ home addresses," he said.

heading: GPT-3 Zero-shot Summary (Standard)
text: Summarize the above article in one sentence: Two people were arrested in England on suspicion of terrorism.

heading: Summary Chain-of-Thought (Stage 1, Element Extraction)
text: What are the important entities in this document? What are the important dates in this document? What events are happening in this document? What is the result of these events? Please answer the above questions: The important entities in this document are the two suspects who were arrested, the police forces who made the arrests, and the locations where the arrests took place. The important dates are the day of the arrests (19:30 BST) and the day of the document (20:00 BST). The events happening in this document are the arrests of the two suspects and the searches of their home addresses. The result of these events is that the two suspects are in police custody and their home addresses are being searched.

heading: GPT-3 Zero-shot-cot Summary (Stage 2, Summary Generation)
text: Let’s integrate the above information and summarize the article in one sentence: On 19:30 BST, two suspects were arrested by police forces in Reading and Wokingham on intelligence-led grounds, and their home addresses are currently being searched.
Table 14: More comparisons between GPT-3 zero-shot summaries before and after using SumCoT. Spans of Entity, Date, Event, and Result are separately highlighted in red, yellow, blue and green. Prompts are presented in italics (Appendix C.1).

heading: Error Type: Date Hallucination
text: Source Document (CNN/DailyMail)
Charity runners taking part in a 10km fun run at the weekend were left exhausted after being sent on an unscheduled two-mile detour. The blunder was believed to have been caused by a race marshal taking a toilet break during the event, missing 300 runners who should have been directed at a junction point. Instead they continued past the unmanned marshall point and had to run for an extra three kilometres while the other 900 competitors followed the correct route. Scroll down for video Blunder: Charity runners taking part in yesterday’s Bournemouth Bay 10K Run (pictured) were left exhausted after being sent on an unscheduled two-mile detour. The bizarre gaffe happened during yesterday’s Bournemouth Bay Run and today the organisers - Bournemouth Borough Council - appealed for those who were affected by the mix-up to contact them for a ’gesture of goodwill.’A local authority spokesman said that it was investigating what happened to the marshal who should have directed runners at a turning point. It was reported that some runners were ’in tears’ while one described the event’s organisation as ’shambolic’. Hayley James, who is four months pregnant and from Poole, said: ’To have a race of that scale with only one marshal on a point is inexcusable.’We saw loads of people walking at the end, some were in tears, I felt so sorry for them - I felt like crying at the 10km mark.’Andy Isaac, from Bournemouth, said the event was ’mayhem’ with one point where an elderly woman managed to drive onto the route and was flashing her lights at oncoming runners. A map shows where up to 300 runners continued along the coastal path after a marshal who was meant to direct them on to a turn went to the toilet. Reaction: Two people vent their frustration at the Bournemouth Bay Run on Twitter yesterday. It also emerged that water stations ran out of supplies during the race, forcing some runners to drink from half-empty bottles that had been left on the ground as they battled against dehydration.Commenting on the Daily Echo website, one runner said: "We had a bottle of water at the three mile station, but at the six mile point they had totally ran out, so nothing. Jon Weaver, head of resort marketing and events at the council, said: ’Unfortunately there was some confusion with marshalling arrangements at one point, but it was a critical point. We apologise unreservedly to those front runners.’In 33 years of running... this is the first time this has happened and as part of our debrief we will be analysing the arrangements carefully... to learn for 2016.’We understand runners have trained for a long time for the event and it’s hard for them and we do empathise with how they are feeling.’It was hoped that the event would have raised more than £70,000 for the British Heart Foundation. Some racers took to Twitter to vent their frustration over the blunder. Rob Kelly wrote: Really disappointed in the #BournemouthBayRun 10k that ended up 13k very poor show bad marshalling #wontbeback.’ And Kim Kelly replied: ’Totally agree and never got to do a 5k as they were 45mins behind schedule :(((.’ What are the important entities in this document? What are the important dates in this document? What events are happening in this document? What is the result of these events? Please answer the above questions:

heading: Element Extraction
text: 1. The important entities in this document are the charity runners, the race marshal, and the Bournemouth Borough Council. 2. The important dates in this document are the date of the race (May 3, 2015) and the date the runners were supposed to be directed at the turning point (May 2, 2015). 3. The events happening in this document are the charity runners being sent on an unscheduled two-mile detour, and the race marshal taking a toilet break. 4. The result of these events is that the charity runners were left exhausted after being sent on an unscheduled two-mile detour.
% (The dates highlighted in blue are not declared in the source document.)
Table 15: Element extraction errors of GPT-3. Prompts are presented in italics (Appendix C.2).

heading: Error Type: Date Hallucination
text: Source Document (BBC XSum)
patrick Bauer and Nathan Byrne helped Karl Robinson’s men complete a perfect smash and grab, cancelling out Zach clough’s free-kick. a victory which furthers the Addicks’ own play-off ambitions was made all the more impressive considering that they were a man short from the 12th minute onwards following lewis page’s red card. after page was sent off for a professional foul on josh vela, clough then made the punishment count double with a brilliant free-kick, his ninth goal of the season. but robinson ’s side refused to lie down, equalising on 23 minutes when jake forster-caskey ’s free-kick was turned in at the far post by defender bauer. the Addicks then grabbed a shock lead in first-half stoppage time when Forster-Caskey led an impressive counter, feeding Byrne to slot the ball under the advancing Ben Alnwick. bolton, now without a win in their last five games in all competitions, threw all they had at Charlton after that and struck the woodwork when Andrew Taylor’s cross deflected off Chris Solly. they failed to seriously test goalkeeper Declan rudd in the second half, however, as they slipped to a second consecutive 2-1 defeat in front of their own fans. Charlton, unbeaten since christmas, have now picked up 11 points from their last five matches to climb to 11th, just six points off a play-off place. report supplied by the press association. What are the important entities in this document? What are the important dates in this document? What events are happening in this document? What is the result of these events? Please answer the above questions:

heading: Element Extraction
text: 1. The important entities in this document are Charlton Athletic, Karl Robinson, Zach Clough, and Ben Alnwick. 2. The important dates in this document are December 25th and January 23rd. 3. The events happening in this document are a football match between Charlton Athletic and Bolton Wanderers, and the aftermath of the match. 4. The result of these events is that Charlton Athletic won the match 2-1.
% (The dates highlighted in blue are not declared in the source document.)

heading: Error Type: Date Hallucination
text: Source Document (BBC XSum)
club director Emilio butragueno confirmed FIFA had asked the Spanish federation for documents relating to transfers over the last five years. but ragueno said the club was “ absolutely calm ” about its behaviour. fifa handed Barcelona a transfer ban until 2016 after they breached regulations when signing 10 players under the age of 18 in 2013 . in a statement, real Madrid said the majority of the 51 players that world football’s governing body had asked for information on were exempt from regulations for various reasons. FIFA rules state that international transfers are only permitted for players who are over the age of 18 - unless the player in question meets one of three qualifying criteria. under-18s can move to a club in a different country if: real claim 10 of the 51 players were Spanish nationals, while 23 foreign players had already been registered with a Spanish club. on Thursday, real completed the # 2.3 m signing of 16-year-old Norway midfielder Martin Odegaard from stromgodset. barcelona are not allowed to sign any players in 2015 after their appeal against their transfer ban was dismissed in December. What are the important entities in this document? What are the important dates in this document? What events are happening in this document? What is the result of these events? Please answer the above questions:

heading: Element Extraction
text: 1. The important entities in this document are FIFA, Barcelona, and Real Madrid. 2. The important dates in this document are 2013, when Barcelona was first given a transfer ban, and December of 2014, when Barcelona’s appeal against the ban was denied. 3. The events in this document are FIFA’s investigation into Barcelona’s transfer practices, and the eventual imposition of a transfer ban on the club. 4. The result of these events is that Barcelona is not allowed to sign any players in 2015.
% (The date when Barcelona’s appeal against the ban was denied is likely to be December 2013 or December 2014 deduced from the bolded part of the source document, so the extracted date highlighted in blue is evidenceinsufficient.)
Table 17: Element extraction errors of GPT-3. Prompts are presented in italics (Appendix C.2).

heading: Error Type: Element Redundancy
text: Source Document (BBC XSum)
there were concerns about the stability of the building, which has been derelict for years, after the collapse of an internal wall last week. contractors began work on Friday night. the environment minister Alex Atwood recommended the demolition in the interests of public safety. mary o’dwyer, chair of Foyle civic trust, said a strategic approach is needed for the future protection of heritage in the city following the demolition . “ many of our members believe that this is a very sad event . “ the factory is part of our industrial heritage and more should have been done to save it . “ others believe something had to be done because it was a health and safety issue, ” she said . “ we would like to see a consensus and a strategic vision of all of the organizations in the city that have a remit for protecting our heritage. ” Foyle road, which was closed between the lower deck of the Craigavon bridge and john street roundabout, has reopened. What are the important entities in this document? What are the important dates in this document? What events are happening in this document? What is the result of these events? Please answer the above questions:

heading: Element Extraction
text: The important entities in this document are the Foyle Civic Trust, the environment minister Alex Atwood, and the contractors who began work on Friday night. The important dates in this document are the collapse of an internal wall last week, and the demolition of the factory on Friday night. The events happening in this document are the demolition of the factory, and the reopening of Foyle Road. The result of these events is that the factory is gone and Foyle Road is open again.
% (The elements highlighted in blue is redundent.)

heading: Error Type: Element Redundancy
text: Source Document (BBC XSum)
some of those residents had only just returned to their homes after last week’s eruptions. the latest eruption was less powerful but sent a large plume of dark grey smoke and ash rising from the crater. calbuco surprised residents of the Los Lagos region last week by bursting into life after decades of inactivity. the national geology and mines service issued a red alert after Thursday’s eruption, warning that the volcano remained unstable. on Friday, residents were warned that heavy rain could mix with ash and rock to produce deadly volcanic mudflows. these “ lahars ” can slide down volcanoes like avalanches. the BBC’s Gideon long in Santiago says that whereas the ash last week was blown in a north-easterly direction, this time it is drifting south-east over the town of cochamo. the town has been blanketed in a cloud of smoke and ash. calbuco is one of the most active volcanoes in chile, having erupted in 1961 and 1972, and many times before that. last week’s eruptions caused the cancellation of flights to and within chile and forced farmers in the area to evacuate their livestock. What are the important entities in this document? What are the important dates in this document? What events are happening in this document? What is the result of these events? Please answer the above questions:

heading: Element Extraction
text: 1. The important entities in this document are the volcano, the residents, and the government. 2. The important dates in this document are the dates of the eruptions. 3. The events happening in this document are the eruptions of the volcano and the resulting evacuations. 4. The result of these events is that the residents have been forced to leave their homes and the area around the volcano is covered in ash.
% (The elements highlighted in blue is redundent.)
Table 19: Element extraction errors of GPT-3. Prompts are presented in italics (Appendix C.2).
Source Document (CNN/DailyMail)
A Tory councillor with a history of odd behaviour was told to put down his knife and fork after being caught tucking into a roast duck dinner during a council meeting. Jonny Bucknell, 58, was enjoying his meal in the council chamber when a Labour rival, Theo Blackwell, spotted him and alerted other councillors. He was forced to put down his cutlery when the mayor, Lazzaro Pietragnoli, interrupted the proceedings to tell him off. Taking a stand: Jonny Bucknell is no stranger to odd behaviour. In 2013 he slept in his car at the Tory party conference. He now says he wants a rule change so he can eat a roast dinner at council meetings. The mayor, who was chairing the meeting of Camden Council in north London, reminded the hungry councillor that eating was banned in the chamber. But the angry diner claims he was unaware eating there was forbidden and said he now aims to campaign for a rule change. The rumpus comes a month after Liberal Democrat councillor Martin Elengorn was caught playing Scrabble during a Richmond Council budget meeting in south-west London. Telling off: Mayor of Camden Council, Lazzaro Pietragnoli, had to tell Mr Bucknell to stop eating. When he first noticed him eating, Mr Blackwell told his fellow councillors: ’It appears that one of our Tory colleagues is consuming a full Sunday roast dinner in the council chamber. ’Could I ask the borough solicitor to give us advice on eating a full roast dinner in the council chamber? It’s a little bit more than a cheeky Snickers.’ The diner was forced to curtail his meal. Mr Bucknell, who has been a councillor for more than ten years and represents Belsize, Hampstead, told the Evening Standard: ’I never knew there was a ban on eating in the chamber. ’They should definitely repeal it. There is nothing wrong with nibbling from a lunch box if you are being discreet. ’It is not as if a cold meal is going to waft around like a McDonald’s. ’I will be campaigning for the repealing of the law that says you can’t nibble from a lunch box in the council chamber.’ The Conservative councillor said the meal, in a plastic box, had travelled home with him after a French snowboarding holiday. ’The chalet always brought out too much food and I can’t stand wasting food,’ he said. He previously found fame when he slept in his Volvo car to save on hotel bills during the Conservative party conference in September 2013. Mr Bucknell said at the time it was to make a stand against what he called ’ridiculous prices’. He said the economy would improve if more people were thrifty like him. After the council meeting mayor Lazzaro Pietragnoli said: ’I understand councillors do a difficult job and sometimes don’t get time to eat dinner. ’I also tend to be quite flexible, but having a big meal in front of him - that was a bit too much.’ A Camden Council spokesman said: ’It is as the mayor said, standing orders say that members should not eat in the chamber.’ No eating: The Camden Council chamber where Councillor Bucknell was forced to curtail his roast dinner.

heading: Dataset-specifc Summary
text: Labour rival alerted colleagues after spotting Councillor tucking into roast. It is not the first time jonny Bucknell, 58, has demonstrated odd behaviour.. In 2013 he slept in his car to make a point while attending Tory conference. Mr Bucknell said he will campaign for rule change about eating at meetings.

heading: Element-aware Summary
text: Jonny Bucknell, a Tory councillor, was found tucking into a roast duck dinner during a council meeting. He now wants a rule change to have dinner at council meetings. It is not the first time for his strange behavior. In September 2013, Mr Bucknell slept in his Volvo car to save on hotel bills during the Conservative party conference to make a stand against what he called ’ridiculous prices’.

heading: GPT-3 Zero-shot Summary
text: A Tory councillor was told to stop eating a roast duck dinner during a council meeting after a Labour rival spotted him and alerted other councillors. The councillor, Jonny Bucknell, claims he was unaware that eating in the council chamber was forbidden and said he now aims to campaign for a rule change.

heading: GPT-3 Zero-shot-cot Summary
text: In September 2013, Councillor Jonny Bucknell made headlines when he slept in his car during the Conservative party conference in order to save on hotel bills. Recently, he has found himself in the news again after being caught eating a roast duck dinner during a council meeting. Councillor Bucknell claims that he was unaware that eating in the council chamber was forbidden, and is now campaigning for a rule change so that he can eat during meetings.
Table 21: Random samples from CNN/DailyMail and BBC XSum datasets (Appendix D).
Source Document (CNN/DailyMail)
This is the dramatic moments armed police swoop on a villa where a Briton linked to the gangland murder of a torture victim was arrested. Paul Monk, 54, from Essex, was wanted by Spanish police for questioning over the kidnap and murder of Francis Brennan, whose badly decomposed body washed up on a Costa Blanca beach in March last year. He was also wanted by the Metropolitan Police on drug offences and had been named on a list of fugitives published as part of the National Crime Agency’s Operation Captura campaign ahead of his detention. This is the dramatic moment that fugitive Paul Monk was arrested by heavily armed police in his Alicante villa. Paul Monk, 54, from Essex, was wanted by Spanish police for questioning over the kidnap and murder of Francis Brennan. Spanish police released footage of their dramatic swoop. This grab for the video shows them approaching the villa at speed. The police move steathily up the steps of Monk’s villa, weapons drawn. Taking no chances: The highly trained, well-armed police moved through the house room by room. Paul Monk was on the UK’s most wanted list on suspicion of drug trafficking. Brennan, 25, from Liverpool, vanished in the resort of Javea in January last year after being kidnapped by men posing as police. His body was wrapped in an industrial-size bin bag with duct tape round it when it appeared on a beach in nearby Orihuela Costa. Civil Guard officers in Alicante confirmed today they believe Monk, from Essex, may be implicated in the violent death and named him as an associate of Paul Scott. Scott, 32, was arrested on a charge of conspiracy to import cocaine after being caught trying to sneak into Britain in a light aircraft last December. He was also wanted for questioning over Mr Brennan’s murder when he was detained. Guardia Civil described him last night as the suspected mastermind of the crime. Monk was detained at a four-bedroom property in Javea near Alicante as he directed workers laying a marble patio around his swimming pool. An imitation firearm with a silencer and nearly 0̆0a3100,000 in cash were also found. He is being held in jail and is expected to be charged and face trial in Spain over Mr Brennan’s murder before being extradited to the UK to face questioning over alleged drugs offences. He has been linked to the handover of one kilo of cocaine in Cockfosters, London, in May 2013 and the seizure of 24 kilos of cannabis in Colchester in October 2013. A Civil Guard spokesman said: ’He never left his house as a security measure to avoid being arrested. ’He got other people to bring him food and other things in the villa where he hid out, leading the life of an authentic fugitive.’ The police raid had air support, with this grab coming from footage of Monk’s villa taken by a helicopter. Wads of money found by armed police after they arrested Monk . Monk is being held in jail and is expected to be charged and face trial in Spain over Mr Brennan’s murder before being extradited to the UK to face questioning over alleged drugs offences. Spanish police search Monk’s property thoroughly for evidence, finding an imitation gun with a silencer. National Crime Agency detectives took part in the raid on the property along with Civil Guard officers. Mr Brennan, from Liverpool, was himself on the run after leaving Britain while awaiting sentence in the UK over the wounding of a man at a Swedish House Mafia music concert in Milton Keynes, Bedfordshire. He travelled to the Costa del Sol with his girlfriend towards the end of 2013 after ignoring the advice of his parents to stay in Britain. Police swooped on Scott last year after he flew from Holland to a remote airstrip in Norfolk. He had fake Belgian identity documents on him and more than 0̆0a31,500 in cash. He was jailed in January for plotting to import 40 tonnes of cocaine into Britain. Police described him as the ’last man outstanding in a criminal gang that plotted to smuggle vast quantities of cocaine into the UK’ after he was sentenced to 14 years in prison at Liverpool Crown Court. Sorry we are not currently accepting comments on this article.

heading: Dataset-specific Summary
text: Paul Monk, 54, was wanted by Spanish Police in connection with a murder.. The Essex man is a suspect in the murder of Francis Brennan.. Brennan’s body washed up on a Costa Blanca beach in March last year.. Police released footage of their swoop on Monk’s Alicante villa.

heading: Element-aware Summary
text: Paul Monk, 54, from Essex, was wanted by Spanish police for questioning over the kidnap and murder of Francis Brennan, 25, from Liverpool. Brennan2̆019 body was found on a beach in March last year. Paul Monk was swoop by armed police on a villa, he will be arrested and face trial in Spain for this murder.

heading: GPT-3 Zero-shot Summary
text: A man wanted for questioning in connection with the murder of a torture victim has been arrested by Spanish police. Paul Monk, 54, from Essex, was detained at his villa in Javea near Alicante as he directed workers laying a marble patio around his swimming pool. An imitation firearm with a silencer and nearly 0̆0a3100,000 in cash were also found. Monk is being held in jail and is expected to be charged and face trial in Spain over the murder before being extradited to the UK to face questioning over alleged drugs offences.

heading: GPT-3 Zero-shot-cot Summary
text: Paul Monk, 54, from Essex, was arrested by Spanish police in his Alicante villa for questioning over the kidnap and murder of Francis Brennan, 25, from Liverpool. Brennan’s body was found in March last year, wrapped in an industrial-size bin bag with duct tape round it, on a beach in nearby Orihuela Costa. Monk is being held in jail and is expected to be charged and face trial in Spain over Mr Brennan’s murder before being extradited to the UK to face questioning over alleged drugs offences.
Table 22: Random samples from CNN/DailyMail and BBC XSum datasets (Appendix D).
Source Document (BBC XSum)
They’ve kicked up a stink with their council, who’ve decided to use the liquid to kill weeds around the city. It’s all part of a trial to get rid of the unwanted plants without using strong chemicals, which some people think aren’t environmentally friendly. But many locals say they’re not happy with the unpleasant pong the trial’s creating. They’ve started a campaign and also say they don’t think vinegar is strong enough to kill the weeds. Bristol city council say they’ll continue with their trial, but will keeping track of feedback from the people who live there.

heading: Dataset-specific Summary
text: People living in bristol have complained about a vinegary whiff in the air.

heading: Element-aware Summary
text: Bristol city is conducting a trial to kill weeds with vinegar, but the unpleasant pong makes the locals unhappy, which led to a conflict between the two.

heading: GPT-3 Zero-shot Summary
text: The city of Bristol is conducting a trial to see if vinegar can be used as an environmentally friendly weed killer, but many locals are not happy with the strong smell.

heading: GPT-3 Zero-shot-cot Summary
text: The city council of Bristol is conducting a trial to kill weeds using vinegar, but many locals are unhappy with the unpleasant smell the trial is creating.
Table 23: Random samples from CNN/DailyMail and BBC XSum datasets (Appendix D).
Source Document (BBC XSum)
Cpl Kylo gorman’s medals for service in Afghanistan and at the queen ’s diamond jubilee were taken from his car in bolton-le-sands on 4 or 5 January. He said he wasn’t that bothered at first, but the reaction online got me thinking "it is a really big deal" . Lancashire police said they were investigating the theft. The port talbot-born 28-year-old, who has served five tours of Afghanistan, said he had left the medals in the car after removing them from a uniform he was having altered. "I didn’t think twice about them. I’m based at raf leeming in north yorkshire during the week and when I came home and the car had been broken into, I realised my medals had gone," he said. "I want my children to be able to look at their dad’s medals and be proud of what I did," he added. Cpl gorman, who has also served in the Falklands, Africa and Cyprus, has been in the RAF for 10 years and is due to leave the armed forces in April.

heading: Dataset-specific Summary
text: A RAF airman has made an appeal for the return of two stolen medals after hundreds shared his post about the theft on social media.

heading: Element-aware Summary
text: Cpl Gorman’s medals were stolen on 4 or 5 january, and he has made an appeal for the return. Lancashire police are investigating the theft.

heading: GPT-3 Zero-shot Summary
text: The medals of a British serviceman were stolen from his car, and he is now appealing for their return.

heading: GPT-3 Zero-shot-cot Summary
text: Cpl Gorman’s medals for service in Afghanistan and at the Queen’s Diamond Jubilee were stolen from his car on 4 or 5 January, and Lancashire police are investigating the theft.
Table 24: Random samples from CNN/DailyMail and BBC XSum datasets (Appendix D).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract 1

heading: 7 A4. Have you used AI writing assistants when working on this paper?
text: Left blank.
B 3 Did you use or create scientific artifacts? 3 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 3.1 Appendix A

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix A

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Ethics Statement

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethics Statement

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 2.1 2.3
C 3 Did you run computational experiments? 3 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 3.1 Appendix C.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 3.1

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3.1
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 2 3 4

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 2.2 3.3

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix A

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix A

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Ethics Statement

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix A

heading: Future Work
text: mary o’dwyer, chair of Foyle civic trust, said a strategic approach is needed for the future protection of heritage in the city following the demolition. “ many of our members believe that this is a very sad event. “ the factory is part of our industrial heritage and more should have been done to save it. “ others believe something had to be done because it was a health and safety issue, ” she said. “ we would like to see a consensus and a strategic vision of all of the organizations in the city that have a remit for protecting our heritage. ” Foyle road, which was closed between the lower deck of the Craigavon bridge and john street roundabout, has reopened. What are the important entities in this document? What are the important dates in this document? What events are happening in this document? What is the result of these events? Please answer the above questions: