Background Summarization of Event Timelines
Authors: Adithya Pratapa
Kevin Small
Markus Dreyer
Venue: EMNLP 2023 Main (2023)
Keywords: text summarization
events
timelines
dataset
evaluation metrics

Abstract: Generating concise summaries of news events is a challenging natural language processing task. While journalists often curate timelines to highlight key sub-events, newcomers to a news event face challenges in catching up on its historical context. In this paper, we address this need by introducing the task of background news summarization, which complements each timeline update with a background summary of relevant preceding events. We construct a dataset by merging existing timeline datasets and asking human annotators to write a background summary for each timestep of each news event. We establish strong baseline performance using state-of-the-art summarization systems and propose a query-focused variant to generate background summaries. To evaluate background summary quality, we present a question-answering-based evaluation metric, Background Utility Score (BUS), which measures the percentage of questions about a current event timestep that a background summary answers. Our experiments show the effectiveness of instruction fine-tuned systems such as Flan-T5, in addition to strong zero-shot performance using GPT-3.5.

Background Summarization of Event Timelines
Adithya Pratapa' Language Technology Institute Carnegie Mellon University vpratapaēcs.. cmu. edu
Kevin Small
Markus Dreyer
Amazon
(smakevin, (dde:eerenddddeeeeeemeaameaeee com
Abstract
this end, we present the task of background news summarization that complements each update in a timeline with a background summary.
Generating concise summaries of news events is challenging natural language processing task. While journalists often curate timelines highlight key sub-events, newcomers news event face challenges in catching up its historical context. In this paper, we address this need by introducing the task background news summarization, which complements each timeline update with a background summary of relevant preceding events. We construct dataset by merging existing timeline datasets and asking human annotators to write back- ground summary for each timestep of each news event. We establish strong baseline per- formance using state-of-the-an summarization systems and propose query focused variant to generate background summaries. To evalu- ate background summary quality, we present a wawooo-nnanwwwwnnnaaeea
Timeline summarization is the task of automati- cally extracting event timelines from a collection of news articles (Chieu and Lee, 2004). Update summarization (Dang and Owczarzak, 2009) in- volves summarizing a set of recent articles, assum- ing the reader is already familiar with a set of back- ground articles. It has since been studied in shared tasks that track events in newswire (Aslam et al., 2015) and Twitter feeds (Sequiera et al., 2018). Our novel proposed task of background summarization presents an orthogonal use case for the update sum- marization task. A background summary provides sufficient historical context to the reader to help them understand the latest news update. It sum- marizes what has happened previously, in order to explain the background of the current news update. Background summaries allow the reader to quickly grasp the historical context of an event without having to read through potentially hun- dreds of news articles or long timelines regarding specific event. One application would be to contex- tualize short-text content (e.g., Tweets) with back- ground information from news articles about the events. In addition to providing much-needed con- text, this can be useful for verifying the factuality of the events described in the tweet (e.g.. Twitter Community Notes). In news-centric conversa- tional AI setting, a background summary may be generated to answer
evaluation metric, Background Utility Score (BUS), which mea- sures the percentage of questions about cur- rent event timestep that background summary answers. Our experiments show effective- ness of instruction fine-tur ine-tuned systems such as Flan-T5, in addition to strong zero-shot perfor- mance using GPT-3.5.
1 Introduction
Events such as natural disasters, political conflicts, and elections are extensively covered by news agen- cies and followed by readers throughout the world. Generating concise summaries of these events is a challenging NLP task (Chen et al., 2021). For popular news stories, journalists curate retrospec- tive timelines that highlight key sub-events on a timeline. However, for a new observer of spe- cific major event, catching up on the historical con- text needed to understand the significance of the sub-event update can be challenging ordeal. To
request to "tell me what I need to know understand this event".
To construct dataset for background summa- rization, we identify existing timeline datasets cen- tered around major news events. Specifically, select the widely-used Timeline (Binh Tran al., 2013), Crisis (Tran et al., 2015), and Social Time- line (Wang al., 2015) datasets identifying major news events from these datasets and prepare a single timeline of events for each major event.
'Code data available at ht amazon-scient summariiss work done during an internship at Amazon.

---

On June 25th, 2009, Dr. Courad Murra Tound pop stat Michael Jack- son unconscious
Angeles mansion. Paramedics while Marray performed CPR lnter died ar UCLA A medical center: ruled homicide.
called
whose death by "acule propolel intoxication" In February, Muray was charged with involuntary manslugghtrr to which he pleaded not gultty Released on $75,000 bail, he was allowed resume practicing medicine, thetic agents, "specifically propofol".
Dr. Courad Murra linds Jackson unconscious in the hodroom Los Angeles mansion. Paramedics called to the house while he performs CPR, according recording
bunned from administering anes-
911 call. Morray travels with the singer in an ambulance to UCLL medical center where Jackson lazer dies.
Michnel Jackson's 's father, Joseph, wrongful lawsuit against Murray.
Feb 8, 2010
Nov 29, 2011
June 25, 2009
June 25, 2010
Murray is charged with involuntary manslaughter. He pleads not quilty. Released on 575,000 bail, he is allowed to resume practicing medicine, hut the judge buns him achninistering amesthetic agents. "specifically prspofaal..
Dr. Murray sentenced four years in county joil. lutge Michael Pastor says the evidence showed him to guilty of "contimuous pattern of lies and deceii".
On June 25th, 2009, Dr. Conrad Murra found pop star Michael Jackson unconscions at his Angels mansion. later died at UCLA medical center. whose death by acute propofol intoxication was ruled a homi- cide. Murray pleaded not guilly to the charge of involuntury munslaugh- ter, prosocutors allagad that Dr. Murray acted negligently in giving Jackson lethal dose of propofel, ... The defense denied- saying the star caused his own death while Marry was out of the room. Dr. Conrad Munay was found guilty of involuntary manslaughter and was sentencing.
On June 25th, 2009, Dr. Courad Murry found pop star Michael Jackson unconscious in the bedroom of his Los Angeles man- sion. later at UCLA madical center. ... drugs was found in the propotel intoxication" singer's a body, whose death by "acule homicide Court documents
revealed that Dr. Murry bought five bottles of propofoll in May 2009 around the time he was hired Jackson.
remanded in castady without
An illustration of the background summarization task. This is a snapshot from the timeline of the Michael Jackson's 's Death event. The timeline above shows four news updates between June 25, 2009, and November 29, 2011, Each update is complemented with a background summary that provides sufficient historical context to the events and entities described in the update. We highlight phrases from the background that provide context specific phrases in the update text.
Figure 1:
The original datasets included multiple timelines for major events, collected from different news agencies. For simplifying our annotation process, we merged all the timelines for a given major event to create a single timeline. We present these time- lines to expert annotators and collect background summaries at each timestep for each news event. Figure provides example from the timeline of Michael Jackson's 's death. The timeline starts on June 25th, 2009 with a news update. The following updates on Feb 8, June 25, and Nov 29 are substan- tiated using background summaries that provide historical context regarding the event.
ground summary does not include content from the current update, the current update can still be used for conditioning the summarization of past updates. This could potentially improve the utility of the backgrounds. For the query-focused summariza- tion, we explore multiple variants of the query and present a comparison of Flan-T5 and GPT-3.5.
Our experiments indicate that the fine-tuned Flan-T5 system outperforms GPT-3.5 on the stan- dard ROUGE metric while underperforming on factuality metrics. This illustrates the challenges associated with automatic metrics for summariza- tion. Goyal al. (2022) and Zhang et al. (2023) highlight the difficulties in using automatic metrics for comparing fine-tuned system summaries against zero-shot summaries from high-quality large lan- guage models (LLMs). Accordingly, we also present a novel ceaestion-aswering--bee ation of background summaries in this work that measures the utility of given background sum- mary to an update. First, we prompt GPT-3.5 generate questions about the background of events and entities mentioned in the update. Second, we extract answers from the background summarics to measure their effectiveness in providing suffi-
On the resulting annotated dataset, we experi- ment with a suite of summarization systems based on Flan-T5 (Chung et al., 2022), LongT5 (Guo et al., 2022), and GPT-3.5 (OpenAI, 2022). We propose to formulate the background summariza- tion task as two different variants: (1) In a generic summarization setting, generate a background summary for the current update time step by using a concatenation of the past updates at time steps through t 1. (2) In a query-focused setting (Dang, 2005; Xu and Laputa, 2022), we utilize the update at time step as a query. While the back-
evalu-

---

cient historical context the update. Our proposed Background Utility Score (BUS) measures the per- centage of questions about current news updates that are answerable by a background summary. Our human evaluation results show the effectiveness of BUS. Our key contributions are as follows:
Update summarization: Dang and Owezarzak (2009) first proposed the task of update summariza- tion. Given two sets of documents & B, the task is to generate a query-focused update summary of the document set assuming the user of the summary has already read the documents from set A. This task has since been studied on documents from newswire (Aslam et al., 2015) and Twitter feeds (Sequiera al., 2018). In contrast to the timeline summarization task, systems do not have access the documents from the future. Updat- ing users about critical news events in real-time is very important to news and government agencies (Guo et al., 2013), However, comprehending these updates can be challenging for new readers. Our proposed task of background summarization serves complimentary purpose to updates.
1. We present the new task of background sum- marization to help readers follow day-to-day updates complex real-world events.
2. We describe an expert-annotated dataset cov- cring 14 major news events from 2005-201 with over 1,100 background summaries.
3. We conduct benchmark experiments using state-of-the-art zero-shot and fine-tuned sum- marization systems. We also explore query- focused summarization that generates the background by using the current update text as a query.
Background summarization: Hayashi et al. (2020) proposed the task of disentangled paper summarization, in which two separate summaries generated for academic paper: one sum- mary describing the paper contribution, and another summarizing paper context. A related setting for news events can a disentangled summarization of updates and backgrounds of events. Chen et al. (2022) presented a dataset of TV series transcripts and human-written recaps (SummScreen). Simi- lar to background in our task setup, recaps can help viewers understand the current episode. A key distinction is that recap typically provides information from the most recent episode(s) only, but does not provide general context to the story. In contrast, background summaries often include information from the very first update to put the current event in context.
4. We propose an effective QA-based evaluation metric, Background Utility Score (BUS), to measure the utility of background summary with respect to contextualizing update.
2 Related Work
Events in the real world are often complex, consist- ing of numerous threads (Liu al., 2017), and reported by large number of news agencies across the world. Tracking these events and providing important and useful updates to users has been the focus of many works in natural language process- ing and information retrieval communities (Chen et al., 2021). We highlight two specific variants of the event summarization task below,
Some prior works have studied the impact of background knowledge in the standard summariza- tion task setup (Louis, 2014; Peyrard and West, 2020). A typical summarization setup requires a system to generate a summary of a collection documents while ignoring any background knowl- edge already known to the receiver (or reader). Peyrard and West (2020) used a broader defini- tion of background. In their setup, the background constitutes document collection that the user is already familiar with, similar to the document A from the above definition of update summarization. Our definition of background constitutes sum- mary of previous updates a given event that are directly relevant to the current, most recent update.
Timeline summarization: Given corpus of documents and a query, the task retrospectively extract important events from the documents and place them along a timeline (Chieu and Lee, 2004). A typical query consists of major events such as the Haitian earthquake or the BP oil spill. Datasets rely on timelines compiled by news journalists from agencies such as BBC, Reuters, and The New York Times, amongst others. Notable datasets for this task include 1imeline17 (Binh Tran et al., 2013), Crisis (Tran al., 2015), Social Timeline (Wang al., 2015), entities dataset (Gholipour Ghalandari and Ifrim, 2020), and T.S-Newsroom (Born et al., 2020).

---

Major event
Sources (# timelines)
Time period
# U
len(U)
len( (B)
Swine flu
T17 (3)
2009
21
52
45
train
Financial crisis
T17 (1)
2008
65
115
147
Iraq war
T17 (1)
2005
155
41
162
Haitian earthquake
T17 (1)
2010
11
100
61
dev
Michael Jackson death
T17 (1)
2009-200
37
36
164
BP oil spill
T17 (5)
2010-2012
118
56
219
NSA leak
SomillTimeline (1)
2014
29
45
50
Gaza conflict
SomillTimeline (1)
2014
38
183
263
MH370 flight disappearance
SocialTimeline (1)
2014
39
39
127
test
Yemen crisis
Crisis (6)
201
1-2012
81
30
125
Russian-U Cussian-UKraine conflict
SocialTimeline (3)
2014
86
112
236
Libyan crisis
T17 (2); Crisis (7)
2011
118
38
177
Egyptian crisis
T17 (1); Crisis (4)
201 2011-2013
129
34
187
Syrian crisis
T17 (4); Crisis (5)
164
30
162
Table I: An overview of the news events used in our background summarization dataset. The events are grouped into train, validation, and test splits. We list the source dataset and the number of source timelines for each event. The time period provides the overall span of the event timeline. The length of the timeline, the average word count of the (rewritten) updates, and newly annotated backgrounds are specified in the final columns.
Long-form summarization: Our proposed task often contains long timelines of events, requir- ing systems to perform long-form summarization. There is growing effort in the community to im- prove long-range summarization systems. This includes works book summarization (Wu al.. 2021), meeting summarization (Zhang et al., 2022), TV script summarization (Chen al., 2022) and evaluation of long-form summarization systems (Krishna et al., 2023).
full perspective of the news story. this work, we extend this to news timelines.
3.1 Task
Given an event timeline consisting of a time series of updates (UI...., Ur), the task is to generate background summaries (B2...., Br) for all up- dates after U1. For each timestep 1, we wish find the background summary B, that maximizes p( BB UI...., U, 1:9) where q a query. the generic baseline setting, q is empty; in the query- focused setting, is set to the current update U,. In the latter case, we do not aim to summarize Ue, but we use it to direct the summarization of the previ- ous updates toward content that can help explain the current update U,-
3 Background Summarization
Event timelines help readers keep track of updates regarding major news events. They provide con- cise overview of the event's progress over time, without the need to read through hundreds or thou- sands of news articles written about the event. How- ever, for sub-event threads can pose a major challenge for the user (Liiu et al., 2017). We postulate that com- plementing each update with a short background summary regarding the event's past can assist the user in understanding the update. Our approach is inspired by the standard inverted pyramid structure of news articles (Pöttker, 2003). Typically, news articles consist of new newsworthy information at the top, followed by further details about the story, and end with necessary background information. This background information helps the reader gain
Note that each background summary B, gen- crated directly from the previous updates, inde- pendently from the previous background summary B,-1. This enables us to include details relevant to U, from particular previous updates that may not be found in B
events, keeping track of all the
3.2 Dataset Construction
To the best of our knowledge, there exist- ing datasets that provide background summaries. Accordingly, we compile a new, expert-annotated dataset for this, building upon three popular news timeline summarization datasets, Timeline

---

(Binh Tran al., 2013), Crisis (Tran et al., 2015), and Social Timeline (Wang al., 2015).
ROUGE-1
ROUGE-2 ROUGE-L
Rewritten updates
Timeline17: compiled from an ensemble of news websites, this dataset provides 17 timelines spanning 9 major events from 2005-2013.
Annotator
80.9
64.4
74.9
Annotator 2
72.9
54.2
66.2
Annotator 3
80.1
63.2
73.3
Crisis: a follow-up to the Timeline17 17 dataset, this covers 25 timelines spanning 4 major events.
Background summaries
Annotator
47.9
21.3
43.3
While it mostly covers a subset of events from 1imeline17, 17, it adds new event (the Yemen crisis).
Annotator
44.9
16.6
39.5
Annotator 3
48.0
21.1
43,4
Social Timeline: compiled timelines covering 4 major events from 2014. The timelines were collected from Wikipedia, NYTimes, and BBC.
Table 2: TAA across 14 major events.
Table provides an overview of the 14 ma- jor news events compiled from the three datasets. Since the timelines were collected from various news websites (CNN, BBC, NYTimes, etc.), many events have more than one timeline. As each time- line covers the same underlying event, we merge them using timestamps to create a single timeline per event. During this merging process, often end up with more than one update text per times- tamp with possibly duplicate content. We ask the annotators to first rewrite the input updates to re- move any duplicate content. Our annotation pro- cess for each news event contains the following three steps:
3.3 Dataset Splits
For our experiments, we split the 14 major events into train (3 events), validation (3 events), and test set (8 events). Table 1 lists the events in cach split. We include mixture of short and long time- lines across the splits and the test set is mostly temporally separated from the train/dev splits.
Considering the strong few-shot summarization capabilities of large language models (Goyal et al., 2022; Zhang et al., 2023), we decided to budget only a small fraction of expert-annotated data for training and development and leave most events to the test set. This allows sufficient data for further fine-tuning instruction-ben baaodd models (Flan, GPT- 3+) our new task while maintaining sufficient diversity in the set.
1. Read the input timeline to get a high-level understanding of the event.
3.4 Inter-annotator Agreement
2. For each timestep, read the provided 'rough' update summary. Rewrite the update into a short paragraph, removing any duplicate or previously reported subevents.
To measure the inter-annotator agreement (IAA), we compute ROUGE scores² with one annotator's summary as the hypothesis and the remaining two annotators' summaries as references. Table 2 presents the IAA scores for both the re-written up- dates and the newly annotated backgrounds. As ex- pected, see high ROUGE scores the rewritten updates. The scores are lower for background sum- maries, indicating the inherent variance in back- ground summaries.
3.
Go through the timeline in a sequential man- ner and write background summary for each timestep.
Based on this process, we hired three profes- sional annotators. For each timeline, we collect three independent (rewritten) update and (new) background pairs. Our full annotator guidelines are provided in Table 5 in the Appendix. Due to minor differences in the rewritten updates in the timelines, do not merge the annotator timelines. Table provides average lengths of rewritten up- dates and newly annotated background summaries for each major event. In our final dataset, each timestep in the timeline has three pairs rewritten updates and background summaries.
3.5 Background Utility Score (BUS)
Automatic metrics such as ROUGE are found correlate poorly with human judgments of sum- maries (Louis and Nenkova, 2013; Peyrard, 2019). Recent studies highlighted the ineffectiveness of standard metrics when comparing fine-tuned and
FFor ROUGE-L, we use the Lsum variant the Hugging-
Face evaluate package. "Multi-reforence ROUGE returns a maximum score among references.

---

(Human) The well is located 5,000 It below the surface the Gulf of Mexion, 84km south cast Venice, Louisiana.
April 24, 2010.
For the first time, oil is found he leaking from the well. Pressure and release rate unknown. A homeland security report critical infrastructure says the problem has "no near-term impact regional national crude oil natural gas supplies."
Where is the well locuted?
(GPT3355 The well located Gulf of Mexico.
(Flmn-15-XL) The well is located the Gulf of Mexico.
(Mmmmn) owned and operated by BP.
What caused the oil leak?
(.PT NIA
(Human)) NIA
Who owns and operates the well? (GPT-3355 NIA
(Flan- Soan-TS-XC TS-XL) The oil started leaking due the explosion of the Deepwater Horizon drilling ri..
(Hln--15-XL) NIA
Figure 2: Examples of question-ansse answer pairs for BUS ($3.5) generated by prompting GPT-3.5. This example shows update text from the BP oil spill event. Questions are generated from the current update Us, and the answers are generated based on three different background summaries B, (Human, GPT-3.5, "lan-T5-XL); N/A means the background summary did not provide an answer. The BUS score calculated per system as the percentage questions answered by its background summaries. See Table 8 in the Appendix for the full background summaries.
zero-shot summaries (Goyal et al., 2022: Zhang et al., 2023). To account for these limitations and the need to evaluate the quality of backgrounds, we propose a QA-based metric for the background summarization task. Our metric, Background Utility Score (BUS), measures the utility of back- ground B, to the corresponding update U..
and GPT-3.5 (OpenAI, 2022).
Flan-T5: an instruction fine-tuned version of T5 (Raffel et al., 2020). We use Flan-T5-XL with a maximum source length of 512 tokens.*
LongT5: a sparse attention variant of TS that utilizes two efficient attentions, local and transient- global. Source length can be significantly longer than the standard 512 token limits of T5-based system ($3.2). We use the Long-T5-TGlobal-X with a maximum source length of 4096.
To measure the utility, we first prompt a GPT- based model to generate (background) questions from the update text (Ut). We then re-prompt the model to extract answers from the background text (B,). BUS measures the percentage of questions answerable by the background. Figure presents examples of generated QA pairs. The background summary should be able to answer any questions the reader may have upon observing an update. While these questions are latent, we sample them by prompting GPT-based model.
GPT-3.5: variant of the InstructGPT model (Ouyang al., 2022) optimized for dialogue ing reinforcement learning with human feedback. We use this model in a zero-shot setting. We set a maximum source length of 3696.6
We explore both generic and query-focused sum- marization settings ($3.1). In the query-focused setting,
BUS is inspired by QuestEval (Scialom et al., 2021), an interpretable QA-based factuality metric for summarization. QuestEval measures the re- call score by extracting questions from the source and computing the exact match F, between answer spans from the source and summary (vice-versa for precision). BUS is also tangentially related to recent LLM-based evaluation systems such in Vicuna (Chiang et al., 2023) that explored the use of chatbots for evaluating chatbots.
the current update (U²) an addi- tional input to the summarization system.
Generic:
We use task prefix 'summarize:" to instruct T5-based systems. For GPT-3.5, task suffix, 'Provide short summary of the above article.'
The input for the T5-based sys- tems follows the template, 'Generate a short query- focused summary of the background. Query:
Query-focused:
4 Experiments
"nttps://hf..cc/gggllleee wo/geoogll/llannnt//////r
For our background summarization task, we exper- iment with three summarization systems, Flan-T? T5 (Chung et al., 2022), LongT5 (Guo et al., 2022),
" hhttps://hf http://ffccc/ooo///////// -ooggle//ooggt-tgllbbll
"At the time of our experiments, this corresponds to the gpt-3. 3.5 turbo-030, version.
com/docs mooes//mool//gtt3332

---

ROUGE-1
ROUGE-2 ROUGE-L
QucstEval
BERTScore P DUS-GPT-33.
generic
Flan-T5-XL
43.5
41.4
20.4
17.4
39.9
37.6
31.2
25.0
86.3
85.6
46.0
42.2
GPT-3.5
40.5
37.7
15.5
11.7
36.6
33.0
37.2
30.5
88.2
87.2
59.1
54.3
query-foeused
Flan-T5-XL
43.0
41.3
20.6
17.4
39.5
37.6
30.8
24.9
86.2
85.6
46.6
43.6
GPT-3.5
40.2
40.5
15.4
12.9
36.1
35.9
36.9
31.7
87.9
87.5
49.9
47.5
Table 3: System performance (dev test) the background summarization task.
<query>, Background: <past updates>." For GPT- 3.5, we use a task suffix, 'Generate a short query- focused summary of the background." We use 512 and 128 limits for source and query. We consider two variants for queries. First, use the full up- date (U,) as the query. Second, first extract named entities from and use those keywords as the query. The named entity-based approach removes any potential noise from the update and focuses solely on extracting background information about important persons or locations specified in the up- date. We use SpaCy English NER model to extract named entities from the query.
On the generic summarization setup, we observe that fine-tuned Flan-T5 outperforms zero-shot GPT- 3.5 on ROUGE. However, the zero-shot GPT-3.5 model fares much better on factuality metrics and BUS. These trends also valid in the NER-based query-focure formulation. Interestingly, we find the guery-focused formulation generally underper- forms the generic task.10 10 In our experiments, we found Long-T5 underperforms Flan-T5 on the dev set (Table 9 in Appendix). We leave further evalua- tion of Long-T5-basch systems for future work.
5.2 Human Evaluation
Across all our systems, when necessary truncate the oldest updates from the input.? We train both Flan-T5 and LongT5 using DeepSpeed's ZeRO Stage (Rasley et al., 2020). We a maxi- mum target length of 400 tokens.
We conduct a human evaluation to determine the relative quality of the muitten backgrounds (subsection 3.2) and those generated by Flan-T5- XL and GPT-3.5 (generic; top-half of Table 3). We chose to evaluate the generic systems instead of query-focused systems due to their superior perfor- mance on ROUGE and factuality metrics on the development set.
We use GPT-3.5 as our question and an- swer generation system (ref. BUS-GPT-3.5), We generate five questions per update and use heuristic patterns on GPT answers to identify unanswerable questions. Following recent work that showed bet- ter human alignment with GPT-4 (Liu al., 2023), we also experiment with BUS- "
BUS:
Setup: We use the Amazon Mechanical Turk (AMT) platform. We sample 1,000 news updates from the test set and pair each one with the three background summaries, displayed in random order. We collect judgments from three annotators about which of the three displayed summaries is the best (i.e., most helpful) and which one is the worst (i.e.. least helpful). We use majority voting to pick the best and worst summaries. Detailed instructions shown Figure 5 in Appendix. Since annota- tors on the AMT platform are non-experts, multiple methods obtain high-quality judgments, including qualification test and time controls; de- tails including fair compensation of the annotators are described in Appendix A.2.
5 Results
5.1 Automatic Evaluation
Table presents the results on validation and test sets for Flan-T5 and GPT-3.5 in both generic and the NER-based query-focused setups. We re- port scores on the standard summarization met- ric ROUGE (Lin, 2004), two factuality metrics QuestEval (Scialom et al., 2021), and BERTScore Precision (Zhang et al., 2020; Pagnoni al., 2021) and our proposed utility metric BUS ($3.5).
Other viable options truncating middle updates ranking updates based their relevance the current update. "see Table in Appendix for the instruction templates. "We use gpt-3. pptt-355-te www.-tttrooooo0 and gpt-4-0613.
We use best scars-g (BWS; Kir- itchenko and Mohammad (2017)); Table 4 shows
Results:
LWW present further analysis in A.5 in the Appendix.

---

Human 0.2430
Flan-T5-XL
GPT-3.5
2 GPT-3.5
TS-XL
-0.0750
-0.1680
500
best
Table 4: Results of the human evaluation on AMT us- ing best-worss worst scaling (BWS). Values range from (worst) -1 (best).
# examples
O
Best
Worst
1,000
worst
750
500
500
BWS
BUS-GPE335 BUS-GPT-4 BUS-Human
250
Aggregated best-worst votes for human- written, Flan-T5-X0 and GPT-3.5 backgrounds the test set. The top and bottom halves report voted-best best and soted-worst system counts respectively.
Figure 4:
Flan-T5 GPT-3.5
Human
Fhm-I5 GPT-3.5 Human
Figure 3: Vote distribution for best and worst systems from our human evaluation.
past work highlighted the difficulties in choosing evaluation dimensions and task design (Khashabi et al., 2022). Goyal et al. (2022) recommends us- ing an evaluation sctup based on how users utilize the system in practice. To this end, we analyze the effectiveness of BUS ($3.5) in measuring the real-world utility of background summaries.
the results. The values are computed as the per- centage of times summary type is chosen as best minus the percentage of times it is selected as worst. Values of 1.0 or 1.0 indicate that the system has been unanimously picked 'best' and 'worst' re- spectively. We observe that the human-writter sum- maries are substantially preferred over both Flan- T5-XL and GPT-3.5 summaries.
6.1 BUS-GPT
Figure 3 presents the vote distribu- tion for the best and worst summaries across the 1k examples. numan--ritter backgrounds are rated the best by least two annotators in 45% of the examples. They were rated the worst in less than 17% of the examples. Flan-T5 and GPT-3.5 have very similar best-vote distributions (23% and 20%). We see unanimous agreement on the best or worst system in less than 15% of the examples.
Agreement:
Setup: Following our human evaluation setup ($5.2), compare human- Flan-T5-XL. and GPT-3.5 backgrounds. We compute the per- centage of answerable questions using BUS ($3.5) and use this score to identify the best and worst systems for each update.
Results: Figure provides the best-worst vote counts on the same 1,000 updates from test set using GPT--35-bbsed BUS (ref. BUS-GPT-3.5) and GPT-4-based BUS (ref. BUS-GPT-4)." For comparison, also include the vote counts from human evaluation ($5.2; ref. BWS)."
Justifications: Annotators tend prefer human backgrounds over GPT-3.5's due their compre- hensiveness. In the justifications we collected part of our AMT evaluation, human backgrounds were described as 'most comprehensive', and pro- viding 'complete context'. On the other hand, GPT- 3.5 backgrounds were described as 'too short', jjust a timeline', and providing 'least information'.
With BUS-GPT-3.55 we observe that human- written backgrounds slightly outperform GPT-3.5. Flan-T5 T5 significantly underperforms. BUS-GPT-4 is more closely aligned with our best-worst scaling human evaluation (BWS). This is in line with simi- lar observations from prior work on GPT-4-based evaluation (Liu et al., 2023).
6 BUS Analysis
Our human evaluation results showed variance amongst Turkers (Fiiuree 3). This is in line with the observations made by prior work on standard sum- marization datasets (Goyal et al., 2022; Zhang et al., 2023). While human evaluation can be very useful,
Overall, BUS -US--P and BUS -PS-4P ex- hibit different trends for human-written and GPT-
For each example, we use BUS to designate one more systems as best (or worst). 12gec subsection A.5 in Appendix for event-level results.

---

3.5 backgrounds. To analyze this discrepancy. we present BUS-human evaluation that uses question-and-answer pairs compiled by humans.
BUS-GPT-3.5 BUS-GPT-4: We notice BUS- GPT-3.5 suffers from answer hallucination, i.c., responds with an answer even if not mentioned in the background text. On the other hand, GPT-4 is better at declining unanswerable questions (Ta- ble 13 in the Appendix). This explains our observa- tion of better human alignment with BUS-GPT-4
6.2 BUS-Human
Instead of relying on GPT-3.5 (or 4), we use Me- chanical Turk to generate question-answer pairs. We first ask annotators to generate five background questions for each of the 1,000 news updates. For each of these tuples of update and questions, we pair it with one of the associated background sum- maries and ask annotators to attempt to answer all five questions using only information in one of the background summaries (or write none if the sum- mary does not contain the answer). We then cal- culate BUS-Human the percentage of answered questions per background summary type." Re- sults are presented in Figure 4 (ref. BUS-Human). shows clear alignment with our hu- man evaluation results (BWS) and BUS-GPT-4, illustrating the effectiveness of our proposed BUS metric. However, this also highlights a potential drawback of using an automatic system such as GPT-3.5 for generating question-ansan answer pairs.
BUS-Human vs BUS-GPT: Our analysis indi- cates human evaluation remains the gold standard for our proposed background summarization task (BWS $5.2; BUS-Human $6.2). GPT-4 presents promising results and could serve as fast, cost- effective alternative to human evaluation.
We believe BUS can be extended to related summarization tasks such as TV recaps (Chen et al., 2022) and disentangled summarization of scientific articles (Hayashi et al., 2020). A BUS- like metric can measure the relevancy of the recap to the current TV episode and the paper context to its contributions.
Applications:
7 Conclusion & Future Work
6.3 Comparison of BUS methods
Questions: We analyze the questions generated by the three variants, BUS GPT-3.5, BUS GPT- 4, and BUS-Human. In the Appendix, we pro- vide questions generated for example updates from three test events, MH370 flight disappearance (Ta- ble 10), Yemen crisis (Table 11), and Libyan crisis (Table 12). Overall, both humans and GPT gen- erate questions that specifically target background knowledge. Turkers' questions specific and short, while GPT questions are more detailed and often contain two or more sub-questions. Questions target aspects such as named entities (Table 10) and past events (Table 11, Table 12).
To help readers follow long and complex event timelines, we propose the task of news background summarization. We compliment cach update in the timeline with background summary that pro- vides sufficient context the readers. We present an expert-annotated dataset for this task with over 1, 100 background summaries from three annotators. On this dataset, we benchmark a suite of state-of: of- the-art summarization systems (Flan-T5, LongT5, and GPT-3.5). Our results show the zero-shot GPT- 3.5 system outperforms the fine-tuned systems on the factuality metrics while underperforming on ROUGE. Given the lack of metric that accurately captures the utility of background summary to the news reader, we propose novel QA-based metric, BUS, which measures the percentage of questions about the updates that are answerable from the re- spective background summaries.
However, we also see questions that do not target background information. Some questions from humans and GPT ask for additional details about events described in the update. See Q3 from Turker 2 and Q4 from GPT-3.5 in Yemen crisis (Table 11), and Q5 from GPT-4 in Libyan war (Table 12). A few questions ask about the consequences of the events described in the update. See Q5 from GPT- 4 in MH370 disappearance (Table 10), Q4 & Q5 from Turker 2 in Yemen crisis (Table 11).
For future work, we plan to explore background summarization directly from news articles instead of past updates. Sub-events previously considered unimportant but directly consequential to the latest news update can be captured this setup. We are also interested in benchmarking aspect-based summarization systems for our task.
13 'Appendix A.3 contains more details about our sctup, an- notation guidelines, and compensation.

---

Limitations
References
Javed Aslam, Fernando Diaz. Matthew Ekstrand-Abuee Richard McCreadie, Virgil Pavlu, and Tetsuya Sakai. 2015. Tree 2015 temporal summarization track overview. In TREC.
Personalized Backgrounds: While a back- ground summary is helpful to any news reader, the utility can vary depending on the reader's familiar- ity with the event. In our BUS analysis (section 6). we observed differences in questions generated by two Turkers. In an ideal setting, systems should be capable of generating personalized background summaries catering to a reader.
Giang Binh Tran, Mohammad Alrifai, and Dat Quoc Nguyen. 2013. Predicting relevant news events for timeline summaries. In Proceedings of the 22nd International Conference World Wide Web, WWW '13 Companion, page 91-92, New York, NY, USA. Association for Computing Machinery.
Our dataset and systems are cur- rently limited to a selection of global, popular events involving disasters and conflicts (Table 1). For events local to specific community, timelines are hard to find, and it's even harder to create back- ground summaries. We acknowledge that back- grounds are equally impactful for local events and leave this extension to future work.
Local Events:
Leo Born, Maximilian Bacher, and Katja Markert. 2020. Dataset reproducibility and IR methods in timeline summarization. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 1763-1771, Marseille, France. European Language Resources Association.
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. SummScreen; A dataset for abstrac- tive screenplay summarization. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Lonng Papers), pages 8602-8615, Dublin, Ireland. Association for Compu- tational Linguistics.
Background from News Articles: We generate backgrounds from past news updates. However, they can also be generated directly from news arti- cles. We leave this extension future work.
Muhao Chen, Hongming Zhang. Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown, and Dan Roth. 2021. Event-centric natural language processing. In Pro- ceedings of the 59th Anmual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Tutorial Abstracts, pages 6-14, Online. Association for Computational Linguistics.
Ethics Statement
We used Mechanical Turk for our human evalua- tion of summarization systems. We ensured prompt and fair pay to our annotators. We provided de- tails about our selection criteria, per-task payment, and bonus in subsection A.2 and subsection A.3. Our background summarization dataset is expert- annotated on top of publicly available timeline sum- marization datasets. We protect our annotators' privacy and remove any personally identifiable in- formation from our data release.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E, Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Hai Leong Chicu and Yoong Keok Lec. 2004. Query based event extraction along a timeline. In Proceed- ings of the 27th Annual International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, SIGIR '04, page 425-432, New York, NY. USA. Association for Computing Machin- ery.
Like other text generation systems, generative summarization systems suffer from hallucina- tions, potentially leading to misinformation. We acknowledge misinformation in backgrounds un- desirable real-world applications. To this end, we report two factuality- based metrics to quantify the factuality of our systems.
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu. Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu. Slav Petrov, Ed Huai hsin Chi. Jeff Dean, Jacob Devlin, Adam Roherts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scal- ing instructionffinttued language models. ArXiv. abs/2210.1 1416.
Acknowledgements
We thank the anonymous reviewers for their valu- able feedback that helped improve our paper. We thank Teruko Mitamura for their support and Ting- Rui Chiang, Aidan San, and Xueguang Ma for help- ful discussions in the carly stages of this project

---

Hoa Dang and Karolina Owezarzak. 2009, Overview of the tac 2008 update summarization task.
Chin-Yeu Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Hoa Trang Dang. 2005. Overview of due 2005, In Pro- ceedings of the document understanding conference, volume 2005, pages 1-12. Citeseer.
Bang Liu, Di Niu, Kunfeng Lai, Linglong Kong, and Yu Xu. 2017. Growing story forest online from mas- sive breaking news. Proceedings of the 2017 ACM Conference on Information and Knowledge Man- agement, CIKM 17. page 777-785, New York, NY, USA. Association for Computing Machinery.
Demian Gholipour Ghalandari and Georgiana Ifrim. 2020. Examining the state-of-the-ar in news timeline summarization, In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 1322-133 1334, Online. Association for Computational Linguistics.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nig evaluation using gpt-4 with better human align- ment. ArXiv.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3, ArXiv, https/2299.123566
Annie Louis. 2014, A Bayesian method to incorpo- rate background knowledge during automatic text summarization. In Proceedings of the 52nd Anmual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 333-338. Baltimore, Maryland. Association for Computational Linguistics.
Mandy Guo, Josbua Ainslic, David Uthus, Santiago On- tanon, Jianmo Ni. Yun-Hsure Sung, and Yinfei Yang. 2022. T5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguisticy; NAACL 2022, pages 724- 736, Seattle, United States. Association for Compu- tational I Linguistics.
Annie Louis and Ani Nenkova. 2013. Automatically as- sessing machine summary content without a gold standard. Computational Linguistics, 39(2):267- 300.
Qi Guo, Fernando Diaz, and Elad Yom-Tov. 2013. Up- dating users about time critical events. In Proceed- ings of the 35th European Conference Advances in Information Retrieval, ECIR' 13, page 483-494, Berlin, Heidelberg. Springer-Verie Verlag.
OpenAL 2022. Introducing chatgpt.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Pe- Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, 22222203.02155.
Hiroaki Hayashi, Wojeiech Kryšcinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2020. What's new? summarizing contributions in scientific litera- ture.
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel Weld. 2022. GENIE: Toward re- producible and standardized human evaluation for text generation. In Proceedings of the 2022 Con- ference on Empiricul Methods in Natural Language Processing, pages 11444- 11444411458,, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 4829, Online. As- sociation for Computational Linguistics.
Svetlana Kiritchenko and Saif Mohammad. 2017, Best- worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465-470, Vancouver, Canada. Association for Computational Linguistics.
Maxime Peyrard. 2019, Studying summarization eval- uation metrics in the appropriate scoring range. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5093- 5100, Florence, Italy. Association for Computational Linguistics.
Maxime Peyrard and Robert West. 2020. KLearn: Back- ground knowledge inference from summarization data. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 2073-2085, Online. Association for Computational Linguistics.
Kalpesh Krishna, Erin Bransom. Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for human evaluation of faithfulness in long-forr form summarization. In Proceed- ings of the 17th Conference of the Furopean Chap- ter of the Association for Computational Linguistics, pages 1650-1669, Dubrovnik, Croatia. Association for Computational Linguistics.
Horst Pöttker. 2003. News and its communicative qual- ity: the inverted pyramid wehen and why did it ap- pear? Journalism Studies, 4(4):501-511.

---

Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, (114))):1667
dialogues and documents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1592- 1604, Dublin, Ireland. Association for Computational Linguistics.
Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase. and Yuxiong He. 2020. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, page 3505-3506, New York, NY, USA. Association for Computing Machinery.
A
Appendix
A.1 Annotation Guidelines for Writing Background Summaries
Table 5 presents the guidelines we presented to the annotators who wrote the summaries for our dataset (from subsection 3.2). We conducted multi- ple rounds of training with the annotators, where we reviewed unnotator's work and provided feed- back on the quality of background summaries.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarkki, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summariza- tion asks for fact- evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
A.2 Details on the MTurk BWS Evaluation
We provide additional details on our Amazon Me- chanical Turk setup (from subsection 5.2). We give detailed instructions to the annotators, see Figure 5, Workers who complete the tasks too quickly are automatically removed from our worker pool: their answers are replaced with new answers. We also use bonus incentive structure. Every worker who passes the automatic time check receives a bonus at the end. In addition, we only use workers from our pool of about 300 trusted workers from previ- ous studies. These were selected in two stages: (1) We only considered workers from countries whose main language is English and who have completed 100 or more HITs so far with an acceptance rate of 95% or higher. (2) In addition, workers must have passed an initial custom qualification test for a related text classification task we have conducted. Moreover, the resulting pool of workers has been used in more than 50 previous experiments, and we have over time removed any workers who have provided low low-quality output in those previous ex- periments. On our batch of 1,000 HITs for the present human evaluation, we allowed any worker to complete maximum of 333 HITs so that no worker can dominate the results. We used three annotators per HIT.
Royal Sequiera, Luchen Tan, and Jimmy Lin. 2018. Overview of the tree 2018 8 real-time summarization track. In TREC.
Giang Tran, Mohammad Alrifai, and Belco Herder. 2015. Timeline summarization from relevant head- lines. In Advances in Information Retrieval, pages 245-256, Cham. Springer International Publishing.
Lu Wang, Claire Cardie, and Galen Marchetti. 2015, Socially-ifformed timeline generation for complex events. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1055-1065, Denver, Colorado. Asso- ciation for Computational Linguistics.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nissan Stien- non, Ryan Lowe, Jan Leike, and Paul Francis Chris- tiano. 2021. Recursively summarizing books with human feedback. ArXiv, abs/2109.10866
Yumo Xu and Mirella Lapata. 2022. Document sum- marization with latent queries. Transactions of the Association for Computational Linguistics, 10:623- 638.
Tianyi Zhang*, Varsha K Felix Wu*, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori Hashimoto. 2023. Benchmarking large language models for news sum- marization. ArXiv, 0bb/22301.1184
Payment: We paid $0.70 per HIT with bonus of $0.05 for all workers who passed automatic quality checks. 39 workers worked on our HITs overall, spending a median time of 169.0 seconds per HIT. This amounts to an average pay of $14.91 per hour per worker.
Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, and Rui Zhang. 2022. Summ": multi-stage sunmarization framework for long input

---

A.3 Details on the MTurk BUS Evaluation
Metries: ROUGE, BERTScore and QuestEi val. 14.15.16
In order to calculate the BUS metries based on numan-written questions and answers (from subsec- tion 6.2), we conducted two separate MTurk evalua- tions: (1) we obtained questions about news events and (2) we obtained answers to these questions given the different background summaries (human- written, generated from GPT-3.5 or from Flan- T5). For both evaluations, used the same gen- eral setup and annotator qualifications described in $A.2.
A.5 Additional Results
Event-level BUS: Similar to the results Fig- ure 4, we report the best-worst vote counts per event in the test set. For each event, we report counts for BWS (Figure 8), BUS BUS-GPT-3.5 (Fig- ure 9), BUS-GPTI (Figure 10) and BUS-Human (Figure 11).
Query-focused Summarization: In Table our query-focused summarization setup did not pro- vide gains. To analyze this behavior, we further experiment with an alternate query format where we use the full update text (U)) as the query. Table 7 presents the results the Flan-T5 system using ROUGE-L, QuestEval and BUS. We notice only slight improvement in the performance when using full update text as the query.
To obtain five background questions for each of Ik news updates, we submitted 1,000 HITs. We paid $0.75 per HIT with bonus of $0.05 for all workers who passed automatic quality checks. 46 workers worked on our HITs overall, spend- ing a median time 179.0 seconds per HIT. This amounts to an average pay of $15.08 per hour per worker. The annotation guidelines and an example annotation are shown in Figure 6. We allowed any worker to complete maximum of 333 HITs so that worker dominate the results.
To obtain answers to the five questions per news update with respect to the three different back- ground summaries, we submitted 3,000 HITs. We paid S0.70 per HIT with a bonus of $0.05 for all workers who passed automatic quality checks. 38 workers worked median time of 144.2 seconds per HIT. This amounts an average pay of $17.47 per hour per worker. The annotation guidelines and an example annotation are shown in Figure 7. We allowed any worker to complete a maximum of 500 HITs.
HITs overall, spending
A.4 Experiment Setup
T5-based systems: We perform training using DeepSpeed ZeRO stage 3 two A6000 GPUs. We train the models for 10 epochs and pick the best model using the ROUGE-L score on the dev set. We use a per-device batch size of and learning rate of le-5. We use the Trainer from Hugging Face in all of our experiments. At inference time, we use beam size of 4, length penalty of 2.0, and no-repeat ngram size of 3.
GPTTbased systems: We use the OpenAl python API for all of our GPT-based systems.
Instructions for BUS-GPT: Table presents our instruction templates for question and answer generation using GPT models.
bertscore "https:/ http://tttt....ooooooaaaaaattttt

---

Instructions (Clikk to colasss)
In this task, we ask you rate thrse teat summaries about news events. You are given updato about news story. These news storles are world
information Nett,, summary of the past ovents news story. Each summary should provide connext reader and hhip them understand the newe update.
story.
update. Then, go through three summaries that provide you with background Information best and worst summartes. Feel free choose multile summariis best for worst) it they are equally eftective for inotfoctivel in helping you understand the news update. Provice short justification your rating the text box.
story.
carefulyy
story.
News Update: Date: 201 -03-0---00 92, Article: The miltary briefly repels the rebels in Port Brega, important ofl ol-producing complex, but the rebets soon Lanuf.
Gacdan's
For the above update, the following three summarie provide the necessary historical contoxt from the news story. A good summary will provide relevant information from the past that helps you better understand the news update. Depanding on their utilly, idensify the best prost pful) and helpfu)
Background Summary t: Libyans went strike January 2011 1 protest the 40-year nute Moammar Gadtani who had been power since 1969. family
EU banned
ammuntion Libya, imposed visa Assembly adopted
Gachall five 47-member Human Fights
remove Libya
Council.
demonatiations vicient between proteeders securit international community reeponds condemnation, sanctions, step down, Anti- unt--Gaddal rebels gain control of several citiee. and the UN General Assembly removes Libya from seet on the Human Rights Council,
Summary
Libyan
Background Summary Following Libyan leader Moammar Gadcal to step down, Gaddafi called the army to suppress the rebellion, leading to much bloodshed which was conderned the UN Security Council and the Arab League. Ao rebeis took control of most eastemn otties and ciosed on Tripoll international intensified,
Egyptan
protesters Benghazi
Libya
UN
sanctions
Rate the best and worst summaries.
Best summary
Summary
Summary
Summary
Worst summary
Summary
Summary
Summary
Justification
Figure 5: This screenshot shows the human anmotation interface to determine the best and the worst background summary for for best-worst scaling. In this example, the random order of displayed summaries is Flan-T5-XL. GPT-3.5, followed by the human-writted summary. Here, both annotators marked the human-wrilten summary as the best and the GPT-3.5 summary as the worst.

---

Instructions (Clck to colepee!
given news story. These news stories are about popular real-orrd events. Assume you have no prior or background information about the news story. Write five qualitions might have about the background newe story.
questions
etory. The questions ahould be benveen and 15 words long. Bee two ecamples ow.
Exaemple
CDC saya flu may first feared. The WHO no plans to increase the alert level from prase five phase elc. The number confermed casea in Scotland risea to thres, friand of lain and Down Askham tasting
Dute:
are
Question What caused the outbreak??
Question
they rolated outbraak?
Questton What were previous assesamente by US CDC and WHO on the flu outbresk?
Question
timeline outbreak
Question How UK and Sootland respand the outbreak??
Example
News Update: Uste: 2009-09-23, the first Capital Hill hearings, hawmakers from both partis questioned polkymakkars. Accordiig James Lookhart Finance Agency, Fannie Mae and Fredide Mac could not continue oover mortiagge losses without assistance. fallure firencial glants Fannle Mon, Freddle Mac, Lehmmnn Brothers, and AIG bringing the total number bursau investigations into the ortala to 26. Major Japanese banks continue to aoquire
Housing g
investigation
froud played
the
Question What are the Cacital Hill hearinge about?
Question Why
mortgage
Question What caused the craiss that is being inventigated by FD!
timeline Question Why are major Japanese bankes acquiring Wal Street sccets? and what are these assers?
Question
until Hill hearings?
Please read the showe instructions carefully before you start the taok, Below is an update from news story.
News Update: Cate: 2011 CS 01, Articee: The MATO arstrike killed his youngest son, Seif al Arab, and three grendchildeen but it did not hurt Gocklafi
Write five questions you might have about the background of this news story. Do not ask questions update.
whose contained directly the events from the past that are relsted to the news update.
questions about the
Background Questions
What roason for airstikas Libya?
Queetton
Who was Salt al Anab Gaddaaf and what was Nis role the Libyen government?
Question
What wore the cieumstanose of the airstrkke that kiledd Seif ai -Arrb Gadcol and his grandchildrr
Question
What was the necciinn the Libyan government airstriw??
Quesstion
long-temm consequencee
Libyan
Submit
Figure 6: This screenshot shows the annotation interface for MTurk annotators to write five questions about news event. The questions from an annotator are shown the text fields as an example.

---

Instructions (Cilck to collapes)
this you
You article
a news story. Theee newe stories are about populr reai-world events, Assume you have no prior nackgrooudd intermation about the news story, You provided five questions. For each answer
question, answers artce, wribe "None".
read above canelully before you atart task. story.
a riteed
Article: Demonstrators Libya are demar dlnng an end to Muammar Gaddaf's 42 yeur ruis, taking part in other
the These with stror g-arm reeponse governmnt securiy forows that hundreck diedd injured and earned neemational condernation. With committed support, excluding troops on the European
protests Benghoss,
Tripor,
ground,
Gaddaf'ss regi me, rebals hava bean fighting government forces over control of strategic cities in coalition killing ciillans
Gadcafi urged NATO negotiate and destroying the netio'' infrastucctuee order seize control production. According the govermment, NATO misa attack on house Tripol has killed Goddaaftss youngest son, Ball al Arab, grandchidren.
airstriees,
question article, write "None'.
the
articie.
available the
Litya?
Question al-Arab Gaddafi his the Libyan govemmeen?
Question What ware the circumstances of the abistrike that Killed Baif al-Arab Gaddal and his
Question roaction the
airstrike?
Question What were the torn consequanocs of the airstrike for the Libyan civil war?
Answers
Gaddall
Answer
He was Goddall's is son, which his political role unspeaifed.
Answer
was NATO miscle attack on house Tripoki
Gaddef urged regoliata rittrikse..
Answer
None
9.bmit
Figure 7: This screenshot shows the annotation interface for MTurk annotators answer five questions about news updates, given one of the background summaries. The answers from an annotator are shown in the text fields an example.

---

Terminology
short text summary of what's new in the news story. This text summarizes the latest events, specifically ones that important to the overall story.
Update:
Background: short text summary that provides sufficient historical context for the current update. Background aims to provide the reader a quick history of the news story, without them having to read all the previous updates. Background should cover past events that help in understanding the current events described in the update. Timestep: day of the event (YYYY-MM-DD),
Timeline: series of timesteps. Each timestep in a timeline is associated with an update and background summary.
Super event: the key news story major event for which constructing a timeline. For instance, 'Egyptian Crisis', 'BP oil spill', 'MH 370 disappearance" are some of the super events from our dataset.
Annotation Steps
We follow a three-stage annotation process,
Stage-O: Read the input timeline to get high-level understanding of the super-event. Stage-1: For each timestep, read the provided 'rough' update summary. Rewrite the update into short paragraph, removing any duplicate previously reported subevents. Stage-2: Go through the timeline in sequential manner and write background summaries for each timestep.
Table 5: Annotation guidelines for the background summarization task.
BUS question generation
Update: (update)
Imagine you read the above update about a news story. You have no prior information about the story. Generate five background questions you might have about the story.
Flan-T5-XL
ROUGE-L
QuestEval
BUS
BUS answer extraction
query: U.
39.5
30.9
46.9
query: NE(U))
39.5
30.8
46.6
Background: [background)
Questions: {questions)
Table 7: Ablation studies on different queries for back- ground summarization task (dev set).
For each question, list answers from the back- ground text when available. Say "unanswerable" if the question is not answered in the background text.
Table 6: Instruction templates for GPT-based question-
answer generation.

---

Human GPPT-35 EE Flan-T5 T5-XL
confine
syrian-crisis
leakleak
mh370-dissppeerance
libyan-war
gaza-conflict
egyptian-crisis
160
140
120
100
80
60
40
20
40
60
100
120
140
160
worst
examples
best
Figure 8: Aggregated best-worst votes for human-written, Flan-T5, GPT-3.5 backgrounds (BWS). The left and right halves report voted- worst and voted- best system counts respectively.

---

Human GPPT-35 E3 Flan-T5-X0
-eiii-
confine
syrian-crisis
leak
mh370-dissppearance
libyan-war
gaza-conflict
egyptian-crisis
160
140
120
100
80
60
40
20
40
60
100
120
140
160
worst
examples
best
Figure 9: Aggregated best-worst votes for Humann-wiitten, Flan-T5, GPT-3.5 backgrounds (BUS-GPT-3.5). The left and right halves report voted-wond and voted-best system counts respectively.

---

Human GPPT-35 BE Flan-T5-XI
confine-confict
crrian-crisis
leakleak
mh370-dissppeerance
libyan-war
gaza-conflict
egyptian-crisis
160
140
120
100
80
60
40
20
40
60
100
120
140
160
worst
examples
best
Figure 10: Aggregated best-worst votes for human-written, Flan-T5 GPT-3.5 backgrounds (BUS-GPT-4). The left and right halves report voted-wort and voted-best system counts respectively.

---

Human GPPT-35 E3 Flan-T5-X0 XL
confine
syrian-crisis
leakleak
mh370-dissppearance
libyan-war
gaza-conflict
egyptian-crisis
160
140
120
100
80
60
40
20
20
40
60
100
120
140
160
worst
examples
best
Figure 11: Aggregated best-worst votes for human- writtten, Flan-T5, GPT-3.5 backgrounds (BUS-Human). The left and right halves report voted-wor and voted-best system counts respectively.

---

Date: April 24, 2010.
Update: For the first time, oil is found to be leaking from the well. Pressure and release rate are unknown. A homeland security report critical infrastructure says the problem has "no near-term impact to regional or national crude oil natural gas supplies."
(Human) On April 20th, BP-operated drilling rig Deepwater Horizon exploded in the Gulf of Mexico, 84km south-cast of Venice, Louisiana, when a blowout preventer failed to activate, leaving 11 dead and 17 injured. The rig has been found upside down about quarter-mile from the blowout preventer. Multiple federal departments and agencies are involved in 28 search-and-re ren-cuu operations covering about 5,300 square miles. Oil sheen has been reported, but the Coast Guard says it's had no indication that oil is leaking from the well 5,000 ft below the surface of the Gulf.
Background:
nA homeland security department risk analysis has said the incident "poses a negligible risk to regional oil supply markets". White House press secretary Robert Gibbs played down any possible effects on President Obama's plans to expand offshore drilling, saying "I don't honestly think it opens up whole new series of questions because, you know, in all honesty I doubt this is the first accident that has happened and I doubt it will be the last".
Background: (GPT-3.5) On April 20, 2010, the Deepwater Horizon drilling rig exploded in the Gulf of Mexico, leaving 11 workers missing and presumed dead, and injuring approximately 17. The blowout preventer failed to activate, and the rig sank to the bottom of the Gulf, potentially allowing oil to spew out from the well. The National Response Team, involving 16 federal departments and agencies, was activated, and chemical dispersants were readied to break up the oil slick. The US coast guard suspended the search for the missing workers and reported no indication of oil leaking from the well. The incident was deemed to pose negligible risk to regional oil supply markets and was not expected to cause sigmificant national cconomic impacts.
Background: Delan-TS-XLL The Deepwater Horizon drilling rig exploded on April 20, 2010 while under construction in the Gulf of Mexico, killing 11 workers. The rig was found upside down about a quarter-mile from the blowout preventer. The force of the explosion broke off the rig's s drillpipe, potentially allowing oil to spew out into the gulf. A National Response Team, involving 16 federal departments and agencies including the White House, Coast Guard, the departments of Defense, Homeland Security and Interior, and Environmental Protection Agency, among others, activated. A Coast Guard official said the Macondo well, which the rrig had been drilling, could be releasing up to 8,000 barrels of oil per day. A remotely operated vehicle with camera went underwater and confirmed no visible oil flow from the well, but there were reports of an oil slick five miles long. Chemical dispersants were readied to break it up. A homeland security department risk analysis said the incident "poses a negligible risk to regional oil supply markets and will not cause significant national cconomic impacts". White House press secretary Robert Gibbs was asked whether the incident might affect Obama's 's plans to expand offshore drilling. He said, "I don't honestly think it opens up whole new series of questions because, you know, in all honesty I doubt this is the first accident that has happened and I doubt it will be the last"
Table 8: This table provides the background summaries generated by humans, GPT-3.5, and Flan-T's T5 for the given update. See Figure for the QA pairs generated from these update-background pairs by prompting gpt-3. a.tt..bbbb.......
ROUGE-1
ROUGE-2
ROUGE-L
QuestEval
BERTScore P
GPT-335
43.5
20.4
39.9
31.2
86.3
46.0
Long-TS Tee-----GGooo 155oallobll-L
40.1
16.7
36.4
33.4
86.9
46.2
Table 9: A comparison of Flan-T5 and Long-T5 T5 systems on the dev set.

---

Date: June 4, 2014.
Update: Australian researchers release recording of an underwater sound that could have been MH370 hitting the water.
BUS-human (Turker 1)
Q1: What is MH370?
Q2: What was the name of the researchers?
Q3: Where was the crash?
Q4: What else could the sound have been?
Q5: How did they record the sound?
BUS human (Turker 2)
Q1: What was the flight path of MH370?
Q2: What were the last known communications from MH370?
Q3: What the search parameters being used by the Australian researchers?
Q4: What are the other possible explanations for the underwater sound?
Q5: What are the implications of the underwater sound for the search for MH370?
BUS-GPT-3.5
Q1: What MH370 and why is it significant?
Q2: How did the Australian researchers obtain the recording of the underwater sound?
Q3: What other evidence has been found regarding the disappearance of MH370?
Q4: What the current status the investigation into the disappearance of MH370?
Q5: What impact could this new evidence have on the families of the passengers and crew on board
MH370?
BUS-GPT-4
Q1: What is MH370 and what happened to it?
Q2: Who are the Australian researchers involved in this investigation?
Q3: How were the researchers able to capture this underwater sound?
Q4: What evidence suggests that this sound could be MH370 hitting the water?
Q5: Has this new evidence brought any significant progress in the investigation of MH370?
Table 10: For an update from the 'MH370 flight disappearance' event, this table provides BUS questions generated by humans (MTurk)). GPT-3.5 and GPT-4.

---

Date: November 23, 2011.
Update: President Salch signs deal brokered by the Gulf Cooperation Council which will see him hand over power to his vice president in exchange for immunity from prosecution for him and his family. He will remain honorary president for 90 days.
BUS human (Turker 1)
Q1: What is Saleh president of?
Q2: What the Gulf Cooperation Council?
Q3: What crimes Saleh immune to prosecution for?
Q4: Who is Salch's vice president?
Q5: What else was in the deal that was signed?
BUS- human (Turker 2)
QI: What were the reasons for the protests against President Saleh?
Q2: What was the role of the Gulf Cooperation Council in mediating the deal?
Q3: What were the terms of the deal that President Saleh signed?
Q4: What was the reaction of the Yemeni people the deal?
Q5: What are the long-term implications of the deal for Yemen?
BUS GPT-3.5
Q1: What led President Saleh's 's decision hand over power to his vice president?
Q2: Who is President Saleh's vice president and what is his political background?
Q3: What is the Gulf Cooperation Council and how did they broker this deal?
Q4: What kind of immunity will President Saleh and his family receive and what crimes are they
being protected from?
Q5: What will happen after the 90-day period when President Saleh's 's honorary presidency ends?
BUS-GPT-4
Q1: Who is President Saleh and what country is he the president of?
Q2: What led to the need for President Salch to hand over power to his vice president?
Q3: What is the Gulf Cooperation Council and what role did they play in brokering this deal?
Q4: Why is immunity from prosecution part of the deal for President Saleh and his family?
Q5: Who is the vice president that will be taking over power and what is their political background?
Table 11: For an update from the 'Yemen crisis' event, this table provides BUS questions generated by humans (MTurk), GPT-3.5 and GPT-4.

---

Dale: November 22, 2011.
Update: Libya's interim prime minister Abdel Rahim al-Keeb announces cabinet.
BUS-human (Turker 1)
Q1: What led to the need for new cabinet announcement in Libya?
Q2: Who was the previous prime minister ?
Q3: What challenges did the interim government face in forming the new cabinet?
Q4: What are the key responsibilities and goals of the new cabinet ?
Q5: How was the new interim prime minister selected?
BUS-human (Turker 2)
Q1: What happened to Libya's previous prime minister?
Q2: Can an interim prime minister create a new cabinet?
Q3: Do the people of Libya like Abdel Rahim al- Keeb?
Q4: When will there be new permanent prime minister?
Q5: What happened Libya's previous cabinet?
BUS GPT-3.5
Q1: Who is Abdel Rahim al-Keeb and how did he become Libya's 's interim prime minister?
Q2: What were the reasons for the formation of new cabinet in Libya?
Q3: Who are the members of the new cabinet and what are their qualifications and experience?
Q4: What are the main challenges facing the new cabinet in Libya and how do they plan to address
them?
Q5: What is the current political situation in Libya and how does the new cabinet fit into the broader political landscape?
BUS-GPT-4
Q1: Who was the previous prime minister of Libya and why did they leave the position?
Q2: What led to Abdel Rahim al- Keeb becoming the interim prime minister?
Q3: What is the political situation in Libya that necessitated new cabinet?
Q4: What the main challenges facing the new cabinet in Libya?
Q5: Who are the members of the new cabinet and what are their political backgrounds?
Table 12: For an update from the 'Libyan crisis' event, this table provides BUS questions generated by humans (MTurk), GPT-3.5 and GPT-4.

---

Date: June 16, 2012.
Update: Escalating violence leads to the suspension of the UN observer mission in Syria. Background (GPT-3.5): The article provides a timeline of events during the Syrian uprising from May 2011 1 to May 2012. It includes the imposition of sanctions Assad and senior members of the government by the BU, violent clashes and protests, defections, and the Arab League peace plan, The article also covers the UN-brokered cease-fire and the presence of UN observers, well the continued violence and massacres that occurred during this period.
BUS 5PT-35
Q1: What was the purpose of the UN observer mission in Syria?
Q2: What the current level of violence in Syria and who is responsible for it?
Q3: How long has the UN observer mission been in Syria and what progress have they made?
Q4: What actions have been taken by the international community to address the violence in Syria?
Q5: What are the potential consequences of suspending the UN observer mission in Syria?
A1: The purpose of the UN observer mission in Syria was to monitor the implementation of UN-brokered cease-fire.
A2: The article does not provide information on the current level of violence in Syria or who is responsible for it.
A3: The UN observer mission has been in Syria since April 2012, and the article does not provide information on the progress they have made.
A4: The international community has imposed sanctions on Assad and senior members of the government by the EU, and the Arab League has proposed a peace plan. The UN has also brokered a cease-fire and sent observers to monitor it.
A5: The potential consequences of suspending the UN observer mission in Syria are not discussed in the background text.
BUS GPT-4
Q1: What is the UN observer mission in Syria and what is its purpose?
Q2: What specific events led to the escalation of violence in Syria?
Q3: How long has the UN observer mission been in Syria?
Q4: What are the potential consequences of suspending the UN observer mission in Syria?
Q5: Who are the main parties involved in the violence Syria?
A1: Unanswerable
A2: Unanswerable
A3: Unanswerable
A4: Unanswerable
A5: Unanswerable
Table 13: For an update from the 'Syrian Crisis' event, this table provides BUS questions and answers extracted GPT-3.5 and GPT-4. GPT-335 suffers from answer hallucination, i.e., responds with an answer even if its not mentioned in the background text. See A1, A3 in BUS-GPT-3.5.


Reviews:
[{"review_id": "pPjpKpurA3", "reviewer": "Reviewer_Lm5f", "summary": "", "questions": "Related to utility of the task, in line 240, it is stated that news articles “end with necessary background information”. Couldn’t the background information from a recent article be used as the background summary, perhaps together with a brief summary from the current article? Have you compared a summary using the article’s background summary to the timeline summaries?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and constructive comments.\n\n> The paper would be strengthened if more applications of the task could be given, to indicate more general utility than just journalists.\n\nIn addition to complementing journalists who create timelines, a background summarization system can assist readers in following complex news events by providing necessary contextual information. An interesting application would be to contextualize short-text content such as tweets with background information extracted from news articles about the events. In addition to providing much-needed context, this can be useful for verifying the factuality of the events described in the tweet itself (e.g., Twitter community notes). Finally, in conversational AI settings, a background summary can inform a “tell me what I need to know to understand this event” request in news-centric experiences.\n\n> For the evaluations comparing summaries, only generated summaries were compared in Figs 3 and 4.\n\nIn our evaluation setup (Figures 3 and 4), we compare the Flan-T5 and GPT-3.5 models from the generic setup (top half of Table 3) and the human-annotated backgrounds (section 3.2). We chose to evaluate the generic systems instead of query-focused systems due to their superior performance on ROUGE and factuality metrics on the development set. We will add a clarification in lines 433–435 to explicitly state this.\n\n> Couldn’t the background information from a recent article be used as the background summary\n\t\nIn general, we agree with the idea of using background content directly from the news articles to construct a background for the entire event. In this work, we focused on extending the existing work on timeline summarization, and therefore, we reused the information already present in previous update summaries. However, we expect future work would focus on including the news articles and content therein to construct the background without the need for timelines.\n\n> The difference between recaps and background summary needs more differentiation (lines 185-213)\n\nAs accurately pointed out by the reviewer, episode recaps and news backgrounds are closely related. A key distinction is that the recap typically only provides information from the most recent episode(s), and doesn’t provide general context to the story. In contrast, background often provides a quick overview of the event that could include information from the very first update.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Proposed the task of background summarization.\n\nMerged several timeline datasets and had humans annotate them with a background summary for each time step to create a new dataset that will be made available.\n\nProposed a query-based evaluation task\n\nBenchmark evaluation showed that the task is difficult—using state-of-the-art summarizers GPT-3.5 and Flan-T5-XL both performed worse in generating a query-focused summary than in generating a generic summary to serve as a background summary.\n\nEvaluation showed that humans generated better queries for evaluating using the proposed BUS evaluation measure than using GPT-3.5.\n\nThe paper’s strength is the setup of a new summarization task which includes a dataset, code, an evaluation method, and benchmark performance. The benchmark evaluation shows that the task is difficult.", "reasons_to_reject": "The paper mentions that journalists create timelines for major events. The paper would be strengthened if more applications of the task could be given, to indicate more general utility than just journalists.\n\nFor the evaluations comparing summaries, only generated summaries were compared in Figs 3 and 4. I would like to see generic summaries (a concatenation of the updates that performed better than the background summary in Table 3) compared to the generated summaries in the figures.", "questions_to_authors": "Related to utility of the task, in line 240, it is stated that news articles “end with necessary background information”. Couldn’t the background information from a recent article be used as the background summary, perhaps together with a brief summary from the current article? Have you compared a summary using the article’s background summary to the timeline summaries?", "paper_topic_and_main_contributions": "The paper proposes a new summarization task of creating a background summary from event timelines, which are a sequence of very brief texts describing key events for a major event that has been curated by journalists. To support this, a dataset was created by combining timeline datasets from three existing datasets and then labeled using Mechanical Turk workers pre-screened for reliability. To provide benchmarks for the task, experiments comparing three LLM summarization systems for generating both generic summaries and query-focused summaries found that generic summaries were best. Also to support the task, a query-based evaluation, Background Utility Score (BUS) was proposed. Human evaluation comparing the different query-focused LLM-generated summaries with human-created background summaries was performed; the evaluation results found the LLM-generated questions used as prompts for summaries did not produce results resembling those from human-generated questions, motivating the use of Mechanical Turk workers for generating questions instead of an LLM, which was tested and observed to perform better.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The difference between recaps and background summary needs more differentiation (lines 185-213).  They seem similar, except in recaps the data is episodes and in background summary, the data is updates:\n\n- Recaps typically present a summary of the past episode(s) that are relevant to events detailed in the current episode. \nand \n- Our definition of background constitutes a summary of previous updates in a given event that are directly relevant to the current, most recent update. \n\nWould you like to emphasize the query aspect as a differentiator?"}}, {"review_id": "TM71tbUrOn", "reviewer": "Reviewer_o6pZ", "summary": "", "questions": "A. A straightforward baseline for the background summary would be a more carefully constructed prompt for GPT3.5, which first asks the model to extract questions from the query update and then to answer the questions using the background, in the form of a paragraph. Basically your BUS metric. Have you tried this? Of course, this summary should then be evaluated by humans. \n\nB. Regarding the BUS metric and the weakness that I emphasis above, can you please explain your reasoning in not enforcing at least some predefined questions.\n\nC. How you verified if two different people came up with similar BUS questions? \n\nD. ROUGE agreement is very low on the human generated backgrounds. While this is explainable,what is the guarantee that the task is not highly subjective and hence good performance cannot be assured?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and constructive comments. We are glad the reviewer found our proposed task and experiments to be interesting and rigorous. Below, we address the reviewer’s concerns regarding one of our contributions, BUS.\n\n> The main weakness is the new metric BUS … Both system and humans do not receive any guidelines\n\nFor the human-based BUS, we provide guidelines for both the question (Figure 6 in the appendix) and answer generation (Figure 7 in the appendix). Due to the space limitations, we could not include these figures in the main paper. For the GPT-based BUS, we provide task-level guidelines (see Table 6 in the appendix). Unlike the human-based BUS, we did not include examples of background questions in the GPT-based BUS. In our preliminary analysis, we found that the GPT-3.5-based system was capable of understanding the notion of a “background question” without any need for examples.\n\n> Biased, in the sense that there are no formal requirements for at least some of the questions that the BUS metric should ask … At least some questions could have been predefined if the event update did not answer them\n\nQuestions about the background context are specific to the events described in an update. This could include a mix of question types as seen in the examples from Figures 2 and 6. Therefore, we do not expect constraints on the question types (why, what, how, etc.,) would necessarily lead to a better metric. However, we agree with the reviewer's concern about sampling different questions from GPT. This could induce some variance in the BUS scores. We believe an average score reported over five questions could be an approximate indicator of the background summary quality.\n\n> A straightforward baseline for the background summary would be a more carefully constructed prompt for GPT3.5 … Basically your BUS metric. Have you tried this?\n\nWe have not tried this baseline yet, but it is an interesting approach. Since this needs additional human evaluation, we could not do this experiment in time for our response.\n\n> How you verified if two different people came up with similar BUS questions?\n\nDue to the limited evaluation budget, we could not collect questions from two or more annotators in human-based BUS. However, we acknowledge this is a valid point and related to the above point on variance in GPT-based BUS (pre-defined question types). We will try to collect expert or crowdsource questions for a small sample and report our findings in the next draft.\n\n> ROUGE agreement is very low on the human generated backgrounds.\n\nWe acknowledge potential variance in background summaries for the same update. Therefore, we include three backgrounds for each update in our dataset. In our results, we also report average scores across the three backgrounds, minimizing the impact of subjectivity in the process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I found the paper well written and easy to follow. In my opinion, the experimental evaluation rigorous. In addition, the problem studied is interesting and has at least one application in journalism. ", "reasons_to_reject": "The main weakness is the new metric BUS that the authors propose should be used to measure how good is a background summary for a given event update. The authors present two variants: GPT is used to propose questions based on the event update or a human is asked. Both system and humans do not receive any guidelines. While this gives freedom to deal with many types of text, it also means that the question-based metric can be very biased when done via GPT or via humans. Biased, in the sense that there are no formal requirements for at least some of the questions that the BUS metric should ask, hence asking two different people or rerunning GPT with slightly different parameters might give completely different questions. At least some questions could have been predefined if the event update did not answer them, and this are the standard questions that a summary should contain: (1) Who, (2) What, (3), When, (4) Where, and (5) Why. If the authors think enforcing these questions is inappropriate, they should argue this. \n\nIn addition, the same bias applies to the creation of the dataset, as the background summaries created by humans should contain the answer to these questions, when the answer is not present in the event update. ", "questions_to_authors": "A. A straightforward baseline for the background summary would be a more carefully constructed prompt for GPT3.5, which first asks the model to extract questions from the query update and then to answer the questions using the background, in the form of a paragraph. Basically your BUS metric. Have you tried this? Of course, this summary should then be evaluated by humans. \n\nB. Regarding the BUS metric and the weakness that I emphasis above, can you please explain your reasoning in not enforcing at least some predefined questions.\n\nC. How you verified if two different people came up with similar BUS questions? \n\nD. ROUGE agreement is very low on the human generated backgrounds. While this is explainable,what is the guarantee that the task is not highly subjective and hence good performance cannot be assured?", "paper_topic_and_main_contributions": "The paper addresses the problem of event background summarization, which is defined as the task of giving all the necessary background to an event such that a reader can understand an update to that event. In particular the authors create a new dataset containing background summaries, they test background summarization generation on SOTA NLP models and finally present a metric to evaluate that the summary is useful for the task. I would say the major contribution is in dataset creation, with the note that the dataset is only in English. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FgcsNzKGBQ", "reviewer": "Reviewer_EUZX", "summary": "", "questions": "A: Regarding the BUS score, how do you plan to address the potential bias introduced by using automatically generated questions from GPT-3.5? Have you considered alternative approaches for generating evaluation questions to mitigate this bias?\n\nB: Could you provide further justification for the choice of a maximum length of 3696 for GPT-3.5 during the evaluation? How does this selection impact the performance comparison across different models?\n\nC: The choice of a split ratio of 3/3/8 for train/dev/test data is unusual in the context of natural language processing tasks where the test set is typically smaller than the training set. Could you please provide a justification for this specific split ratio and explain why the test set is significantly larger than the training set and development set?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and constructive comments. We are glad the reviewer acknowledged the importance of our proposed background summarization task as well as our dataset contribution.\n\n> how do you plan to address the potential bias introduced by using automatically generated questions from GPT-3.5?\n\nAs correctly pointed out by the reviewer, human-based BUS varies from GPT-3.5-based BUS. Human-based utility judgments are ideal and generally accepted as the gold-standard, but they are expensive to collect and are difficult to use during model development. GPT-3.5-based BUS is an efficient and cost-effective alternative that is useful for picking the best summarization systems during development and LLM-based evaluation has gained recent traction in the community (see below).\n\nNew results: Specifically, to account for any bias of using GPT-3.5-based BUS for evaluating GPT-3.5-based background summaries, we conducted a new BUS evaluation using GPT-4. This follows recent work in the literature that uses GPT-4-based evaluation (lines 534–536 in our paper). We present the results below. We report best-worst scaling scores, i.e., the percentage of times a summary type is chosen as best minus the percentage of times it was chosen as worst (lines 451–460). Our results indicate that GPT-4-based BUS is more aligned with the BUS-human results as well as the human best-worst scores (BW) results:\n\n| Method | Human | Flan-T5-XL | GPT-3.5 |\n| :--- | :---: | :---: | :---: |\n| BW | 0.243 | -0.075 | -0.168 |\n| BUS--GPT-3.5 | 0.181 | -0.297 | 0.095 |\n| BUS--Human | 0.163 | -0.07 | -0.138 |\n| BUS--GPT-4 | 0.7 | -0.353 | -0.627 |\n\nWe see the extensive human evaluations, both with human best-worst scores as well as the novel question-based BUS-Human evaluation method, as one of the contributions of our paper. We will point this out clearly in the next draft. These human evaluations led us to discover the bias inherent in GPT-3.5-based evaluations, which we will discuss in more detail in the paper as well. We believe that these findings will be a foundation for future work in automatically evaluating background summaries.\n\nWe would also like to highlight that prior work on related tasks (Section 2) such as recaps of TV scripts and disentangled contribution-context of scientific papers did not include a utility-based metric. We hypothesize our metric can be applicable in these tasks where the recap should be relevant to the current episode and the context should be sufficiently related to the contributions of a paper.\n\n> Could you provide further justification for the choice of a maximum length of 3696 for GPT-3.5 during the evaluation?\n\nGPT-3.5 allows up to 4096 tokens for both the input and completion text. We allocate 400 tokens for the completion (background summary) and budget the remainder (3696 tokens) to the input/source text. We compare it against a long-context model (LongT5 [4096 tokens]) in our experiments. Our results indicate that GPT-3.5 outperforms LongT5 even though it uses slightly shorter input text.\n\n> The paper should provide a more comprehensive explanation of why NER is employed and how it contributes to the generation of relevant questions.\n\nWe apologize for the lack of clarity about the use of NER systems in our query-focused setup. We consider two variants for queries. First, we use the full update text ($U_{t+1}$) as the query. Second, we first extract named entities from $U_{t+1}$ and use those keywords as the query. The named entity-based approach removes any potential noise from the update and focuses solely on extracting background information about important persons or locations specified in the update. As we show in Appendix A.6, both systems provide similar performance. We only included query ablations with Flan-T5 and not GPT-3.5 due to associated costs with running OpenAI API.\n\n> Could you please provide a justification for this specific split ratio and explain why the test set is significantly larger than the training set and development set?\n\nThanks for raising this important question. Recent literature in text summarization showed strong abilities of few-shot and zero-shot summarization using large language models (Goyal et al., arXiv 2022, Zhang et al., arXiv 2023). Systems such as Flan-T5 and possibly GPT-3.5 are already instruction fine-tuned on standard summarization datasets such as CNN DailyMail, Multi News, etc., and therefore capable of performing summarization with limited or sometimes no additional training data.\n\nConsidering these system capabilities, we decided to budget only a small fraction of expert-annotated data for training and development and leave most events to the test set for evaluation (Table 1). This allows sufficient data for further fine-tuning instruction models (Flan, GPT-3+) to our new task as well as maintaining sufficient diversity in the test set. All three splits include a mix of short and long timelines and the test set is mostly temporally separated from the train/dev events (Table 1).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a new and interesting task of background news summarization. This novel task has the potential to attract the interest of researchers in the field of NLP and temporal information analysis.\n\n2. The paper creates a new data collection for the task. This resource can serve as a benchmark dataset for future research.", "reasons_to_reject": "1. The potential bias introduced by the BUS score, as it relies on questions automatically generated by GPT-3.5. The bias in question generation could impact the overall evaluation and may not accurately reflect the true utility of the background summaries. The experiments show variations when human-generated questions are used, which further raises concerns about the reliability and generalizability of the metric (as also suggested by the authors).\n[EDIT: Addressed by reviewers, results may be relevant and should be included in the revised version if possible]\n2. The evaluation using different models with different maximum lengths, such as GPT-3.5 with a max length of 3696, may significantly influence the performance of the summarization systems. The choice of maximum length might not be sufficiently justified, and variations in model capabilities could obscure the fair comparison of different methods.\n[EDIT: Addressed by reviewers]\n3. Section 4, which discusses the use of Named Entity Recognition (NER) to generate questions, lacks clarity and needs improvement. The paper should provide a more comprehensive explanation of why NER is employed and how it contributes to the generation of relevant questions. \n[EDIT: Addressed by reviewers, maybe a few lines of the response may be needed in the paper also.]", "questions_to_authors": "A: Regarding the BUS score, how do you plan to address the potential bias introduced by using automatically generated questions from GPT-3.5? Have you considered alternative approaches for generating evaluation questions to mitigate this bias?\n\nB: Could you provide further justification for the choice of a maximum length of 3696 for GPT-3.5 during the evaluation? How does this selection impact the performance comparison across different models?\n\nC: The choice of a split ratio of 3/3/8 for train/dev/test data is unusual in the context of natural language processing tasks where the test set is typically smaller than the training set. Could you please provide a justification for this specific split ratio and explain why the test set is significantly larger than the training set and development set?", "paper_topic_and_main_contributions": "The paper proposes a new task, namely Background summarization of event timelines. It introduces a dataset constructed by merging existing timeline datasets and enriched with manually collected information by human annotators. The paper employs standard metrics alongside a novel metric called BUS (Background Utility Score) for evaluation. The BUS metric assesses whether the background summary effectively helps answer reader questions.\n\nThe main contributions of the paper are as follows:\n\nNew Task Introduction: The paper introduces the task of background news summarization, recognizing the need to complement timeline updates with relevant background summaries for each timestep of a news event.\n\nData Collection: To facilitate research in this new task, the paper constructs a dataset by merging existing timeline datasets and enhancing it with background summaries written by human annotators for each timestep of each news event. This dataset provides a valuable resource for further exploration and development of background summarization systems.\n\nEvaluation Metric: The authors propose a novel evaluation metric called Background Utility Score (BUS) to assess the quality of background summaries.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing references have been identified in the paper. However, it is noted that the authors skip citing relevant works related to timeline summarization, focusing only on datasets."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the task of background summarization from event timelines and proposes an experimental framework that includes a benchmark dataset and a dedicated evaluation metric. Overall, reviewers are all positive about this work after the authors's responses and discussion. They agree on the the contribution and novelty of the paper (interesting new task, introduced human-annotated dataset). Reviewers raised a concern about the potential bias/weakness of the introduced evaluation metric that was discussed in the rebuttal.
Decision: Accept-Main